Data fetching and visualization using d3.js

complex-664440_1280

Data fetching is the process of grabbing data from a data source. In our lora network, packets are shipped off to The Things Network API.

On the web, javascript is what we got for doing computation in a browser. Ajax is a set of techniques used on the client-side to create asynchronous web applications.

ALSO READ: Temperature monitoring with lora

To fetch resources asynchronously, utilize the XMLHttpRequest object:

var trtHttp = (function (my) {
    var http = new XMLHttpRequest();

    my.getJson = function (url, cb) {
        http.onreadystatechange = function () {
            if (http.readyState === XMLHttpRequest.DONE) {
                if (http.status === 200) {
                    cb(JSON.parse(http.responseText));
                }
            }
        };

        http.open("GET", url, true);
        http.send();
    }

    return my;
}(trtHttp || {}));

The wrapping of getJson inside trtHttp is a javascript way of namespacing functions.
With this pattern, member functions of trtHttp can be spread out in different files.

Fetch data from our temperature sensor:

var url = 'http://thethingsnetwork.org/api/v0/nodes/02031003/?limit=200&format=json';

trtHttp.getJson(url, function (data) {
    console.log(data);
});

If you don’t feel the need for going this lowlevel, you can use jQuery:

$.getJSON(url, function (data) {
    console.log(data);
});

This achieve the same thing.

ALSO READ: Sending text with lorawan towards the things network

Data visualization

D3.js is a javaScript library for manipulating documents based on data. Let’s create a bar chart from a dataset containing integers between 0 and 10.

barchart-integers-2

var data = [1,2,3,4,5,6,7,8,9]

var margin = {
    bottom: 10,
    top: 10,
    left: 50,
    right: 20
};

var width = 500;
var height = 300;

var chartWidth = width - margin.left - margin.right;
var chartHeight = height - margin.top - margin.bottom;

var yScale = d3.scale.linear()
    .domain([0, 10])
    .range([chartHeight, margin.bottom + margin.top])

var yAxis = d3.svg.axis()
    .scale(yScale)
    .orient('left');

var svg = d3.select('body')
    .append('svg')
    .attr('width', width)
    .attr('height', height)

var chart = svg.append('g')
    .attr('width', chartWidth)
    .attr('height', chartHeight)
    .attr('transform', "translate(" + margin.left + ",0)")

svg.append('g')
    .attr('transform', "translate(30,0)")
    .attr("class","axis")
    .call(yAxis)

chart.selectAll('rect')
    .data(data)
    .enter()
    .append('rect')
    .attr('fill', function (d) {
        return "rgb(0, 0, " + (d * 30) + ")";
    })

    .attr('x', function (d, i) {
        return i * chartWidth / data.length
    })

    .attr('y', function (d) {
        return yScale(d);
    })

    .attr('width', chartWidth / data.length-1)

    .attr('height', function (d) {
        return yScale(0) - yScale(d);
    })

To fit our data of integers onto an svg we need to convert the integer range [0, 10] to [0, 300]. This is needed because otherwise the bars would be very small since the height of the svg is 300 pixels.
The d3.scale function performs the conversion. We need only provide it with an input domain and an output range.

To use the data from our temperature node, we simply wrap the code inside a json data fetch block:

trtHttp.getJson(url, function (data) {

    var data = data.map(function (item) {
        return item.data_json;
    })

    [...]
})

The temperature node location is in our office right now, so temperature values vary between 22 and 23 celcius.
The graph is not particurlarly exciting. Here it is:

barchart-temperatures

Temperature monitoring with lora

IMG_20151216_134116This is how I programmed the NUCLEO-L152RE microcontroller and Grove – Temperature&Humidity Sensor Pro to monitor temperature and humidity. The temperature readings are shipped off to the things network via our LoRaWAN network.

The purpose of this is to help others get started using the LoRaWAN technology to build IOT applications.

ALSO READ: Sending text with lorawan towards the things network

Temp_humi_pro
Grove – Temperature and Humidity Sensor Pro

To read temperature data from the Grove sensor I imported the DHT C++ library. Next I added an include for the header file

#include "DHT.h"

The temperature is read with


DHT sensor(A1, AM2302); // AM2302 is the product name of the sensor
int err = sensor.readData();

float getTemperature() {

    int err = 1;
  
    while(err != 0) {
        wait(2.0f); // Wait 2 seconds
        err = sensor.readData();
    }
    
    return sensor.ReadTemperature(CELCIUS);
}

The humidity sensor is accessed in a similar way: sensor.ReadHumidity()

I plugged the grove sensor into A0 of the grove shield. This turned out not to work because the Semtech lora transceiver uses this port for NRESET. I used A1 instead.

I’m occasionally getting

Err 6

from sensor.readData(), which is a checksum (CRC) error.
To get a temperature value without error, I read the sensor in a loop until there is no error.

After a few flashings of binary onto the microcontroller I got this error:


Jan 18 11:11:15 bergman kernel: VFS: busy inodes on changed media or resized disk sdb
cp: error writing ‘/media/d/NODE_L152RE/LoRaWAN_send_text_2_NUCLEO_L152RE.bin’: No space left on device
cp: failed to extend ‘/media/d/NODE_L152RE/LoRaWAN_send_text_2_NUCLEO_L152RE.bin’: No space left on device

I’m unable to repair the root cause of this. But replugging the device fixes the symptom.

To execute the temperature reading code each 60s I add this at the end of the onSendFrame function:

os_setTimedCallback(j, os_getTime() + sec2osticks(60), onSendFrame);

Live demo

temp_and_hum

See our live demo for real-time humidity and temperature readings! Also see our github repo for its code.

The things network API do not support TLS yet. I had to add an exception in our apache config to circumvent browser blocking of non-TLS content:

RewriteRule ^/demo/lora-temperature - [L]

The full code

#include "mbed.h"
#include "lmic.h"
#include "debug.h"
#include "Serial.h"

#include "DHT.h"

#define LORAWAN_NET_ID (uint32_t) 0x00000000
#define LORAWAN_DEV_ADDR (uint32_t) 0x02031003
#define LORAWAN_ADR_ON 1
#define LORAWAN_CONFIRMED_MSG_ON 1
#define LORAWAN_APP_PORT 3//15

DHT sensor(A1, AM2302);

Serial co2(D8, D2);

static uint8_t NwkSKey[] = {
    0x2B, 0x7E, 0x15, 0x16, 0x28, 0xAE, 0xD2, 0xA6, 
    0xAB, 0xF7, 0x15, 0x88, 0x09, 0xCF, 0x4F, 0x3C
};

static uint8_t ArtSKey[] = {
    0x2B, 0x7E, 0x15, 0x16, 0x28, 0xAE, 0xD2, 0xA6, 
    0xAB, 0xF7, 0x15, 0x88, 0x09, 0xCF, 0x4F, 0x3C
};

osjob_t initjob;
osjob_t sendFrameJob;
u1_t n = 0;

void os_getArtEui (uint8_t *buf) {} // ignore
void os_getDevEui (uint8_t *buf) {} // ignore
void os_getDevKey (uint8_t *buf) {} // ignore

float getTemperature() {

    int err = 1;
  
    while(err != 0) {
        wait(2.0f);
        err = sensor.readData();
    }
    
    return sensor.ReadTemperature(CELCIUS);
}

float getHumidity() {
    return sensor.ReadHumidity();
}
    
void onSendFrame (osjob_t* j) {
 
    const unsigned char cmd_get_sensor[] = {
        0xff, 0x01, 0x86, 0x00, 0x00,
        0x00, 0x00, 0x00, 0x79
    };

    for (int i = 0; i < 9; i++) {
        co2.putc(cmd_get_sensor[i]);
        wait(0.01);
    }
    
    char message[32];
    
    float temperature = getTemperature();
    printf("Temperature is %4.2f \r\n", temperature);
    
    float humidity = getHumidity();
    printf("Humidity is %4.2f \r\n", humidity);

    sprintf(message, "%4.2f %4.2f", temperature, humidity);
    
    int frameLength = strlen(message); // keep it < 32
    for (int i = 0; i < frameLength; i++) {
        LMIC.frame[i] = message[i];
    }
    int result = LMIC_setTxData2(LORAWAN_APP_PORT, LMIC.frame, 
        frameLength, LORAWAN_CONFIRMED_MSG_ON); // calls onEvent()
          
    os_setTimedCallback(j, os_getTime() + sec2osticks(60), onSendFrame);
}

void onInit (osjob_t* j) {
    LMIC_reset();
    LMIC_setAdrMode(LORAWAN_ADR_ON);
    LMIC_setDrTxpow(DR_SF12, 14);
    LMIC_setSession(LORAWAN_NET_ID, LORAWAN_DEV_ADDR, NwkSKey, ArtSKey);
 
    onSendFrame(NULL);
}

void onEvent (ev_t ev) { // called by lmic.cpp, see also oslmic.h
    debug_event(ev);
    if (ev == EV_TXCOMPLETE) {
        os_setCallback(&sendFrameJob, onSendFrame);
    }
}

int main (void) {
    debug_str("Entered main\r\n");
    os_init();
    os_setCallback(&initjob, onInit);
    os_runloop(); // blocking
}

Sending text with lorawan towards the things network

This is a writeup describing how I programmed the NUCLEO-L152RE microcontroller to ship off some ascii text to our lora gateway. Our gateway is connected to the things network. Our address space is 02:03:10:xx, which we have claimed on the TTN wiki.

STM32 Nucleo board
STM32 Nucleo board

To connect the lora radio chip, we use the Seeed Grove Shield V2 expansion board.

 As an expansion board, Base Shield v2 has many Grove connectors, making it convenient for you to use Grove products together
As an expansion board, Base Shield v2 has many Grove connectors, making it convenient for you to use Grove products together

To communicate to the lora gateway we use the SX1276MB1xAS transceiver.

The SX1276MB1MAS and SX1276LB1LAS are both fitted with the SX1276 transceiver which, added to a high-performance FSK / OOK RF transceiver modem, features the LoRa™ long range modem.
The SX1276MB1MAS and SX1276LB1LAS are both fitted with the SX1276 transceiver which, added to a high-performance FSK / OOK RF transceiver modem, features the LoRa™ long range modem.

The three devices all connected looks like this:

IMG_20160114_140004

Producing the binary

To create binaries from code I used the compiler from developer.mbed.org. It is a web IDE that simplifies and speeds up the creation and deployment of devices based on ARM microcontrollers. To create a program I imported the LoRaWAN_send_text repository from a user called tamberg.

The first change I do in the code is to set the device address:

#define LORAWAN_DEV_ADDR (uint32_t) 0x02031002

First I hit compile and a binary LoRaWAN_send_text_NUCLEO_L152RE.bin is downloaded to my local storage.

Next I plug the Nucleo into my machine using micro-USB cable. This presents a USB mass storage device to my machine. To upload the binary I conveniently move the file over and the microcontroller is flashed immediately after.

mv -v LoRaWAN_send_text_2_NUCLEO_L152RE.bin /media/d/NODE_L152RE/

Shortly after, the LD1 LED blinks furiously indicating that new binary is flashed onto device.

A few seconds later the data has arrived to TTN and is exposed on their HTTP API. Here is the data:

{
  "data_raw": "gAIQAwKAAAADNTFQd4kvk4XcLgU++F/zahI=",
  "gateway_eui": "AA555A0008060353",
  "node_eui": "02031002",
  "frequency": 868.1,
  "data_plain": "YO Trondheim ",
  "time": "2016-01-14T13:43:08.175Z",
  "rssi": -96,
  "snr": 9.0,
  "datarate": "SF12BW125",
  "data": "WU8gVHJvbmRoZWltIA=="
}

The "gateway_eui": "AA555A0008060353" is our lora gateway located on the rooftop at olavskvartalet in Trondheim.

The full code

// License: Revised BSD License, see LICENSE.TXT, (c)2015 Semtech

#include "mbed.h"
#include "lmic.h"
#include "debug.h"

#define LORAWAN_NET_ID (uint32_t) 0x00000000
// TODO: enter device address below, for TTN just set ???
#define LORAWAN_DEV_ADDR (uint32_t) 0x02031002
#define LORAWAN_ADR_ON 1
#define LORAWAN_CONFIRMED_MSG_ON 1
#define LORAWAN_APP_PORT 3//15

static uint8_t NwkSKey[] = {
    // TODO: enter network, or use TTN default
    // e.g. for 2B7E151628AED2A6ABF7158809CF4F3C =>
    0x2B, 0x7E, 0x15, 0x16, 0x28, 0xAE, 0xD2, 0xA6, 
    0xAB, 0xF7, 0x15, 0x88, 0x09, 0xCF, 0x4F, 0x3C
};

static uint8_t ArtSKey[] = {
    // TODO: enter application key, or use TTN default
    // e.g. for 2B7E151628AED2A6ABF7158809CF4F3C =>
    0x2B, 0x7E, 0x15, 0x16, 0x28, 0xAE, 0xD2, 0xA6, 
    0xAB, 0xF7, 0x15, 0x88, 0x09, 0xCF, 0x4F, 0x3C
};

osjob_t initjob;
osjob_t sendFrameJob;
u1_t n = 0;

void os_getArtEui (uint8_t *buf) {} // ignore
void os_getDevEui (uint8_t *buf) {} // ignore
void os_getDevKey (uint8_t *buf) {} // ignore

void onSendFrame (osjob_t* j) {
    const char* message = "YO Trondheim "; // ASCII only
    int frameLength = strlen(message); // keep it < 32
    for (int i = 0; i < frameLength; i++) {
        LMIC.frame[i] = message[i];
    }
    int result = LMIC_setTxData2(LORAWAN_APP_PORT, LMIC.frame, 
        frameLength, LORAWAN_CONFIRMED_MSG_ON); // calls onEvent()
}

void onInit (osjob_t* j) {
    LMIC_reset();
    LMIC_setAdrMode(LORAWAN_ADR_ON);
    LMIC_setDrTxpow(DR_SF12, 14);
    LMIC_setSession(LORAWAN_NET_ID, LORAWAN_DEV_ADDR, NwkSKey, ArtSKey);
    onSendFrame(NULL);
}

void onEvent (ev_t ev) { // called by lmic.cpp, see also oslmic.h
    debug_event(ev);
    if (ev == EV_TXCOMPLETE) {
        os_setCallback(&sendFrameJob, onSendFrame);
    }
}

int main (void) {
    debug_str("main\r\n");
    os_init();
    os_setCallback(&initjob, onInit);
    os_runloop(); // blocking
}

This OS-like code is the IBM LoRaWAN C-library (LMiC) portable implementation of the LoRaTM MAC specification for the C programming language. It drives the SEMTECH SX1272 radio.

The LMiC library can be accessed via a set of API functions, run-time functions, callback functions, and a global LMIC data structure.

The LMiC library offers a simple event-based programming model where all protocol events are dispatched to the application’s onEvent() callback function. In order to free the application of details like timings or interrupts, the library has a built-in run-time environment to take care of timer queues and job management.

LoRaWAN gateway på olavskvartalet

Radioteknologi har eksistert lenge. Flere og flere ting blir påkoblet internett. Etterhvert som teknologien blir mindre kostbar vil det bli lønnsomt å koble flere av våre ting på internett. Bruk av WiFi (802.11) som infrastruktur for tingenes internett er for kostbart. På grunn av høyt batterikrav og kort rekkevidde.

Vi i Trådløse Trondheim mener LoRaWAN er en god kandidat til å bygge infratruktur på.

LoRa er en forkortelse som står for long range. Forkortelsen beskriver den mest verdifulle egenskapen til denne teknologien.

LoRaWAN er en Low Power Wide Area Network (LPWAN) standard, for trådløse batteridrevne enheter til bruk i tingenes internett.

Det tyske firmaet Kerlink har produsert en LoRa gateway med navn LoRa IoT Station 868 MHz.

LoRa_station_435

Vi har kjøpt en slik gateway og plassert den på taket på olavskvartalet i Trondheim.

IMG_0953
Dag og Thomas etter vellykket installering av LoRaWAN gateway på olavskvartalet.
Utsikt ned fra taket på olavskvartalet.
Utsikt ned fra taket på olavskvartalet.

For å finne dens IP adresse jaktet vi DHCP-loggen like etter påkobling via ethernet. LoRa gateway-en fungerer som en trådløs router. Den mottar trådløse pakker fra alle enheter som følger LoRaWAN standarden. For å gjøre dataene lett tilgjengelig bør vi dytte pakken videre til en internett-sky.

Vi har foreløpig valgt å dytte pakkene videre til the things network (TNN). Transporten gjøres med UDP på port 1700.

For å instruere gatewayen til å dytte pakker til TTN ssh-et jeg inn til den og gjorde denne endringen i /mnt/fsuser-1/forwarder_network_demo/global_conf.json:


"gateway_conf": {
/* change with default server address/ports, or overwrite in local_conf.json */
"server_address": "croft.thethings.girovito.nl",
"serv_port_up": 1700,
"serv_port_down": 1700,
/* adjust the following parameters for your network */
"keepalive_interval": 10,
"stat_interval": 30,
"push_timeout_ms": 100,
/* forward only valid packets */
"forward_crc_valid": true,
"forward_crc_error": false,
"forward_crc_disabled": true,
"autoquit_threshold" : 3
}

For å lytte etter trafikk på port 1700 er tcpdump et ypperlig verktøy:

tcpdump -vX port 1700

I mitt tilfelle detekterer den:


14:31:29.740799 IP (tos 0x0, ttl 47, id 41867, offset 0, flags [DF], proto UDP (17), length 32)
ec2-54-72-145-119.eu-west-1.compute.amazonaws.com.1700 > trt-olav-loragw01.trtr.no.50113: UDP, length 4
0x0000: 4500 0020 a38b 4000 2f11 c877 3648 9177 E.....@./..w6H.w
0x0010: 0a09 0e02 06a4 c3c1 000c b279 0128 a204 ...........y.(..
0x0020: 0000 0000 0000 0000 0000 0000 0000 63ba ..............c.
0x0030: c4a4 ..

TTN tar imot pakkene fra vår gateway og eksponerer dem via et HTTP API. Vår gateway id er AA555A0008060353.
Jeg forventer at pakkene som blir sendt til TTN havner på http://thethingsnetwork.org/api/v0/gateways/AA555A0008060353/. Men siden er fortsatt tom av en eller annen grunn.