Data Logging and Displaying with InfluxDB and Grafana

I  was able to collect sensor data (temperature, humidy, pressure) for a while and send the data out via MQTT. What I never finished is to display the data in a nice graphical UI. Unfortunately to do this via Internet and any hosted service (e.g. Adafruit, or AWS or Google), a requirement is to use TLS (HTTPS or MQTT over TLS). Not a problem in general, but for an ESP8266 with Espruino and about 20 KByte RAM available for user programs, the TLS stack is way too large and RAM consuming. See here. ESP32 is better (512 KByte RAM ionstead of 80 KByte in total) and it could do TLS, but I have 1 ESP32 and about 10 ESP8266 modules...

Thus I am forced to use simple and unencrypted MQTT locally with a local MQTT broker. One subscriber process can get updates and send to an Internet connectable service (i.e. the many IoT related services there are). There the data is stored and displayed.

But then I thought that instead of sending the data to another hosted DB, I can instead host a small DB myself. I can run a time series DB and a graphical front-end. E.g. ELK stack or InfluxDB & Grafana. See here for a short comparison of ElasticSearch vs. InfluxDB. For the latter I found a very useful Docker Compose setup here, so InfluxDB & Grafana it was!

How to create users in InfluxDB

Grafana should only have read-only access. The submit process should only have a write-only access. Create those users:

CREATE USER submit WITH PASSWORD 'submitpw';
GRANT WRITE ON demo TO submit;
CREATE USER grafanaro WITH PASSWORD 'testro';
GRANT READ ON demo TO grafanaro;

Then there's the DB root account of course which you can find in the docker-compose.yml file.

How to submit data into InfluxDB

This simple shell script will do and quickly generate some moderately useful data:

#!/bin/bash
pingAvg=`ping -c 5 router.lan | tail -1 | awk -F/ '{print $5}'`
curl -i -XPOST 'http://influxdb.lan:8086/write?db=demo' -u submit:submitpw --data-binary 'ping router='$pingAvg

Run it at least twice as Grafana won't display much if there's only one data point. Then configure a panel on a new dashboard in Grafana:

Reload the graph, set the time frame to the last 15min, and you should see something which proves that data ingestion works.

Next step

Create a small program to subscribe data updates from the MQTT broker and send it securely to InfluxDB.

Comments

AWS S3 Signed URL's

I saw some questions on the web regarding signed S3 URLs. Those would allow someone else (not an AWS IAM user) to access S3 objects. E.g. if I have a program which has permissions to a given S3 object, I can create a signed URL which allows anyone with the knowledge of that URL to (e.g.) read the object. Or write. A simple example would be a video training web site: I could give the user a URL which is valid 24h to they can watch a video as many times as they like, but 24h only. The alternative would be the URL of the S3 object directly.

There are many ways to solve this problem, but signed URLs is what AWS offers.

Since there were so many postings and questions around this, I wondered what the problem was. The documentation at https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#getSignedUrl-property certainly looked straightforward.

So a quick program created:

const AWS = require('aws-sdk')

const s3 = new AWS.S3()
// above is using ~/.aws/config.json to get my API key credentials
// That API key inside config.json obviously has permission to the object.
// A normal web browser cannot access the S3 URL ythough as the
// bucket is not public.

const myBucket = 'BUCKET'
const myKey = 'FILE.json'
const signedUrlExpireSeconds = 60 * 5 // 5min

const url = s3.getSignedUrl('getObject', {
    Bucket: myBucket,
    Key: myKey,
    Expires: signedUrlExpireSeconds
})

console.log(url)

and it all worked (AccessKeyId has access to the S3 object):

harald@blue:~/js/aws$ node sign.js 
https://BUCKET.s3.amazonaws.com/FILE.json?AWSAccessKeyId=AXXXXXXXXXXXXXXXXXXA&Expires=1529832632&Signature=D7eArF9AMFyWr%2FLoXcCQ0pA72i8%3D
harald@blue:~/js/aws$ curl "https://BUCKET.s3.amazonaws.com/FILE.json?AWSAccessKeyId=AXXXXXXXXXXXXXXXXXXA&Expires=1529832632&Signature=D7eArF9AMFyWr%2FLoXcCQ0pA72i8%3D"
{
      "AWSTemplateFormatVersion" : "2010-09-09",
      "Resources" : {
[...]
}

It's as easy as I thought.

Comments

Yubikey, PGP and SSH and Chromebooks

https://pgp.mit.edu/pks/lookup?op=get&search=0x9A043EF5DC61A9D5 is where my PGP public key can be found.

Why the sudden interest in PGP?

Basically it enables me to put my secret key on a Yubikey instead of a computer. They secret key is secured by a PIN which can only guessed x times (3 is default), then you have to use a PUK (3 times again), and then it's locked. Only a full reset will get it out of this and that will erase the private key too. That's way better than a passphrase secured private key file on a computer which can be cracked over time, though it might take a long time if the passphrase is a good one.

And the best: It works out-of-the-box with my Chromebook and the Secure Shell App via the Smart Card Connector! It also works on my Linux desktop with Chrome. Windows does not work though as the Smart Card Connector does not work as expected as the Chrome Smart Card Connector does not work on non-Linux.

The links to read (not in any particular order):

  1. https://www.esev.com/blog/post/2015-01-pgp-ssh-key-on-yubikey-neo/
  2. https://chromium.googlesource.com/apps/libapps/+/HEAD/nassh/doc/hardware-keys.md
  3. http://deferred.io/2017/08/03/yubikey4-gpg-ssh-u2f.html
  4. http://www.engineerbetter.com/blog/yubikey-ssh/

While not trivial to set up, it's very rewarding to know to not have a private ssh key on an inherently insecure computer.

Comments

High Speed Oven

At Subway (the sandwich company) they got high speed ovens which toast bread in like 20 seconds. I always wondered what magic is inside to be that quick.Today I found out that you can even buy those ovens if you have a need for them. Like this one: https://www.turbochef.com/product/bullet/

And the magic they use: hot air as well as microwaves.

Before you think about buying one: it'll cost you about US$10k, and the Japanese version uses 200V and 30A (resp. 400V/16A for those in 230V countries).

Comments

OTC Medicine in Japan

OTC medicine is plenty available in Japan, but it's hard to match it to what you know in your home country. Reading Katakana helps a lot and having studied Chemistry too, but it would be so nice to get a nice English overview of various  Japanese OTC medicine.

Luckily someone did that: https://lifeabroad.jp/html/medical_health/otc.html

Of course do not take that web page as the definite truth and cross-check the ingredients and dosage.

Comments