Feb 192017
HAProxy, docker-compose and Let's Encrypt

Linode started to offer US$5 VMs and they are available in Tokyo (9ms ping as opposed to 122ms ping to California), so I could not resist to get another one and use it for some experimenting which I simply don’t dare to do on this very blog page (and my wife is using it for work too).



  • 2 web servers serving web server stuff (a simple static index.html serves here)
  • 3 virtual web addresses:
    • first web server
    • second web server
    • round-robin web server which serves data from the first and second web server
  • Everything with SSL without any warning from web browsers


  • 1 VM with a public IP address
  • 3 FQDNs pointing all to above IP address
    • www1.qw2.org
    • www3.qw2.org
    • www4.qw2.org
  • 2 web servers (www1, www3)
  • 1 load-balancer (www4)
  • 3 certificates for above FQDNs

Let’s Encrypt Certificates

I use acme.sh for my few certificate needs. This is how to get a new certificate issued:

./acme.sh --issue --dns dns_linode --dnssleep 1200 -d www4.qw2.org

This is using the DNS API from Linode (who hosts my DNS records). See more details here. It creates the required TXT record and removes it later again. I found that 1200 seconds wait time works. 900 does not always. I end up using 10 seconds, suspend the acme.sh command (shell ^Z), and use “dig -t TXT _acme-challenge.www4.qw2.org” until it returns some TXT record. Then continue the suspended acme.sh command.

You should then have a new directory www4.qw2.org in your acme.sh directory with those files:

harald@blue:~/.acme.sh$ ls -la www4.qw2.org/ 
total 36 
drwxr-xr-x  2 harald users 4096 Feb 19 10:16 . 
drwx------ 17 harald users 4096 Feb 19 00:26 .. 
-rw-r--r--  1 harald users 1647 Feb 19 10:16 ca.cer 
-rw-r--r--  1 harald users 3436 Feb 19 10:16 fullchain.cer 
-rw-r--r--  1 harald users 1789 Feb 19 10:16 www4.qw2.org.cer 
-rw-r--r--  1 harald users  517 Feb 19 10:16 www4.qw2.org.conf 
-rw-r--r--  1 harald users  936 Feb 19 00:26 www4.qw2.org.csr 
-rw-r--r--  1 harald users  175 Feb 19 00:26 www4.qw2.org.csr.conf 
-rw-r--r--  1 harald users 1675 Feb 19 00:26 www4.qw2.org.key

You’ll need the fullchain.cer and the private key www4.qw2.org.key later.

Repeat for www1 and www3 too.

Note that the secret key is world readable. the .acme.sh directory is therefore secured with 0700 permissions.

Setting up the Web Servers

Using lighttpd herte. The full directory structure:

harald@lintok1:~$ tree lighttpd 
├── 33100 
│   ├── etc 
│   │   ├── lighttpd.conf 
│   │   ├── mime-types.conf 
│   │   ├── mod_cgi.conf 
│   │   ├── mod_fastcgi.conf 
│   │   ├── mod_fastcgi_fpm.conf 
│   │   └── www1.qw2.org 
│   │       ├── combined.pem 
│   │       └── fullchain.cer 
│   └── htdocs 
│       └── index.html 
├── 33102 
│   ├── etc 
│   │   ├── lighttpd.conf 
│   │   ├── mime-types.conf 
│   │   ├── mod_cgi.conf 
│   │   ├── mod_fastcgi.conf 
│   │   ├── mod_fastcgi_fpm.conf 
│   │   └── www3.qw2.org 
│   │       ├── combined.pem 
│   │       └── fullchain.cer 
│   └── htdocs 
│       └── index.html 
└── docker-compose.yml

Using the lighttpd.conf is simple and can be done in 5 or 10 minutes. The part for enabling https is this:

$SERVER["socket"] == ":443" { 
ssl.engine    = "enable" 
ssl.pemfile   = "/etc/lighttpd/www1.qw2.org/combined.pem" 
ssl.ca-file   = "/etc/lighttpd/www1.qw2.org/fullchain.cer" 

fullchain.cer is the one you get from the Let’s Encrypt run. “combined.pem” is created via

cat fullchain.cer www1.qw2.org.key > combined.pem

Here the content of docker-compose.yml:

  image: sebp/lighttpd 
    - /home/harald/lighttpd/33100/htdocs:/var/www/localhost/htdocs 
    - /home/harald/lighttpd/33100/etc:/etc/lighttpd 
    - 33100:80 
    - 33101:443
  restart: always
  image: sebp/lighttpd 
    - /home/harald/lighttpd/33102/htdocs:/var/www/localhost/htdocs 
    - /home/harald/lighttpd/33102/etc:/etc/lighttpd 
    - 33102:80 
    - 33103:443
  restart: always

To start those 2 web servers, use docker-compose:

docker-compose up

If you want to have a reboot automatically restart the service, then use do “docker-compose start” afterwards which installs a service.

To test, access: http://www1.qw2.org:33100, https://www1.qw2.org:33101, http://www3.qw2.org:33102, https://www3.qw2.org:33103

They all should work, and the https pages should find a proper security status (valid certificate, no name mismatch etc.).

Adding HAProxy

HAProxy (1.7.2 as of the time of writing) can be the SSL termination and forwarded traffic between the web server and HAProxy is unencrypted (resp. can be encrypted via another method), or HAProxy can simply forward traffic. Which one is preferred depends on the application. In my case it makes most sense to let HAProxy handle SSL.

First the full directory structure:

├── docker-compose.yml 
└── etc 
    ├── errors 
    │   ├── 400.http 
    │   ├── 403.http 
    │   ├── 408.http 
    │   ├── 500.http 
    │   ├── 502.http 
    │   ├── 503.http 
    │   ├── 504.http 
    │   └── README 
    ├── haproxy.cfg 
    └── ssl 
        └── private 
            ├── www1.qw2.org.pem 
            ├── www3.qw2.org.pem 
            └── www4.qw2.org.pem

The www{1,3}.qw2.org.pem were copied from the lighttpd files.


harald@lintok1:~/haproxy/etc$ cat haproxy.cfg  
        user nobody 
        group users 
        # Admin socket 
        stats socket /var/run/haproxy.sock mode 600 level admin 
        stats timeout 2m 
        # Default SSL material locations 
        #ca-base /usr/local/etc/haproxy/ssl/certs 
        #crt-base /usr/local/etc/haproxy/ssl/private 
        # Default ciphers to use on SSL-enabled listening sockets. 
        # For more information, see ciphers(1SSL). 
        tune.ssl.default-dh-param 2048 
        ssl-default-bind-options no-sslv3 no-tls-tickets 
        ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA
        ssl-default-server-options no-sslv3 no-tls-tickets 
        ssl-default-server-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-R
    log     global 
    mode    http 
    option  dontlognull 
    timeout connect 5000 
    timeout client  50000 
    timeout server  50000 
    errorfile 400 /usr/local/etc/haproxy/errors/400.http 
    errorfile 403 /usr/local/etc/haproxy/errors/403.http 
    errorfile 408 /usr/local/etc/haproxy/errors/408.http 
    errorfile 500 /usr/local/etc/haproxy/errors/500.http 
    errorfile 502 /usr/local/etc/haproxy/errors/502.http 
    errorfile 503 /usr/local/etc/haproxy/errors/503.http 
    errorfile 504 /usr/local/etc/haproxy/errors/504.http 
    stats enable 
    stats uri /stats 
    stats realm Haproxy\ Statistics 
    stats auth admin:SOME_PASSWORD 
frontend http-in 
    bind *:80 
    acl is_www1 hdr_end(host) -i www1.qw2.org 
    acl is_www3 hdr_end(host) -i www3.qw2.org 
    acl is_www4 hdr_end(host) -i www4.qw2.org 
    use_backend www1 if is_www1 
    use_backend www3 if is_www3 
    use_backend www4 if is_www4 
frontend https-in 
    bind *:443 ssl crt /usr/local/etc/haproxy/ssl/private/ 
    reqadd X-Forward-Proto:\ https 
    acl is_www1 hdr_end(host) -i www1.qw2.org 
    acl is_www3 hdr_end(host) -i www3.qw2.org 
    acl is_www4 hdr_end(host) -i www4.qw2.org 
    use_backend www1 if is_www1 
    use_backend www3 if is_www3 
    use_backend www4 if is_www4 
backend www1 
    balance roundrobin 
    option httpclose 
    option forwardfor 
    server s1 www1.qw2.org:33100 maxconn 32 
backend www3 
    balance roundrobin 
    option httpclose 
    option forwardfor 
    server s3 www3.qw2.org:33102 maxconn 32 
backend www4 
    balance roundrobin 
    option httpclose 
    option forwardfor 
    server s4-1 www1.qw2.org:33100 maxconn 32 
    server s4-3 www3.qw2.org:33102 maxconn 32 
listen admin 
    bind *:1936 
    stats enable 
    stats admin if TRUE

Replace “SOME_PASSWORD” with an admin password for the admin user who can stop/start backends via the Web UI.

Here the docker-compose.yml file to start HAProxy:

harald@lintok1:~/haproxy$ cat docker-compose.yml  
  image: haproxy:1.7 
    - /home/harald/haproxy/etc:/usr/local/etc/haproxy 
    - 80:80 
    - 443:443 
    - 1936:1936
  restart: always

To start haproxy, do:

docker-compose up

The Result

Now http://www1.qw2.org as well as https://www1.qw2.org works. No need for specific ports like 33100 or 33101 anymore. Same for www3.qw2.org. www4.qw2.org is a round-robin of www1 and www3, but it’s using the www4 certificate when using https. In all cases HAProxy terminates the SSL connections and it’s presenting the correct certificates.

Related: on http://www4.qw2.org:1936/haproxy?stats you can see the statistics of HAProxy.

Connecting it all all

Running 2 web servers plus the load-balancer with all of them internally connected and only the load-balancer visible on port 80 resp. 443 needs a new docker-compose.yml (changed to version 3 syntax) and a slight matching change haproxy.conf file:

harald@lintok1:~/three$ cat docker-compose.yml 
version: '3' 
    image: sebp/lighttpd 
      - /home/harald/lighttpd/33100/htdocs:/var/www/localhost/htdocs 
      - /home/harald/lighttpd/33100/etc:/etc/lighttpd 
      - 80 
    restart: always 
    image: sebp/lighttpd 
      - /home/harald/lighttpd/33102/htdocs:/var/www/localhost/htdocs 
      - /home/harald/lighttpd/33102/etc:/etc/lighttpd 
      - 80 
    restart: always 
    image: haproxy:1.7 
      - /home/harald/three/haproxy/etc:/usr/local/etc/haproxy 
      - 80:80 
      - 443:443 
      - 1936:1936 
    restart: always

No need for lighttpd to handle SSL anymore (no more port 443 needed to be exposed at all). Only the HAProxy is visible from outside. Small changes are needed on haproxy.conf, but only in the backend section:

backend www1 
    balance roundrobin 
    option httpclose 
    option forwardfor 
    server s1 lighttpd-33100:80 maxconn 32 
backend www3 
    balance roundrobin 
    option httpclose 
    option forwardfor 
    server s3 lighttpd-33102:80 maxconn 32 
backend www4 
    balance roundrobin 
    option httpclose 
    option forwardfor 
    server s4-1 lighttpd-33100:80 maxconn 32 
    server s4-3 lighttpd-33102:80 maxconn 32

And with “docker ps” we can see what’s happening under the hood of docker-compose:

harald@lintok1:~/three$ docker ps 
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS         
     PORTS                                                              NAMES 
742a2e5388f2        sebp/lighttpd       "lighttpd -D -f /e..."   4 minutes ago       Up 3 minutes   
     80/tcp                                                             three_lighttpd-33100_1
9d4c61e6c162        sebp/lighttpd       "lighttpd -D -f /e..."   4 minutes ago       Up 3 minutes   
     80/tcp                                                             three_lighttpd-33102_1 
2e41dfa26ac9        haproxy:1.7         "/docker-entrypoin..."   4 minutes ago       Up 3 minutes>80/tcp,>443/tcp,>1936/tcp   three_haproxy_1 




Jan 092017

One problem for using HTTPS was in the past that sharing DNS names with one web server was not supported. HTTP can handle this for longer.

However HTTPS also supports virtual HTTPS servers: SNI does that.

And here is how to use it with HAProxy:

frontend https-in 
        bind *:443 ssl crt /etc/haproxy/ssl/private/ 
        reqadd X-Forward-Proto:\ https 
        acl is_site1 hdr_end(host) -i www1.qw2.org 
        acl is_site2 hdr_end(host) -i www2.qw2.org 
        use_backend site1 if is_site1 
        use_backend site2 if is_site2

All the magic is in the bind line where a directory with PEM certificates (concat of fullchain.cer and the key)

Now you can have https for everything and HAProxy will handle all the secure connectivity for you.

Jan 092017

To run boxbackup on a system which uses systemd, use this little systemd service file (taken from here and modified for Ubuntu):

# /etc/systemd/system/boxclient.service
# This service file runs an Box Backup daemon that runs backups on demand.

Description=Box Backup Client

ExecStart=/usr/bin/bbackupd -F -c /etc/boxclient/bbackupd.conf


Enable and start via

systemctl enable boxclient.service
systemctl start boxclient.service

 Posted by at 17:40  Tagged with:
Dec 182016
Orange Pi Zero - I/O

I finally figured out how to do simple I/O on the GPIO pins on the Orange Pi Zero. It’s actually the same as Raspberry Pi and it relies on /sys/class/gpio/

Most important: Use Armbian, and I in particular use 3.4.113 (legacy).



Linux GPIO SOC Label CON4 CON4 Label SOC Linux GPIO
Vcc3V3-EXT 1 2 DCIN-5V
12 PA12 TWI0-SDA 3 4 DCIN-5V
11 PA11 TWI0-SCK 5 6 GND
6 PA6 PWM1 7 8 UART1_TX PG6 198
GND 9 10 UART1_RX PG7 199
1 PA1 UART2_RX 11 12 PA7 PA7 7
0 PA0 UART2_TX 13 14 GND
3 PA3 UART2_CTS 15 16 TWI1-SDA PA19  19
VCC3V3-EXT 17 18 TWI1-SCK PA18 18
15 PA15 SPI1_MOSI 19 20 GND
16 PA16 SPI1_MISO 21 22 UART2_RTS PA2 2
14 PA14 SPI1_CLK 23 24 SPI1_CS PA13 13
GND 25 26 PA10 PA10?

Note: PA10 does not seem to work.


To enable one GPIO pin, do this as root:

echo 7 >/sys/class/gpio/export
chmod a+rw /sys/class/gpio/gpio7/*

Now in NodeJS you can do (as non-root):

var Gpio = require('onoff').Gpio, 
  led = new Gpio(7, 'out'); 
function blinkled7() { 
  var state=false; 
  return function () { 
    if (state) led.writeSync(0) 
    else led.writeSync(1); 
    state = ! state; 
setInterval(f, 1000);

which blinks GPIO pin 7 (AKA PA7). You can also watch pins:

var Gpio = require('onoff').Gpio,
  led = new Gpio(7, 'out'),
  button = new Gpio(16, 'in', 'both');

button.watch(function (err, value) {
  if (err) {
    throw err;
  console.log("Changing to ", value);

process.on('SIGINT', function () {

which makes the LED match what PA16 is. If you connect PA16 to e.g. PA6, you can change PA6, which is detected by PA16 which is reflected in the LED change on PA7!



The 2 on-board LED’s can be controlled via

echo [0|1] > /sys/class/leds/[red|green]_led/brightness

which map to PA17 (red) and PL10 (green), but those are (of course) claimed by the LED driver.


Not figured out yet.


Dec 182016

Letzte Woche gab’s Vanillekipferl. Hier das Rezept (http://www.chefkoch.de/rezepte/403911129675192/Uromas-Vanillekipferl.html). Auf die Hälfte reduziert:

  • 150g Mehl
  • 125g Butter
  • 60g Mandeln, gemahlen
  • 50g Zucker
  • Vanillezucker/Vanillearoma
  • Zum Wenden: Puderzucker, Vanillezucker

Alles mischen, in Folie gewickelt 30min in den Kühlschrank stellen. In der Zwischenzeit Puderzucker und Vanillezucker mischen und in eine verschliessbare Schüssel geben (zum wiederverwenden des Zuckers).

Kipferlteig in eine Rolle mit etwa 4cm Durchmesser rollen. 1cm Scheiben abschneiden. Daraus die Kipferl formen und recht eng aneinander auf ein mit Backpapier belegtem Blech legen. Bei 175°C ca. 20 Minuten backen.

Kurz (3 min) kühlen lassen und dann in die Puderzucker-Zucker Mischung wenden. Abkühlen lassen und in eine Dose verschliessen.


Dec 032016

Since I keep forgetting what pins I used for the ‘595 ICs which drive my 16 segment LED clock, here the pinout:

1: Vcc   2: GND
3: /SCK  4: NC
5: SER   6: NC
7: /RCK  8: NC
9:      10: NC

Since it’s HC595, Vcc is 2 up to 6V

And while on this topic, the 16 segment has this order when sending a 16 bit stream:


Bit 0 is t, bit 15 is a.

Nov 192016

I was looking for a recipe of something my grandma made. She called it “Gebackenes Zeug” which unfortunately is not a proper name. it contained Quark and mashed potatoes, but that’s all I knew. But today I found something which is at least similar if not identical except for the shape: Sächsische Quarkkeulchen. Here the recipe:

Zutaten für die Quarkkeulchen
500g Pellkartoffeln
500g abgetropfter Magerquark
150g Mehl
40g Zucker
70g Rumrosinen
Mark von ½ Vanilleschote
1 Eigelb
Abrieb von ½ Zitrone
Butterschmalz zum Braten
Puderzucker zum Bestäuben

Zubereitung der sächsischen Quarkkeulchen
1.    Die Kartoffeln weichkochen, pellen und durch die Kartoffelpresse drücken. Mit den restlichen Zutaten zu einem Teig vermengen.
2.    Aus dem Teig gleichgroße Quarkkeulchen formen und in heißem Butterschmalz beidseitig goldgelb ausbraten. Kurz vorm Servieren alle Quarkkeulchen mit etwas Puderzucker bestreuen.

Taken from here. Being in Japan means: no Quark, so I used Philadelphia cream cheese. I also added ascorbic acid (AKA Vitamin C) to make it a bit sour. No rainsins as I had none. No egg as cream cheese has enough fat. No lemon since I had none. And I had to use normal butter instead of clarified butter. No powdered sugar either.

So I had to change it a lot and I only used 3 potatoes and about 150g cream cheese. Turned out to be tasty though and it’s definitely similar to “Gebackenes Zeug”.

Will try again. Will make pictures then.

Nov 192016
Orange Pi Zero - Neat

Got myself a (actually two) Orange Pi Zero: US$7, quad core ARM Cortex A7@1.2GHz, 256MB RAM, WLAN, FastEthernet, 1 USB, USB-to-go for power. All in a (about) 5×5 cm² package. Add in a 8GB microSD card, and it’s a small capable little board.

While the memory looks on the small side, itś plenty to run one program. Armbian uses about 40MB itself when running:

harald@opz1:~$ free 
              total        used        free      shared  buff/cache   available 
Mem:         247012       36720      142740        2168       67552      193175 
Swap:        131068           0      131068

leaving >200MB left. Given that I can have a (small, limited) web server runnig on an ESP8266 with about 40KB RAM, 256MB is plenty for a single-purpose server/controller.

Ethernet works (as expected). WLAN works via simple nmtui command.

Controlling the 2 LED’s is easy too:

root@opz1:/# cd /sys/class/leds/red_led 
root@opz1:/sys/class/leds/red_led# ls 
brightness  device  max_brightness  power  subsystem  trigger  uevent 
root@opz1:/sys/class/leds/red_led# echo 1 >brightness  

That turns on the red LED. Similar for the green LED.

To find out what GPIO’s exist, use this:

root@opz1:/sys/class/leds/red_led# cat /sys/kernel/debug/gpio                                              
GPIOs 0-383, platform/sunxi-pinctrl, sunxi-pinctrl: 
 gpio-10  (?                   ) out hi 
 gpio-17  (red_led             ) out hi 
 gpio-202 (xradio_irq          ) in  lo 
 gpio-354 (?                   ) out hi 
 gpio-362 (green_led           ) out hi

To export a GPIO, do

# echo 15 >/sys/class/gpio/export

and then in /sys/class/gpio/gpio15/ you can see the standard Linux kernel GPIO things like direction (in/out) and value (0/1)

See also the schematics which show what port of the H2+ connects to what thing: orange-pi-zero-schanetics-v1_11 (sp!)


Nov 062016

During today’s Dev Japan Meetup I finally had the time to do something I always wanted to do but never had time to implement: Create a fast link from an end-device (AKA browser or phone application) to my LED display. That display is a simple 10×10 WS2812 LEDs strip originally connected to an Arduino with a Bluetooth receiver, but replaced by a Wemos D1 mini flashed with Espruino since that has WiFi and more RAM.

Here the important (but incomplete) part of the Espruino program:

var host = "the_ws_server.co.jp";
var WebSocket = require("ws");
var ws = new WebSocket(host,{
  path: '/',
  port: 8080, // default is 80
  protocol : "echo-protocol", // websocket protocol name (default is none)
  protocolVersion: 13, // websocket protocol version, default is 13
  origin: 'Espruino',
  keepAlive: 60

ws.on('open', function() {
  console.log("Connected to server");

ws.on('message', function(msg) {
  console.log("MSG: " + msg);
  if (msg == "R") {
    colorize(40, 10, 10);
  } else if (msg == "G") {
    colorize(10, 40, 10);
  } else if (msg == "B") {
    colorize(10, 10, 40);
  esp8266.neopixelWrite(NodeMCU.D4, leds);

The logic is as simple as it looks: connect to a WS server and wait for incoming messages. If it’s “R”, or “G”, or “B”, then colorize the LED array.

Here a section of the browser part:

var ws = new WebSocket("ws://the_ws_server.co.jp:8080/");

ws.onopen = function(evt) {
  var conn_status = document.getElementById('conn_text');

ws.onmessage = function(evt) {
  var newMessage = document.createElement('p');
  newMessage.textContent = "Server: " + evt.data;

ws.onclose = function(evt) {
  alert ("Connection closed");

$(".color").click(function(evt) {

and the buttons look like

<button type="submit" class="color" val="R">Red</button>
<button type="submit" class="color" val="G">Green</button>
<button type="submit" class="color" val="B">Blue</button>

The one missing part is the websocket server in the middle which relays messages, which I took quite literally from here from the Espruino Websocket docs.

This is anything but clean code, and not yet a complete and instructive example application, but it’s the first step and a good proof-of-concept.

Next step is a web page to have a 10×10 grid of buttons which can be turned on/off by touching, and the corresponding commands are sent to the LED display.