Logging with Docker

In a typical Docker environment you’ll have plenty of containers (probably in multiple networks?) on the same machine. Let’s assume, you need to debug some problems of a container, eg. because it doesn’t send mails anymore.. What would you do? Correct, you’d go and check the logs.

By default, Docker logs the messages of every container into a json file. On a Debian-based system you’ll probably find the file at /var/lib/docker/containers/CONTAINERID/CONTAINERID-json.log. However, to properly look into the logs you would use Docker’s logs tool. This will print the logs, just as you would expect cat to dump the logs in /var/log. docker-logs can also filter for time spans using --since and --until, and it is able to emulate a tail -f with --follow.

However, the logs are only available for exsiting containers. That means, if you recreate the application (i.e. you recreate the container), you’ll typically loose the log history… If your workflow includes the --rm, you will immediately trash the log of a container when it’s stopped. Fortunatelly, Docker provides other logging drivers, to e.g. log to AWS, fluentd, GPC, and to good old syslog! :)

Here I’ll show how to use the host’s syslog to manage the logs of your containers.

Log to Syslog

Telling Docker to log to the host’s syslog is really easy. You just need to use the built-in syslog driver:

docker run --log-driver syslog [other options etc]

Voilà, the container will log to the syslog and you’ll probably find the messages in /var/log/syslog. Here is an example of an Nginx, that I just started to serve my blog on my laptop:

Feb 21 16:06:32 freibeuter af6dcace59a9[5606]: - - [21/Feb/2018:15:06:32 +0000] "GET /2018/02/21/logging-with-docker/ HTTP/1.1" 304 13333 "http://localhost:81/" "Mozilla/5.0 (X11; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0" "-"

By default, the syslog driver uses the container’s ID as the syslog tag (here it is af6dcace59a9), but you can further configure the logging driver and, for example, set a proper syslog tag:

docker run --log-driver syslog --log-opt tag=binfalse-blog [other options etc]

This way, it is easier to distinguish between messages from different containers and to track the logs of an application even if the container gets recreated:

Feb 21 16:11:16 freibeuter binfalse-blog[5606]: - - [21/Feb/2018:15:11:16 +0000] "GET /2018/02/21/logging-with-docker/ HTTP/1.1" 200 13333 "http://localhost:81/" "Mozilla/5.0 (X11; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0" "-"

If you’re using Docker Compose, you can use the logging keyword to configure logging:

version: '2'
      restart: unless-stopped
      image: nginx
      container_name: website
        - /srv/web/default/:/usr/share/nginx/html
        driver: syslog
          tag: docker/website

Here, I configured an nxinx that just serves the contents from /srv/web/default. The interesting part is, however, that the container uses the syslog driver and the syslog tag docker/website. I always prefix the tag with docker/, to distinguish between log entries of the host machine and entries from Docker containers..

Store Docker logs seperately

The workaround so far will probably substantially spam your /var/log/syslog, which may become very annoying… ;-)

Therefore, I recommend to write Docker’s logs to a seperate file. If you’re for example using Rsyslog, you may want to add the following configuration:

if $syslogtag contains 'docker/' then /var/log/docker
& ~

Just dump the snippet to a new file /etc/rsyslog.d/docker.conf and restart Rsyslog. This rule tells Rsyslog to write messages that are tagged with docker/* to /var/log/docker, and not to the default syslog file anymore. Thus, your /var/log/syslog stays clean and it’s easier do monitor the Docker containers.

Disentangle the Container logs

Since version 8.25, Rsyslog can also be used to split the docker logs into individual files based on the tag. So you can create separate log files, one per container, which is even cleaner! The idea is to use the tag name of containers to implement the desired directory structure. That means, I would tag the webserver of a website with docker/website/webserver and the database with docker/website/database. We can then tell Rsyslog to allow slashes in program names (see the programname section at www.rsyslog.com/doc/master/configuration/properties.html) and create a template target path for Docker log messages, which is based on the programname:


$template DOCKER_TEMPLATE,"/var/log/%programname%.log"

if $syslogtag contains 'docker/' then ?DOCKER_TEMPLATE

Using that configuration, our website will log to /var/log/docker/website/webserver.log and /var/log/docker/website/database.log. Neat, isn’t it? :)

Inform Logrotate

Even though all the individual logfiles will be smaller than a combined one, they will still grow in size. So we should tell logrotate of their existence!

Fortunatelly, this is easy as well. Just create a new file /etc/logrotate.d/docker containing something like the following:

        rotate 7
                invoke-rc.d rsyslog rotate > /dev/null

This will rotate the files ending in *.log in /var/log/docker/ and its subdirectories everyday and keep compressed logs for 7 days. Here I’m using a maximum depth of 3 subdirectories – if you need to create a deeper hierarchy of directories just add another /var/log/docker/*/*/*/*.log etc to the beginning of the file.

Dockerising a Contao website II

In a previous post I explained how to run a Contao website in a Docker infrastructure. That was a good opening. However, after running that setup for some time I discovered a few issues…

A central idea of Docker is to install the application in an image and mount persistent files into a running container. Thus, you can just throw away an instance of the app and start a new one very quickly (e.g. with an updated version of the app). Unfortunately, using Contao it’s not that straight-forward – at least when using the image decribed earlier.

Here I’m describing how I fought the issues:

Issues with Cron

The first issue was Contao’s Poor-Man-Cron. This cron works as follows:

  • The browser requests a file cron.txt, which is supposed to contain the timestamp of the last cron run.
  • If the timestamp is “too” old, the browser will also request a cron.php, which then runs overdue jobs.
  • If a job was run, the timestamp in cron.txt will be updated, so cron.php won’t be run every time.

Good, but that means the cron.txt will only be written, if a cron job gets executed. But let’s assume the next job will only be run next week end!? The last cron-run-time is stored in the database, but the cron.txt won’t exist by default. That means, even if the cron.php is run, it will know that there is no cron job to execute and, therefore, exit without creating/updating the cron.txt. Especially when using Docker you will hit such a scenario every time when starting a new container.. Thus, every user creates a 404 error (as there is no cron.txt), which is of course ugly and spams the logs..

I fixed the issue by extending the Contao source code. The patch is already merged into the official release of Contao 3.5.33. In addition, I’m initialising the cron.txt in my Docker image with a time stamp of 0, see the Dockerfile.

Issues with Proxies

A typical Docker infrastructure (at least for me) consists of bunch containers orchestrated in various networks etc.. Usually, you’ll have at least one (reverse) proxy, which distributes HTTP request to the container in charge. However, I experienced a few issues with my proxy setup:


While the connection between client (user, web browser) and reverse proxy is SSL-encrypted, the proxy and the webserver talk plain HTTP. As it’s the same machine, there is no big need to waste time on encryption. But Contao has a problem with that setup. Even though, the reverse proxy properly sends the HTTP_X_FORWARDED_PROTO, Contao only sees incomming HTTP traffic and uses http://-URLs in all documents… Even if you ignore the mixed-content issue and/or implement a rewrite of HTTP to HTTPS at the web-server-layer, this will produce twice as much connections as necessary!

The solution is however not that difficult. Contao does not understand HTTP_X_FORWARDED_PROTO, but it recognises the $_SERVER['HTTPS'] variable. Thus, to fix that issue you just need to add the following to your system/config/initconfig.php (see also Issue 7542):

	$_SERVER['HTTPS'] = 1;

In addition, this will generate URLs including the port number (e.g. https://example.com:443/etc), but they are perfectly valid. (Not like https://example.com:80/etc or something that I saw during my tests… ;-)

URL encodings in the Sitemap

The previous fix brought up just another issue: The URL encoding in the sitemap breaks when using the port component (:443).. Conato uses rawurlencode to encode all URLs before writing them to the sitemap. However, rawurlencode encodes quite a lot! Among others, it converts :s to %3A. Thus, all URLs in my sitemap looked like this: https://example.com%3A443/etc - which is obviously invalid.

I proposed using htmlspecialchars instead to encode the URLs, but it was finally fixed by splitting the URLs and should be working in release 3.5.34.

Issues with Cache and Assets etc

A more delicate issue are cache and assets and sitemaps etc. Contao’s backend comes with convenient buttons to clear/regenerate these files and to create the search index. Yet, you don’t always want to login to the backend when recreating the Docker container.. Sometime you simply can’t - for example, if the container needs to be recreated over night.

Basically, that is not a big issue. Assets and cache will be regenerate once they are needed. But the sitemaps, for instance, will only be generated when interacting with the backend.

Thus, we need a solution to create these files as soon as possible, preferably in the background after a container is created. Most of the stuff can be done using the Automator tool, but I also have some personal scripts developed by a company, that require other mechanisms and are unfortunately not properly integrated into Contao’s hooks landscape. And if we need to touch code anyways, we can also generate all assets and rebuild the search index manually (precreating necessary assets will later on speed up things for users…). To generate all assets (images and scripts etc), we just need to access every single page at the frontend. This will then trigger Contao to create the assets and cache, and subsequent requests from real-life users will be much faster!

The best hack that I came up with so far looks like the following script, that I uploaded to /files/initialiser.php to Contao instance:

define ('TL_MODE', 'FE');
require __DIR__ . '/../system/initialize.php';

$THISDIR = realpath (dirname (__FILE__));

$auto = new \Automator ();
// purge stuff
$auto->purgeSearchTables ();
$auto->purgeImageCache ();
$auto->purgeXmlFiles ();

// regenerate stuff
$auto->generateXmlFiles ();

// get all fe pages
$pages = \Backend::findSearchablePages();

if (isset($GLOBALS['TL_HOOKS']['getSearchablePages']) && is_array($GLOBALS['TL_HOOKS']['getSearchablePages'])) {
	foreach ($GLOBALS['TL_HOOKS']['getSearchablePages'] as $callback) {
		$classname = $callback[0];
		if (!is_subclass_of ($classname, 'Backend'))
			$pages =  (new $classname ())->{$callback[1]} ($pages);

// request every fe page to generate assets and cache and search index
curl_setopt($ch, CURLOPT_USERAGENT, 'conato-cleaner');
# maybe useful to speed up:
#curl_setopt($ch, CURLOPT_MAXCONNECTS, 50);
#curl_setopt($ch, CURLOPT_NOBODY, TRUE);
#curl_setopt($ch, CURLOPT_TIMEOUT_MS, 150);
#curl_setopt($ch, CURLOPT_CONNECTTIMEOUT_MS, 150);

foreach ($pages as $page) {
	curl_setopt($ch, CURLOPT_URL, $page);

The first 3 lines initialise the Contao environment. Here I assume that ../system/initialize.php exists (i.e. the script is saved in the files directory). The next few lines purge existing cache using the Automator tool and subsequently regenerate the cache – just to be clean ;-)

Finally, the script (i) collects all “searchable pages” using the Backend::findSearchablePages() functionality, (ii) enriches this set of pages with additional pages that may be hooked-in by plugins etc through $GLOBALS['TL_HOOKS']['getSearchablePages'], and then (iii) uses cURL to iteratively request each page.


The first part should be reasonably fast, so clients may be willing to wait until the cache stuff is recreated. Accessing every frontend page, however, may require a significant amount of time! Especially for larger web pages.. Thus, I embedded everything in the following skeleton, which advises the browser to close the connection before we start the time-consuming tasks:

* start capturing output
ob_end_clean ();
ignore_user_abort ();
ob_start() ;

* run the tasks that you want your users to wait for

// e.g. purge and regenerate cache/sitemaps/assets
$auto = new \Automator ();
$auto->purgeSearchTables ();
// ..

* flush the output and tell the browser to close the connection as soon as it received the content
$size = ob_get_length ();
header ("Connection: close");
header ("Content-Length: $size");
ob_end_flush ();
flush ();

* from here you have some free computational time

// e.g. collect pages and request the web sites
// users will already be gone and the output will (probably) never show up in a browser.. (but don't rely on that! it's still sent to the client, it's just outside of content-length)
$pages = \Backend::findSearchablePages();
// ...

Here, the browser is told to close the connection after a certain content size arrived. I buffer the content that I want to transfer using ob_start and ob_end_flush, so I know how big it is (using ob_get_length). Everything after ob_get_length can safely be ignored by the client, and the connection can be closed.
(You cannot be sure that the browser really closes the connection. I saw curl doing it, but also some versions of Firefox still waiting for the script to finish… Nevertheless, the important content will be transferred quick enough).

In addition, I created some RewriteRules for mod_rewrite to automatically regenerate missing files. For example, for the sitemaps I added the following to the vhost config (or htaccess):

RewriteEngine On

RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^/share/(.*)\.xml.*$ https://example.com/files/initialiser.php?target=sitemap&sitemap=$1 [R=302,L]

That means, if for example /share/sitemap.xml not yet exists, the user gets automagically redirected to our initialiser.php script! In addition, I added some request parameters (?target=sitemap&sitemap=$1), so that the initialiser.php knows which file was requested. It can then regenerate everything and immediately output the new content! :)

For example, my snippet to regenerate and serve the sitemap looks similar to this:

// ...

$auto = new \Automator ();
// ...
$auto->generateXmlFiles ();

if ($_GET['target'] == 'sitemap') {
	$sitemaps = $auto->purgeXmlFiles (true);
	$found = false;
	foreach ($sitemaps as $sitemap) {
		if ((!isset ($_GET['sitemap']) || empty ($_GET['sitemap'])) || $_GET['sitemap'] == $sitemap) {
			$xmlfile = $THISDIR . "/../share/" . $sitemap . ".xml";
			// if it still does not exists -> we failed...
			if (!file_exists( $xmlfile )) {
				// error handling
			// otherwise, we'll dump the sitemap
			else {
				header ("Content-Type: application/xml");
				readfile ($xmlfile);
			$found = true;
	if (!$found) {
		// error handling

Thus, the request to /share/somesitemap.xml will never fail. If the file does not exist, the client will be redirected to /files/initialiser.php?target=sitemap&sitemap=somesitemap, the file /share/somesitemap.xml will be regenerated, and the new contents will immediately be served. So the client will eventually get the desired content :)

Please be aware, that this script is easily DOS-able! Attackers may produce a lot of load by accessing the file. Thus, I added some simple DOS protection to the beginning of the script, which makes sure the whole script is not run more than once per hour (3600 seconds):

$dryrun = false;
$runcheck = "/tmp/.conato-cleaner-timestamp";

if (file_exists ($runcheck) && filemtime ($runcheck) > time () - 3600) {
	$dryrun = true;
    if (!isset ($_GET['target']) || empty ($_GET['target']))
        die ();
	touch ($runcheck);

If $dryrun is true, it won’t regenerate cache etc, but still serve the sitemap and other files if requested.. However, if there is also no $_GET['target'] defined, we don’t know what to serve anyway and can die immediately…

You could include the script at the footer of your webpage, e.g. using

<script src="/files/initialiser.php"></script>

(you may want to make sure that the generated output, if any, is valid JavaScript. E.g. embed everything in /*...*/ or something…)

This way you would make sure, that every request produces a fully initialised system. However, this will probably also create unnecessary load every hour… You could increase the time span in the DOS-protection-hack, but I guess it should be sufficient to run the script only if a missing file is requested. Earlier requests then need to wait for pending assets etc, but to be honest, that should not be too long (or you have a different problem anyway…).

And if your website provides an RSS feed, you could subscribe to it using your default reader, which will regularly make sure that the RSS feed is generated if missing.. (and thus trigger all the other stuff in our initialiser.php) – A feed reader as the poorest-man-cron ;-)


As I said earlier, my version of the script contains plenty of personalised stuff. That’s why I cannot easily share it with you.. :(

However, if you have trouble implementing it yourself just let me know :)

Dockerising a Contao website

I’m a fan of containerisation! It feels much cleaner and systems don’t age that quickly.

Latest project that I am supposed to maintain is a new Contao website. The company who built the website of course just delivered files and a database. The files contain the Contao installation next to Contao extensions next to configuration and customised themes.. All merged into a blob… Thus, in the files it is hard to distinguish between Contao-based files and user generated content. So I needed to study Contao’s documentation and reinstall the website to learn what files should go into the Docker image and which files to store outside.

However, I finally came up with a solution that is based on two Contao images :)

A general Contao image

The general Contao image is supposed to contain a plain Conato installation. That is, the recipe just installs dependencies (such as curl, zip, and ssmtp) and downloads and extracts Contao’s sources. The Dockerfile looks like this:

FROM php:apache
MAINTAINER martin scharm <https://binfalse.de/contact/>

# for mail configuration see https://binfalse.de/2016/11/25/mail-support-for-docker-s-php-fpm/

RUN apt-get update \
 && apt-get install -y -q --no-install-recommends \
    wget \
    curl \
    unzip \
    zlib1g-dev \
    libpng-dev \
    libjpeg62-turbo \
    libjpeg62-turbo-dev \
    libcurl4-openssl-dev \
    libfreetype6-dev \
    libmcrypt-dev \
    libxml2-dev \
    ssmtp \
 && apt-get clean \
 && rm -r /var/lib/apt/lists/*

RUN wget https://download.contao.org/3.5/zip -O /tmp/contao.zip \
 && unzip /tmp/contao.zip -d /var/www/ \
 && rm -rf /var/www/html /tmp/contao.zip \
 && ln -s /var/www/contao* /var/www/html \
 && echo 0 > /var/www/html/system/cron/cron.txt \
 && chown -R www-data: /var/www/contao* \
 && a2enmod rewrite

RUN docker-php-source extract \
 && docker-php-ext-configure gd --with-freetype-dir=/usr/include/ --with-jpeg-dir=/usr/include/ \
 && docker-php-ext-install -j$(nproc) zip gd curl mysqli soap \
 && docker-php-source delete

RUN php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');" \
 && php -r "if (hash_file('SHA384', 'composer-setup.php') === '544e09ee996cdf60ece3804abc52599c22b1f40f4323403c44d44fdfdd586475ca9813a858088ffbc1f233e9b180f061') { echo 'Installer verified'; } else { echo 'Installer corrupt'; unlink('composer-setup.php'); } echo PHP_EOL;" \
 && mkdir -p composer/packages \
 && php composer-setup.php --install-dir=composer \
 && php -r "unlink('composer-setup.php');" \
 && chown -R www-data: composer

The first block apt-get installs necessary stuff from the Debian repositories. The second block downloads a Contao 3.5 from https://download.contao.org/3.5/zip, extracts it to /var/www/, and links /var/www/html to it. It also creates the cron.txt (see github.com/contao/core/pull/8838). The third block installs a few required and/or useful PHP extensions. And finally the fourth block retrieves and installs Composer to /var/www/html/composer, where the Contao-composer-plugin expects it.

That’s already it! We have a recipe to create a general Docker image for Contao. Quickly setup an automatic build and .. thada .. available as binfalse/contao.

A personalised Contao image

Besides the plain Contao installation, a Contao website typically also contains a number of extensions. Those are installed through composer, and they can always be reinstalled. As we do not want to install a load of plugins everytime a new container is started we create a personalised Contao image. All you need is the composer.json that contains the information on which extensions and which versions to install. This json should be copied to /var/www/html/composer/composer.json, before composer can be run to install the stuff. Here is an example of such a Dockerfile:

FROM binfalse/contao
MAINTAINER martin scharm <https://binfalse.de/contact/>

COPY composer.json composer/composer.json

USER www-data

# we need to run it this twice... you probably know the error:
# 'Warning: Contao core 3.5.31 was about to get installed but 3.5.31 has been found in project root, to recover from this problem please restart the operation'
# not sure why it doesn't run the necessary things itself? seems idiot to me, but... yes.. we run it twice if it fails...

RUN php composer/composer.phar --working-dir=composer update || php composer/composer.phar --working-dir=composer update

USER root

This image can then be build using:

docker build -t contao-personalised .

The resulting image tagged contao-personalised will contain all extensions required for your website. Thus, it is highly project specific and shouldn’t be shared..

How to use the personalised Contao image

The usage is basically very simple. You just need to mount a few things inside the container:

  • /var/www/html/files/ should contain files that you uploaded etc.
  • /var/www/html/templates/ may contain your customised layout.
  • /var/www/html/system/config/FILE.php should contain some configuration files. This may include the localconfig.php or a pathconfig.php.

Optionally you can link a MariaDB for the database.

Tying it all together using Docker-Compose

Probably the best way to orchestrate the containers is using Docker-Compose. Here is an example docker-compose.yml:

version: '2'

      build: /path/to/personalised/Dockerfile
      restart: unless-stopped
      container_name: contao
        - contao_db
        - "8080:80"
        - $PATH/files:/var/www/html/files
        - $PATH/templates:/var/www/html/templates:ro
        - $PATH/system/config/localconfig.php:/var/www/html/system/config/localconfig.php

      image: mariadb
      restart: always
      container_name: contao_db
        MYSQL_DATABASE: contao_database
        MYSQL_USER: contao_user
        MYSQL_PASSWORD: contao_password
        MYSQL_ROOT_PASSWORD: very_secret
        - $PATH/database:/var/lib/mysql

This assumes that your personalised Dockerfile is located in path/to/personalised/Dockerfile and your website files are stored in $PATH/files, $PATH/templates, and $PATH/system/config/localconfig.php. Docker-Compose will then build the personalised image (if necessary) and create 2 containers:

  • contao based on this image, all user-based files are mounted into the proper locations
  • contao_db a MariaDB to provide a MySQL server

To make Contao speak to the MariaDB server you need to configure the database connection in $PATH/system/config/localconfig.php just like:

$GLOBALS['TL_CONFIG']['dbDriver'] = 'MySQLi';
$GLOBALS['TL_CONFIG']['dbHost'] = 'contao_db';
$GLOBALS['TL_CONFIG']['dbUser'] = 'contao_user';
$GLOBALS['TL_CONFIG']['dbPass'] = 'contao_password';
$GLOBALS['TL_CONFIG']['dbDatabase'] = 'contao_database';
$GLOBALS['TL_CONFIG']['dbPconnect'] = false;
$GLOBALS['TL_CONFIG']['dbCharset'] = 'UTF8';
$GLOBALS['TL_CONFIG']['dbPort'] = 3306;
$GLOBALS['TL_CONFIG']['dbSocket'] = '';

Here, the database should be accessible at contao_db:3306, as it is setup in the compose file above.

If you’re running contao with “Rewrite URLs” using an .htaccess you also need to update Apache’s configuration to allow for rewrites. Thus, you may for example mount the follwoing file to /etc/apache2/sites-available/000-default.conf:

<VirtualHost *:80>
    ServerAdmin webmaster@localhost
    DocumentRoot /var/www/html
    <Directory /var/www/>
        AllowOverride All
        Options FollowSymLinks
    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined

This tells Apache to allow everything in any .htaccess file in /var/www.

When everything is up running the Conato install will be available at port 8080 (see ports definition in the compose file) of the machine hosting the Docker containers.

Mail support

The image above comes with sSMTP installed. If you need support for email with your Contao installation, you just need to mount two more files into the container:

Tell PHP to mail through sSMTP

The following file tells PHP to use the ssmtp binary for mailing. Just mount the file to /usr/local/etc/php/conf.d/mail.ini:

[mail function]
sendmail_path = "/usr/sbin/ssmtp -t"

Configure sSMTP

The sSMTP configuration is very easy. The following few lines may already be sufficient, when mounted to /etc/ssmtp/ssmtp.conf:


For more information read Mail support for Docker’s php:fpm and the Arch Linux wiki on sSMTP or the Debian wiki on sSMTP.

Archiving a (Wordpress) Website

I needed to migrate a lot of tools and projects that we’ve been working on in the SEMS group at the University of Rostock. Among others, the Wordpress website needed to be serialised to get rid of PHP and all the potential insecure and expensive Wordpress maintenance. I decided to mirror the page using HTTrack and some subsequent fine tuning. This is just a small report, maybe interesting if you also need to archive a dynamic web page.

Prepare the page

Some stuff in your (Wordpress) installation are properly useless after serialisation (or have never been working either) - get rid of them. For example:

  • Remove the search box - it’s useless without PHP. You may add a link to a search engine instead…?
  • Remove unnecessary trackers like Google analytics and Piwik. You probably don’t need it anymore and users may be unnecessarily annoyed by tracking and/or 404s.
  • Disable unnecessary plugins.
  • Check that manual links (e.g. in widgets) are still up-to-date, also after archiving..
  • Check for unpublished drafts in posts/pages. Those will be lost as soon as you close the CMS.
  • Recreate sitemap and rss feeds (if not created automatically)

I also recommend to setup some monitoring, e.g. using check_link, to make sure all resources are afterwards accessible as expected!

Mirror the website

I decided to mirror the web content using HTTrack. That’s basically quite simple. At the target location you only need to call:

httrack --mirror https://sems.uni-rostock.de/

This will create a directory sems.uni-rostock.de containing the mirrored contend. In addition you’ll find logs in hts-log.txt and the cached content in hts-cache/.

However, I tweaked the call a bit and actually executed HTTrack like this:

httrack --mirror '-*trac/*' '-*comments/feed*' '-*page_id=*' -%k --disable-security-limits -%c160 -c20  https://sems.uni-rostock.de/

This ignores all links that match *trac/* (there was a Trac running, but that moved to GitHub and an Nginx will permanently redirect the traffic), in addition it will keep connections alive (-%k). As I’m the admin of the original site (which I know won’t die too soon, and in worst case I can just restart it) I increased the speed to a max of 160 connections per second (-%c160) and max 20 simultaneous connections (-c20). For that I also needed to disable HTTrack’s security limits (--disable-security-limits).

That went quite well and I quickly had a copy of the website. However, there were a few issues…

Problems with redirects.

Turns out that HTTrack has problems with redirects. At some point we installed proper SSL certificates and since then we were redirecting traffic at port 80 (HTTP) to port 443 (HTTPS). However, some people manually created links that point to the HTTP resources, such as http://sems.uni-rostock.de/home/. If HTTrack stumbles upon such a redirect it will try to remodel that redirect. However, in case of redirects from http://sems.uni-rostock.de/home/ to https://sems.uni-rostock.de/home/, the target is the same as the source (from HTTrack’s point of view) and it will redirect to … itself.. -.-

The created HTML page sems.uni-rostock.de/home/index.html looks like that:

<!-- Created by HTTrack Website Copier/3.49-2 [XR&CO'2014] -->

<!-- Mirrored from sems.uni-rostock.de/home/ by HTTrack Website Copier/3.x [XR&CO'2014], Wed, 24 Jan 2018 07:16:38 GMT -->
<!-- Added by HTTrack --><meta http-equiv="content-type" content="text/html;charset=iso-8859-1" /><!-- /Added by HTTrack -->
<META HTTP-EQUIV="Content-Type" CONTENT="text/html;charset=UTF-8"><META HTTP-EQUIV="Refresh" CONTENT="0; URL=index.html"><TITLE>Page has moved</TITLE>
<A HREF="index.html"><h3>Click here...</h3></A>
<!-- Created by HTTrack Website Copier/3.49-2 [XR&CO'2014] -->

<!-- Mirrored from sems.uni-rostock.de/home/ by HTTrack Website Copier/3.x [XR&CO'2014], Wed, 24 Jan 2018 07:16:38 GMT -->

As you can see, both the link and the meta refresh will redirect to the very same index.html, effectively producing a reload-loop… And as sems.uni-rostock.de/home/index.html already exists it won’t store the content behind https://sems.uni-rostock.de/home/, which will be lost…

I have no idea for an easy fix. I’ve been playing around with the url-hacks flag, but I did not find a working solution.. (see also forum.httrack.com/readmsg/10334/10251/index.html)

What I ended up with was to grep for this page and to find pages that link to it:

grep "Click here" -rn sems.uni-rostock.de | grep 'HREF="index.html"'

(Remember: some of the Click here pages are legit: They implement proper redirects! Only self-links to HREF="index.html" are the enemies.)

At SEMS we for example also had a wrong setting in the calendar plugin, which was still configured for a the HTTP version of the website and, thus, generating many of these problematic URLs.

The back-end search helped a lot to find the HTTP links. When searching for http://sems in posts and pages I found plenty of pages that hard-coded the wrong link target.. Also remember that links may also appear in post-excerpts!

If nothing helps, you can still temporarily disable the HTTPS redirect for the time of mirroring.. ;-)

Finalising the archive

To complete the mirror I also rsync‘ed the files in wp-content/uploads/, as not all files are linked in through the web site. Sometimes we just uploaded files and shared them through e-mails or on other websites.

I also manually grabbed the sitemap(s), as HTTrack apparently didn’t see them:

wget --quiet https://sems.uni-rostock.de/sitemap.xml -O sems.uni-rostock.de/sitemap.xml
wget --quiet https://sems.uni-rostock.de/sitemap.xml -O - | egrep -o "https?://[^<]+" | wget --directory-prefix=sems.uni-rostock.de -i -

iptables: log and drop

Linux has a sohpisticated firewall built right into the kernel: It’s called iptables! I’m pretty sure you heard about it. You can do realy crazy things with iptables. But here I just want to log how to log+drop a packet in a single rule.

Usually, you would probably do something like that:

iptables -A INPUT -j LOG --log-level warning --log-prefix "INPUT-DROP:"
iptables -A INPUT -j DROP

Works perfectly, but dramatically messes your rules table up.. Especially, if you want to log+drop packets that match a complicated filter. You’ll end up with twice as many table entries as desired..

The trick is to instead create a new rule chain that will log+drop in sequence:

iptables -N LOG_DROP

So here I created a new chain called LOG_DROP. We can now append (-A) two new rules to that chain, which do the actual drop+log:

iptables -A LOG_DROP -j LOG --log-level warning --log-prefix "INPUT-DROP:"
iptables -A LOG_DROP -j DROP

(similar like the first code above, just not for the INPUT chain but for the LOG_DROP chain)

That’s basically it! If you now need to log+drop a packet you can append a new rule to e.g. the INPUT chain that routes the packet to the LOG_DROP chain:

iptables -A INPUT [...filter specification...] -j LOG_DROP

You should consider to limit the number of redundant log entries per time to prevent flooding of your logs.. For more documentation you should consult the manual of iptables(8).

Common Name vs Subject Alternative Names

You probably heard about the conflict between the fields Common Name (CN) and Subject Alt Names (subjectAltName) in SSL certificates. It seems best practice for clients to compare the CN value with the server’s name. However, RFC 2818 already advised against using the Common Name and google now takes the gloves off. Since Chrome version 58 they do not support the CN anymore, but throw an error:

Subject Alternative Name Missing

Good potential for some administrative work ;-)

Check for a Subject Alternative Names

You can use OpenSSL to obtain a certificate, for example for binfalse.de:

openssl s_client -showcerts -connect binfalse.de:443 </dev/null 2>/dev/null

Here, openssl will connect to the server behind binfalse.de at port 443 (default port for HTTPS) to request the SSL certificate and dump it to your terminal. openssl can also print the details about a certificate. You just need to pipe the certificate into:

openssl x509 -text -noout

Thus, the whole command including the output may look like this:

openssl s_client -showcerts -connect binfalse.de:443 </dev/null | openssl x509 -text -noout
    Version: 3 (0x2)
    Serial Number:
  Signature Algorithm: sha256WithRSAEncryption
    Issuer: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
      Not Before: May 12 07:11:00 2017 GMT
      Not After : Aug 10 07:11:00 2017 GMT
    Subject: CN = binfalse.de
    Subject Public Key Info:
      Public Key Algorithm: rsaEncryption
        Public-Key: (4096 bit)
        Exponent: 65537 (0x10001)
    X509v3 extensions:
      X509v3 Key Usage: critical
        Digital Signature, Key Encipherment
      X509v3 Extended Key Usage: 
        TLS Web Server Authentication, TLS Web Client Authentication
      X509v3 Basic Constraints: critical
      X509v3 Subject Key Identifier: 
      X509v3 Authority Key Identifier: 
      Authority Information Access: 
        OCSP - URI:http://ocsp.int-x3.letsencrypt.org/
        CA Issuers - URI:http://cert.int-x3.letsencrypt.org/
      X509v3 Subject Alternative Name: 
      X509v3 Certificate Policies: 
          CPS: http://cps.letsencrypt.org
          User Notice:
            Explicit Text: This Certificate may only be relied upon by Relying Parties and only in accordance with the Certificate Policy found at https://letsencrypt.org/repository/
  Signature Algorithm: sha256WithRSAEncryption

As you can see in the X.509 extension this server’s SSL certificate does have a Subject Alternative Name:

X509v3 Subject Alternative Name:

To quick-check one of your websites you may want to use the following grep filter:

openssl s_client -showcerts -connect binfalse.de:443 </dev/null | openssl x509 -text -noout | grep -A 1 "Subject Alternative Name"

If that doesn’t print a proper Subject Alternative Name you should go and create a new SSL certificate for that server!

#android: No Internet Access Detected, won't automatically reconnect -aka- Connected, no Internet.

Android: No Internet Access Detected, won't automatically reconnect.
Android: No Internet Access Detected, won't automatically reconnect.

Hands up: who knows what an android device does when it sees a WiFi network coming up? Exactly, since Lollipo (Android 5) your phone or tablet leaks a quick HTTP request to check if it has internet access. This check is, for example, done with clients3.google.com/generate_204, a “webpage” that always returns an HTTP status code 204 No Content. Thus, if the phone receives a 204 it is connected to the internet, otherwise it assumes that this network does not provide proper internet access or is just a captive portal. However, that way Google of course always knows when you connect from where. And how often. And which device you’re using. etc… :(

How to prevent the leak

Even if people may like that feature, that is of course a privacy issue – so how can we counter that?

I briefly mentioned that a few years ago. You could use AdAway (available from F-Droid, source from GitHub) to redirect all traffic for clients3.google.com and clients.l.google.com to nirvana.

I already maintain a convenient configuration for AdAway at stuff.lesscomplex.org/adaway.txt, which blocks Google’s captive portal detection.

However, blocking that “feature” also comes with some drawbacks…

The downside of blocking captive portal detection

Android: Connected, no Internet.
Android: Connected, no Internet.

The consequences of blocking all request of the captive portal detection are obvious: your phone assumes that no network hat internet access. And therefore, it wouldn’t connect automatically, saying

No Internet Access Detected, won’t automatically reconnect. see image on top

That will probably increase your mobile data usage, as you always need (to remember) to do connect manually. And even if you manually connect to a network “without internet” the WiFi icon will get an exclamation mark and the phone says

Connected, no Internet. see second image


What can we do about it?

Disable captive portal detection

With a rooted phone you can simply disable captive portal detection. Just get a root-shell through adb (or SSH etc) to run the following command:

settings put global captive_portal_detection_enabled 0

One small drawback of that approach: you need to execute that again after flashing a new image… However, I guess you’ll anyway have a small workflow for re-flashing your phone – just add that tiny bit to it ;-)

Another drawback is that you loose the captive portal detection… Of course, that’s what you intended, but sometimes it may be useful to have that feature in hotels etc..

Change the server for captive portal detection with the Android API

You can also change the URL to the captive portal server to a server under your control. Let’s say you have a site running at scratch.binfalse.de/generate_204 that simulates a captive portal detection server backend(!?) and always returns 204, no matter what request. Then you can use that URL for captive portal detection! Override the captive portal server on a root-shell (adb or SSH etc) by calling:

settings put global captive_portal_server scratch.binfalse.de

This way you retain the captive portal detection without leaking data to Google. However, you will again loose the setting when flashing the phone again..

Change the server for captive portal detection using AdAway

Another option for changing the captive portal detection server is to change its IP address to one that’s under your control. You can do that with AdAway, for example. Let’s say your captive portal detection server has the IP address, then you may add the following to your AdAway configuration: clients3.google.com clients.l.google.com

The webserver at should then of course accept requests for the foreign domains.

This way, you also don’t leak the data to Google and you will also keep the settings after flashing the phone (as long as you leave AdAway installed). However, there are also some things to keep in mind: First, I could imagine that Google may be a bit upset if you redirect their domains to a different server? And second, you don’t know if those are the only servers used for captive portal detection. If Google at some point comes up with another domain for captive portal detection, such as captive.google.com, you’re screwed.

Supplementary material

See also the CaptivePortal description at the android reference.

Create captive portal detection server with Nginx

Just add the following to your Nginx configuration:

location /generate_204 { return 204; }

Create captive portal detection server with Apache

If you’re running an Apache web server you need to enable mod_rewrite, then create a .htaccess in the DocumentRoot containing:

<IfModule mod_rewrite.c>
	RewriteEngine On
	RewriteCond %{REQUEST_URI} /generate_204$
	RewriteRule $ / [R=204]

Create captive portal detection server with PHP

A simple PHP script will also do the trick:

<?php http_response_code (204); ?>

Docker MySQL Backup

Even with Docker you need to care about backups.. ;-)

As you usually mount all the persistent data into the container the files will actually be on your host. Thus, you can simply do the backup of these files. However, for MySQL/MariaDB I prefer having an actual SQL-dump. Therefore I just developed the Docker MySQL-Backup tool. You will find the sources at the corresponding GitHub repository.

How does Docker MySQL-Backup work?

The tool basically consists of two scripts:

The script /etc/cron.daily/docker-mysql-backup parses the output of the docker ps command to find running containers of the MySQL image. More precisely, it looks for containers of images that start with either \smysql or \smariadb. The actual filter command is

docker ps --format '{{.Names}}\t{{.Image}}'  | /bin/grep '\s\(mysql\|mariadb\)' | awk '{print $1}'

That of course only matches the original MySQL/MariaDB image names (if you have a good reason to derive an own version of that image please tell me!). For every matching $container the script will exec the following command:

docker exec "$container" \
	sh -c 'exec mysqldump --all-databases -uroot -p"$MYSQL_ROOT_PASSWORD"' \
	| ${GZIP} -9 > "${BACKUP_DIR}/${NOW}_complete.sql.gz"

With the following variables:

  • $BACKUP_DIR is a concatenation of $BACKUP_BASE (configured in /etc/default/docker-mysql-backup) and the container name,
  • $NOW is the current time stamp as date +"%Y-%m-%d_%H-%M".

Thus, the backups are compressed, organised in subdirectories of $BACKUP_BASE, and the SQL-dumps have a time stamp in their names. $BACKUP_BASE defaults to /srv/backup/mysql/, but can be configured in /etc/default/docker-mysql-backup.

Last but not least, the script also cleans the backups itself. It will keep the backups of the last 30 days and all backups of days that end with a 2. So you will keep the backups from the 2nd, the 12th, and the 22nd of every month.

As the script is stored in /etc/cron.daily/ the cron tool will execute the backup script on a daily basis.

Restore a dump

Restoring the dump is quite easy. Let’s assume your container’s name is $container and the dump to restore carries the time stamp $date. Then you just need to run:

docker exec "$container" -v "${BACKUP_BASE}/docker_${container}":/srv sh -c \
      'exec gunzip < /srv/${date}_complete.sql.gz | mysql -uroot -p"$MYSQL_ROOT_PASSWORD"'

This will mount the backup directory in /srv of the running container and then decompress and import the SQL-dump on the fly.


Manual installation through GitHub

Clone the Docker MySQL-Backup repository:

git clone https://github.com/binfalse/docker-mysql-backup.git

Copy the backup script to the cron.daily (most likely /etc/cron.daily/) directory on your system:

cp docker-mysql-backup/etc/cron.daily/docker-mysql-backup /etc/cron.daily/

Copy the configuration to /etc/default/:

cp docker-mysql-backup/etc/default/docker-mysql-backup /etc/default/

Installation from my Apt repository

If you’re running a Debian-based system you may want to use my apt-repository to install the Docker MySQL-Backup tool. In that case you just need to run

aptitude install bf-docker-mysql-backup

Afterwards, look into /etc/default/docker-mysql-backup for configuration options. This way, you’ll always stay up-to-date with bug fixes and new features :)

Automatically update Docker images

Automatically Update Docker Images
Automatically Update your Docker Images

Docker is cool. Jails tools into containers. That of course sounds clean and safe and beautiful etc. However, the tools are still buggy and subject to usual attacks, just as they were running on your main host! Thus, you still need to make sure your containers are up to date.

But how would you do that?

Approaches so far

docker-compose pull

On the one hand, let’s assume you’re using Docker Compose, then you can go to the directory containing the docker-compose.yml and call

docker-compose pull
docker-compose up -d --remove-orphans

However, this will just update the images used in that Docker Compose setup – all the other images on your system wouldn’t be updated. And you need to do that for all Docker Compose environments. And if you’re running 30 containers of the same image it would check 30 times for an update of that image – quite a waste or power and time..


On the other hand, you may use the dupdate tool, introduced earlier:

dupdate -s

It is able to go through all your images and update them, one after the other. That way, all the images on your system will be updated. However, dupdate doesn’t know about running containers. Thus, currently running tools and services won’t be restarted..

Better: Docker Auto-Update

Therefore, I just developed a tool called Docker Auto-Update that combines the benefits of both approaches. It first calls dupdate -s to update all your images and then iterates over a pre-defined list of Docker Compose environments to call a docker-compose up -d --remove-orphans.

The tool consists of three files:

  • /etc/cron.daily/docker-updater reads the configuration in /etc/default/docker-updater and does the regular update
  • /etc/default/docker-updater stores the configuration. You need to set the ENABLED variable to 1, otherwise the update tool won’t run.
  • /etc/docker-compose-auto-update.conf carries a list of Docker Compose environments. Add the paths to the docker-compose.yml files on your system, one per line

As it’s installed in /etc/cron.daily/, cron will take care of the job and update your images and containers on a daily basis. If your system is configured properly, cron will send an email to the systems administrator when it updates an image or restarts a container.

You see, no magic, but a very convenient workflow! :)



To install the Docker Auto-Update tool, you may clone the git repository at GitHub. Then,

  1. move the ./etc/cron.daily/docker-updater script to /etc/cron.daily/docker-updater
  2. move the ./etc/default/docker-updater config file to /etc/default/docker-updater
  3. update the setup in /etc/default/docker-updater – at least set ENABLED=1
  4. create a list of Docker Compose config files in /etc/docker-compose-auto-update.conf - one path to a docker-compose.yml per line.

Debian Package

If you’re using a Debian based system you may install the Docker-Tools through my apt-repository:

aptitude install bf-docker-tools

Afterwards, configure /etc/default/docker-updater and at least set ENABLED=1. This way, you’ll stay up-to-date with bug fixes etc.


The tool will update your images and containers automatically – very convenient but also dangerous! The new version of an image may break your tool or may require an updated configuration.

Therefore, I recommend to monitor your tools through Nagios/Icinga/check_mk or whatever. And study the mails generated by cron!

Rsync of ZFS data with a FreeBSD live system

Booting into FreeBSD
Booting into FreeBSD

Let’s assume you rendered your FreeBSD system unbootable.. Yeah, happens to the best, but how can you still copy the data stored on a ZFS to another machine? You probably just shouted RSYNC - but it’s not that easy.

You would need a FreeBSD live os (either on a USB pen drive or on a CD/DVD) and boot into that system. However, by default you do not have network, the ZPool is not mounted, there is no rsync and SSH is not running, and the live os is not writable, which brings another few issues…

This is a step-by-step how-to through all the obstacles. Just boot into your live os (get it from freebsd.org) and go on with the following…

Get Networking

By default your live system does not have networking setup correctly. Call ifconfig to see if the network interface is up. If it’s not you can bring it up using:

ifconfig em0 up

(assuming your inteface is called em0)

If it is up, you need to configure it. When you’re using a DHCP server you can just ask for an IP address using:

dhclient em0

Otherwise you need to configure the addresses manually:

ifconfig em0 inet netmask

Afterwards you should be able to ping other machines, such as


Mount the ZPool

Your ZPool won’t be mounted by default; you need to do it manually. To list all pools available on that machine just call:

zpool import

This searches through the devices in /dev to discover ZPools. You may specify a different directory with -d (see man page for zpool). To actually import and mount your ZPool you need to provide its name, for example:

zpool import -f -o altroot=/mnt zroot

This will import the ZPool zroot. Moreover, the argument -o altroot=/mnt will mount it to /mnt instead of / and the -f will mount it even if it may be in use by another system (here we’re sure it isn’t, aren’t we?).

Create some Writeable Directories

The next problem is, that you do not have permissions to write to /etc, which you need to e.g. create SSH host keys etc. However, that’s also not a big issue as we have the unionfs filesystem! :)

UnionFS will mount a directory as an overlay over another directory. Let’s assume you have some space in $SPACE (maybe in the ZPool that you just mounted or on another USB drive), then you can just create a few directories:

mkdir $SPACE/{etc,var,usr,tmp}

and mount it as unionfs to the root’s equivalents:

mount_unionfs $SPACE/etc /etc
mount_unionfs $SPACE/var /var
mount_unionfs $SPACE/usr /usr
mount_unionfs $SPACE/tmp /tmp

Now we can write to /etc, while the actual changes will be written to $SPACE/etc! Isn’t that a great invention?

Start the SSH service

Now that /etc is writable we can start caring about the SSH daemon. First, we need to configure it to allow root to login. Add the follwing line to the /etc/ssh/sshd_config:

PermitRootLogin yes

Then, we can start the ssh daemon using:

service sshd onestart

It will automatically create host keys and all the necessary things for a first start of SSH. If that was successful, port 22 should now be open:

# sockstat -4 -l
root     sshd       938   4  tcp4   *:22                  *:*
root     syslogd    542   7  udp4   *:514                 *:*

Set root Password

To be able to login you of course need to set a root password:

passwd root

Aftwerwards, you should be able to login through SSH from any other machine. Go ahaed and give it a try!

Install and Run rsync

Almost there, but the freeBSD live image doesn’t come with rsync installed. So we need to do it manually:

pkg install rsync

This will first tell us that not even pkg is installed, but answering the question with y it will automatically install itself. And as everything is mounted as UnionFS, the stuff will actually be installed to $SPACE/... instead of /. However, you should now be able to do the rsync job from where ever you want :)