Pages

Monday, September 28, 2015

How to create an IPv6-only consul cluster with docker

Why?

  • we're using docker to run consul (and registrator and our services) in, and IPv6 makes this easier (no NAT => better performance)
  • it's easier to maintain one stack
  • consul is known to give issues with NAT and docker (https://github.com/docker/docker/issues/8795)
  • IPv4 is legacy and obsolete ;-)
Consul 0.5.2 has some issues running such a setup, but if you're building consul from master (which includes some fixes (see https://github.com/hashicorp/consul/commits?author=42wim) it will work fine.

Issues to be aware of:

  • the IPv4 version of consul listens by default on private address ranges, when using IPv6 you'll be running on 'public' addresses. So be sure you're firewalling those from the internet.
  • If you're using consul recursive powers, you'll also need IPv6 dns recursors. (e.g. google's 2001:4860:4860::8888)
  • Not IPv6 related, but for extra stability, enable leave_on_terminate.
  • Also not Ipv6 related, but I've noticed that the default LAN settings for consul can be a bit too strict when running on vmware hosts. This patch increase the probetimeout to 2 seconds (instead of 500msec)



Consul extra configuration server and client

Extra settings below necessary for the consul server and client agent setup


Configuration:
{
        "recursor": "[2001:4860:4860::8888]",
        "leave_on_terminate": true,
        "client_addr": "::",
        "addresses": { "http": "::"}
}

Consul server setup

The consul server are running as a docker host mode container (which means, they share the same network namespace as the host).

The reason here is that we need a fixed IPv6 address for the servers because we're forwarding our dns requests to those servers. (ofcourse with some extra work we could make a script that dynamically update our dns forwards to the dynamic IP address).

Our server has multiple IPv6 addresses so we'll have to add a -advertise and -bind flag

consul agent -server -advertise 2001:db8::1 -bind 2001:db8::1 -bootstrap-expect 3 -retry-join [2001:db8::1]:8301 -retry-join [2001:db8::2]:8301 -retry-join [2001:db8::3]:8301

Using consul-docker as our consul docker container (for client and server)

Consul client setup 

You'll need to cherry-pick this PR into your local build: https://github.com/hashicorp/consul/pull/1219.
The IPv6 address in the docker container will be random and we want to bind to the IPv6 address.
This patch looks for the first 'public' IPv6 address and uses this address to advertise.

So we start the client with:

consul agent -bind :: -join consul.service.consul

Gotcha's here:
bind :: actually binds to IPv4 and IPv6 addresses in the container, but because we advertise the IPv6 address the IPv4 address won't be used.

Other software

Registrator

We also use registrator to register our services in consul. So every time a container starts or stops, registrator handles the consul service registration process.

Also for registrator some extra fixes are needed to have IPv6 support. (not yet merged, see https://github.com/gliderlabs/registrator/pull/229)

Because we're running consul on IPv6 this means registrator also needs to connect to the IPv6 address.

registrator consul://server1.node.consul:8500

Registrator then can register other services that are running on the docker host, like e.g elasticsearch.

Registrator-netfilter

Besides main registrator we also run registrator-netfilter which automatically firewalls the IPv6 services in the container. The containers are no longer NATted but directly accessible, so they need to be firewalled.

Docker

A /64 is allocated for docker and a /80 is given to each docker host, running with the switches

--ipv6=true --fixed-cidr-v6=2001:db8::/80

Elasticsearch

ES is also run ipv6 only, using registrator, registrator-netfilter and consul.
You can find the relevant commands to give to docker below:

docker run --net bridge -e SERVICE_NAME=es -e SERVICE_9200_TAGS=http-data 
-e SERVICE_9300_TAGS=transport-data -e SERVICE_9200_IPV6=tcp -e SERVICE_9300_IPV6=tcp 
-e ADVERTISE_IPV6=yes

Tuesday, February 10, 2015

tmux memory usage on linux



So a while ago I switched from screen to tmux. My reason for switching was that GNU screen didn't work in my docker containers and tmux did ;-)

All was well for a few months and I was replacing screen with tmux everywhere. It did have some other niceties besides working in containers and seem to do its job.

Until


USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
wim       1660  1.3 12.8 135056 131404 ?       Ss    2014 722:46 tmux -u


Notice anything special above ? Compare it with screen.


USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
wim      29595  0.0  4.5  48784 46116 ?        Ss    2014   3:49 SCREEN -c mscreen


The tmux session has 8 open windows and 10000 history limit. (set -g history-limit 10000)
The screen session has 39 open windows and 10000 history limit (defscrollback 5000)

So, tmux seems to be using an awful lot of memory. Two times more than screen, for a 'lighter' session setup.

A quick google showed that other people were having the same issues

My first thought was, 'memoryleak', so I checked the code, but everything seemed to be free'd correctly.

I joined the #tmux channel on freenode for some help and got told that it's a specific glibc (linux) issue. Although the memory was free'd, Glibc wasn't releasing it back to the OS.

But you could force it by using malloc_trim(0). And maybe you could use specific glibc environment variables to control memory allocation behaviour to also emulate malloc_trim().

Too much time googling and testing was wasted, I couldn't get it too work, the memory wasn't getting released back to the OS.

So I made a small patch to tmux which
- calls malloc_trim(0) when a window gets destroyed
- also free's memory when you clear your history manually in a window (and also call malloc_trim())

The patch works for me but YMMV

I tried to get this patch into upstream tmux, but was told: 'It's up to glibc to decide how malloc works'.

PS: if you set history-limit 0, tmux actually uses less memory than screen (and doesn't grow), but ofcourse you don't have a scrollback ;-)


Saturday, January 24, 2015

Rancid 3.2 alpha + git


Rancid lovers rejoice, a 3.2 alpha version is released with (at least) 2 interesting features.

- Git support: based on the patch by jcollie.

But with a 'small' difference, not one repository for all the groups, but a repository per group.
Maybe fine if your starting from scratch, but for my situation I like the one repository setup of the original patch.

You can find the latest version with the original setup of one repository for everything, together with some other minor patches on https://github.com/42wim/rancid/commits/mypatches3.1.99

- WLC support: Now you can backup your Cisco Wireless Lan Controllers configuration out of the box. One patch less to maintain. Hurrah!

I'm running Rancid in a Docker setup, so upgrading and testing was quite easy.
No issues found yet with this version.