Why?
- we're using docker to run consul (and registrator and our services) in, and IPv6 makes this easier (no NAT => better performance)
- it's easier to maintain one stack
- consul is known to give issues with NAT and docker (https://github.com/docker/docker/issues/8795)
- IPv4 is legacy and obsolete ;-)
Issues to be aware of:
- the IPv4 version of consul listens by default on private address ranges, when using IPv6 you'll be running on 'public' addresses. So be sure you're firewalling those from the internet.
- If you're using consul recursive powers, you'll also need IPv6 dns recursors. (e.g. google's 2001:4860:4860::8888)
- Not IPv6 related, but for extra stability, enable leave_on_terminate.
- Also not Ipv6 related, but I've noticed that the default LAN settings for consul can be a bit too strict when running on vmware hosts. This patch increase the probetimeout to 2 seconds (instead of 500msec)
Consul extra configuration server and client
Extra settings below necessary for the consul server and client agent setup
Configuration:
{
"recursor": "[2001:4860:4860::8888]",
"leave_on_terminate": true,
"client_addr": "::",
"addresses": { "http": "::"}
}
Consul server setup
The consul server are running as a docker host mode container (which means, they share the same network namespace as the host).The reason here is that we need a fixed IPv6 address for the servers because we're forwarding our dns requests to those servers. (ofcourse with some extra work we could make a script that dynamically update our dns forwards to the dynamic IP address).
Our server has multiple IPv6 addresses so we'll have to add a -advertise and -bind flag
consul agent -server -advertise 2001:db8::1 -bind 2001:db8::1 -bootstrap-expect 3 -retry-join [2001:db8::1]:8301 -retry-join [2001:db8::2]:8301 -retry-join [2001:db8::3]:8301
Using consul-docker as our consul docker container (for client and server)
Consul client setup
You'll need to cherry-pick this PR into your local build: https://github.com/hashicorp/consul/pull/1219.The IPv6 address in the docker container will be random and we want to bind to the IPv6 address.
This patch looks for the first 'public' IPv6 address and uses this address to advertise.
consul agent -bind :: -join consul.service.consul
Gotcha's here:
bind :: actually binds to IPv4 and IPv6 addresses in the container, but because we advertise the IPv6 address the IPv4 address won't be used.
Other software
Registrator
We also use registrator to register our services in consul. So every time a container starts or stops, registrator handles the consul service registration process.Also for registrator some extra fixes are needed to have IPv6 support. (not yet merged, see https://github.com/gliderlabs/registrator/pull/229)
Because we're running consul on IPv6 this means registrator also needs to connect to the IPv6 address.
registrator consul://server1.node.consul:8500
Registrator then can register other services that are running on the docker host, like e.g elasticsearch.
Registrator-netfilter
Besides main registrator we also run registrator-netfilter which automatically firewalls the IPv6 services in the container. The containers are no longer NATted but directly accessible, so they need to be firewalled.
Docker
A /64 is allocated for docker and a /80 is given to each docker host, running with the switches
--ipv6=true --fixed-cidr-v6=2001:db8::/80
Elasticsearch
ES is also run ipv6 only, using registrator, registrator-netfilter and consul.You can find the relevant commands to give to docker below:
docker run --net bridge -e SERVICE_NAME=es -e SERVICE_9200_TAGS=http-data
-e SERVICE_9300_TAGS=transport-data -e SERVICE_9200_IPV6=tcp -e SERVICE_9300_IPV6=tcp
-e ADVERTISE_IPV6=yes
