Using nomad to deploy/manage containers (on a mostly IPv6 network)

Overview

This blog will show an overview of our container deployment using Nomad and IPv6-(only) where possible.

The focus for this setup was to empower developers and keeping things simple for them and us (the admins).
They can configure firewalls, storage, traefik, internet accessibility and more and have an https enabled project running in minutes.

As we run our own physical datacenters we have the luxury to use IPv6 where we want/can and do not need the mess of overlay networks and NAT.

Our developers create a nomadjob using a nomadgen.toml file which simplifies the nomad hcl plans (and extends it with features nomad itself cannot do).
You can find an example below. Most of the lines are self explanatory, i've added some comments

This nomadgen.toml gets checked into gitea, where Jenkins will pick it up and send it to nomadguard.

Nomadguard will:

• turn the toml back into a complete hcl with all of our infrastructure parameters filled in.
• run our nomad validator on it which checks if the user specified correct settings, uses the correct namespace and authorizations etc.
• runs nomad modifier on it, which allows us to modify jobs to automatically scale multi-containers over our 2 datacenters, move on specific nodes, limit memory, cpu or other things we need to do without developer interaction.

This modified nomad job will then get deployed to the nomad scheduler which will send it to a nomad node that has the resources the developer asked for.

On that nomad node the nomad agent will contact vault to resolve any needed secrets and it will start the docker container. At startup the incoming and outgoing firewall for that container will be configured. As we're using IPv6 our containers are routable and direct accessible, so we need to have a firewall in place for that. We use a modified registrator to add specific ipset entries to allow access to or from the container.

At the same time nomad will also register it's IPv6 addresses in our consul DNS (so traefik can start sending traffic to it).

If the container is running a http/https service this will be automatically exposed on https://p-es-cerebro.cloud.internal.domain (where p is the first letter of the tier (production/quality/test))

All of this infrastructure around the containers is IPv6 only and the containers themselves only allow IPv6 ingress, but they do have dualstack egress as some of the services they need (on the inter- or intranet) are not dualstack yet.

At the edge we have a netscaler that talks dualstack (IPv4/IPv6) to the users and moves to IPv6 only when talking to traefik and containers.

We're using default docker from Centos 7 and every nomad node is getting a IPv6 /80 range routed which it can use to give the containers their ipv6 addresses.

Extra tooling

We have some more tooling available for developers as they need to debug their deployments. This is where we have nomadctl which allows them to ask information about their job, see logs (coming from elasticsearch) and enter containers.

$ nomadctl ps cerebro
Exec ID    |Job/Task        |Node          |Uptime     |CPU |Mem(max)         |Extra
           |p-es-cerebro OK |              |           |    |                 |
c4996e3c72 |p-es-cerebro    |p-cloud-dc1-9 |3 days ago |26  |948 MiB(1.2 GiB) |
59702e6d47 |p-es-cerebro    |p-cloud-dc2-8 |3 days ago |27  |872 MiB(1.1 GiB) |

Or exec into a container

$ nomadctl exec c4996e3c72
Welcome wim (ssh cert verified)
welcome to p-es-cerebro on p-cloud-dc1-9
# ss -an | grep 9000
tcp    LISTEN     0      100                          :::9000                                     :::*

Issues

Of course there were issues, but not that many ;-)

• Especially in the early days of our setup we had some IPv6 issues in the hashicorp tools, but as they are opensource it's easy to fix those. (in contrast to hardware vendors where bugs are ignored or takes years to fix ..)
• Nomad 0.8 to 0.9 was troublesome because a lot of nomad stuff internally was rewritten and caused some issues in our setup.

The main takeaway after 5 years is that the nomad/consul/vault infrastructure is really solid and needs no babysitting.
And yes, IPv6 is (mostly) ready for production!

Routable IPv6 containers with podman

Hacking podman to have "rootless" routable ipv6 containers using a small root daemon.

Podman is great, but to have it replace our current docker setup it also needs ipv6 support (which it has using slirp4netns), but this isn't reachable from other containers or outside the host.

We don't care about incoming legacy IP (ipv4).

What do we want

When a user starts a container, the container should have a routable IPV6 address and register it's name in consul. That way we can have multiple containers talk to eachother, no matter from which host they're started. (and this all needs to work on centos 7.6)

What do we need

From podman

• The id of the user that started the container
• The PID of the container so we can use this to enter the same network namespace
• The name of the container so we can register this in consul
• A way to talk to v6pod

External to podman (v6pod will handle this)

• Be compatible with our current docker IPv6 ranges (/80)
• Creating a bridge and add the gateway IPv6 address to it (::1)
• Creating a veth pair
• Generate a (dynamic) IPv6 address in the /80 range and add to veth that will come into the container
• Add one of the veths to the bridge, the other in the network namespace of the user
• Add a default IPv6 route to the bridge
• Register the name of the container to the generated IPv6 address in consul
• Deregister the name when the container stops

Modifying libpod/podman

Code here: https://github.com/42wim/libpod/tree/rootlessv6

1) Executing user

Some investigation into what happens when running podman run (rootless)
Podman tries to create a user namespace, join this and become root in it and re-executes itself in that namespace.

We need to save the id of the executing user somewhere, the environment looks a good place.
So we create a v6pod_user variable which contains the userid of the user running podman.

2) Pid of the container

This could be added somewhere better probably, but I kept it in the same method.

We don't have access to the container PID yet there because it hasn't started, but we already have the container ID that will be used.

So I save the container ID in the v6pod_id environment variable.

v6pod will then look into /run/user/" + userID + "/runc/" + containerID + "/state.json file to get the PID

Overview below of what happens when podman runs executes again but now in the user namespace.

3) Name of the container

We could've set this using another variable, but to be more flexible (maybe we need more information about the container in the future) we choose not to.

Podman saves it create-config in the path: "/run/user/" + userID + "/libpod/tmp/socket/" + containerID + "/artifacts/create-config" which contains a lot information and also the container name.

4) Talk with v6pod

Here we just hijack the slirp4netns command (which enables userspace networking) and replace it with a v6pod-slirp4netns bash file which contains:

#!/bin/bash
/bin/curl -XPOST -d "user=$v6pod_user&id=$v6pod_id" http://localhost:6781/api/activate
/bin/slirp4netns "$@"
/bin/curl -XPOST -d "user=$v6pod_user&id=$v6pod_id" http://localhost:6781/api/deactivate

So we use the variables we set above to do all the networking stuff we need, then let slirp4netns do it setup so we still have outgoing IPv4 besides IPv6. When the container ends, slirp4netns exists and we do a deregistration.

Modifying slirp4netns

slirp4netns sets an ipv4 and ipv6 address and gateways. We do the IPv6 part now, so this needs to be disabled in slirp4netns.

source https://github.com/42wim/slirp4netns/commits/v6rootless

v6pod

v6pod is a go daemon with a rest interface that has a /activate and /deactivate entrypoint.

It implements the requirements of above:

• Be compatible with our current docker IPv6 ranges (/80)
• Creating a bridge and add the gateway IPv6 address to it (::1)
• Creating a veth pair
• Generate a (dynamic) IPv6 address in the /80 range and add to veth that will come into the container
• Add one of the veths to the bridge, the other in the network namespace of the user
• Add a default IPv6 route to the bridge
• Register the name of the container to the generated IPv6 address in consul
• Deregister the name when the container stops

Soon at https://github.com/42wim/v6pod

Buildah inside a centos 7.5 docker container on a centos 7.5 host

Our current solution uses Jenkins to start a Nomad job which starts a (unprivileged) docker container in which a developers Dockerfile is being build (as root) using the docker on the host.

The goal is to replace the docker build in the container by buildah so that we don't need to make the docker on the host available inside the container.

The path to this wasn't as straightforward unfortunately, a lot of yaks needed shaving.

Start of the journey

We're starting with a basic container where we install buildah in

# docker run --rm -ti centos:7 /bin/bash
[root@7387c68139dd /]# yum -y install buildah

And a very simple Dockerfile

FROM centos:7
RUN uptime

Yak 1 - overlay problems

Out of the box running buildah in the container will give an overlay error.

# buildah bud -t test .
ERRO[0000] 'overlay' is not supported over extfs at "/var/lib/containers/storage/overlay"
ERRO[0000] 'overlay' is not supported over extfs at "/var/lib/containers/storage/overlay"
kernel does not support overlay fs: 'overlay' is not supported over extfs at "/var/lib/containers/storage/overlay": backing file system is unsupported for this graph driver
kernel does not support overlay fs: 'overlay' is not supported over extfs at "/var/lib/containers/storage/overlay": backing file system is unsupported for this graph driver

Spoiler: The real reason this doesn't work is because it tries to do a mount call, which can only be done with the SYS_ADMIN capability (or in a privileged container).

Using --storage-driver vfs fixed this problem.

On to the next one.

Yak 2 - mount namespace error aka unshare(CLONE_NEWNS) permission aka the wrong yak

Spoiler: this yak is a red herring

# buildah --storage-driver vfs bud -t test .
STEP 1: FROM centos:7
Getting image source signatures
Copying blob sha256:aeb7866da422acc7e93dcf7323f38d7646f6269af33bcdb6647f2094fc4b3bf7
 71.24 MiB / 71.24 MiB [====================================================] 4s
Copying config sha256:75835a67d1341bdc7f4cc4ed9fa1631a7d7b6998e9327272afea342d90c4ab6d
 2.13 KiB / 2.13 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
STEP 2: RUN uptime
error running container: error creating new mount namespace for [/bin/sh -c uptime]: operation not permitted
error building at step {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] Command:run Args:[uptime] Flags:[] Attrs:map[] Message:RUN uptime Original:RUN uptime}: exit status 1

strace to the rescue

unshare(CLONE_NEWNS)              = -1 EPERM (Operation not permitted)

After some googling I found that centos/rhel kernels have user namespace disabled by default and need to have a kernel parameter set to get this working.

We can enable this by running on the host

sudo grubby --args="namespace.unpriv_enable=1 user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"

And also set the maximum number of user namespaces that any user in the current user namespace may create by running

echo "user.max_user_namespaces=15000" >> /etc/sysctl.conf

Now we can reboot the server

And come to the conclusion that it still doesn't work.

Yak 3 - outdated buildah version

Thanks to the #buildah channel on freenode, I found out that the problem of yak 2 was actually an outdated buildah version.
Centos has only a buildah 1.2 rpm, but 1.4 or higher was needed so I'd have to build my own.

You can have this pleasure too with the following script containing a modified RPM spec.

Run a new centos:7 container

# docker run -ti -v /tmp:/tmp centos:7 /bin/bash

and run following commands in the container:

yum -y group install development
yum -y install wget
cd /root/rpmbuild/SOURCES
wget "https://github.com/containers/buildah/tarball/608fa843cce45e7ee58ccb71a90297b645a984d3" -O buildah-608fa84.tar.gz
tar zxvf buildah-608fa84.tar.gz
mv containers-buildah-608fa84 buildah-608fa843cce45e7ee58ccb71a90297b645a984d3 
tar zcvf buildah-608fa84.tar.gz buildah-608fa843cce45e7ee58ccb71a90297b645a984d3
rm -rf buildah-608fa843cce45e7ee58ccb71a90297b645a984d3
cd ../SPECS
wget https://gist.githubusercontent.com/42wim/848fba2ed2d64d457f56eeebef0e85a2/raw/bb3ad3c524529ed921626fb077b8ff78a56783fc/buildah.spec -O buildah.spec
yum-builddep -y buildah.spec
rpmbuild -ba buildah.spec

This will give you your RPMs

Wrote: /root/rpmbuild/SRPMS/buildah-1.4-1.git608fa84.el7.centos.src.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/buildah-1.4-1.git608fa84.el7.centos.x86_64.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/buildah-debuginfo-1.4-1.git608fa84.el7.centos.x86_64.rpm

Yak 4 - proc mount error

Progress, a new error when running buildah 1.4!

# buildah --storage-driver vfs bud -t test .
STEP 1: FROM centos:7
Getting image source signatures
Copying blob sha256:205941c9c2d103bcdff0bc72d8836e0ffc4573ec0e6e524ec1a59606062a289f
 71.25 MiB / 71.25 MiB [====================================================] 4s
Copying config sha256:e26dc8af6a3b1856b9f4a893d5b51855c02dfe3b9cec58a4e55002036528c669
 2.14 KiB / 2.14 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
STEP 2: RUN uptime
container_linux.go:336: starting container process caused "process_linux.go:399: container init caused \"rootfs_linux.go:58: mounting \\\"/proc\\\" to rootfs \\\"/tmp/buildah596035765/mnt/rootfs\\\" at \\\"/proc\\\" caused \\\"operation not permitted\\\"\""
error running container: error creating container for [/bin/sh -c uptime]: : exit status 1
error building at step {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] Command:run Args:[uptime] Flags:[] Attrs:map[] Message:RUN uptime Original:RUN uptime}: exit status 1
ERRO[0012] exit status 1

Again thanks to #buildah channel, I found out that running --isolation chroot would solve it.

Victory!

Finally it works, we have an image created by buildah running in an unprivileged container.

# buildah --storage-driver vfs bud --isolation chroot -t test .
STEP 1: FROM centos:7
STEP 2: RUN uptime
 21:30:55 up 32 min,  0 users,  load average: 0.39, 0.12, 0.08
STEP 3: COMMIT containers-storage:[vfs@/var/lib/containers/storage+/var/run/containers/storage]localhost/test:latest
Getting image source signatures
Skipping fetch of repeat blob sha256:f972d139738dfcd1519fd2461815651336ee25a8b54c358834c50af094bb262f
Skipping fetch of repeat blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
Copying config sha256:26e3b2177f9e9db1bdc8f49083d09dbb980a99ed4e606f4dc45b79ca865588ce
 1.17 KiB / 1.17 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
--> 26e3b2177f9e9db1bdc8f49083d09dbb980a99ed4e606f4dc45b79ca865588ce

But after testing a new yak appears.

Yak 5 - a lot of diskspace

This one is related to yak 1, because we're using the vfs storage driver, it uses the disk not very space efficient (according to https://docs.docker.com/storage/storagedriver/vfs-driver/) a more complicated docker build uses gigabytes of disk when using the vfs storage driver compared to the overlay driver.

To run with the overlay driver we need access to the mount call which means we have to run our docker container with CAP_SYS_ADMIN which is unfortunate.

# docker run --rm --add-cap SYS_ADMIN -ti centos:7 /bin/bash

Conclusion

It's possible to run buildah in an unprivileged container but only using the vfs storage driver, but beware of the disk usage when building images!

Interesting links:

• https://github.com/containers/fuse-overlayfs this will probably fix the CAP_SYS_ADMIN issue for overlay
• https://kinvolk.io/blog/2018/04/towards-unprivileged-container-builds/ an overview of what's the problem and what's getting fixed regarding to unprivileged builds, must read!
• https://buildah.io everything about buildah

How to create an IPv6-only consul cluster with docker

Why?

• we're using docker to run consul (and registrator and our services) in, and IPv6 makes this easier (no NAT => better performance)
• it's easier to maintain one stack
• consul is known to give issues with NAT and docker (https://github.com/docker/docker/issues/8795)
• IPv4 is legacy and obsolete ;-)

Consul 0.5.2 has some issues running such a setup, but if you're building consul from master (which includes some fixes (see https://github.com/hashicorp/consul/commits?author=42wim) it will work fine.

Issues to be aware of:

• the IPv4 version of consul listens by default on private address ranges, when using IPv6 you'll be running on 'public' addresses. So be sure you're firewalling those from the internet.
• If you're using consul recursive powers, you'll also need IPv6 dns recursors. (e.g. google's 2001:4860:4860::8888)
• Not IPv6 related, but for extra stability, enable leave_on_terminate.
• Also not Ipv6 related, but I've noticed that the default LAN settings for consul can be a bit too strict when running on vmware hosts. This patch increase the probetimeout to 2 seconds (instead of 500msec)

Consul extra configuration server and client

Extra settings below necessary for the consul server and client agent setup

Configuration:
{
        "recursor": "[2001:4860:4860::8888]",
        "leave_on_terminate": true,
        "client_addr": "::",
        "addresses": { "http": "::"}
}

Consul server setup

The consul server are running as a docker host mode container (which means, they share the same network namespace as the host).

The reason here is that we need a fixed IPv6 address for the servers because we're forwarding our dns requests to those servers. (ofcourse with some extra work we could make a script that dynamically update our dns forwards to the dynamic IP address).

Our server has multiple IPv6 addresses so we'll have to add a -advertise and -bind flag

consul agent -server -advertise 2001:db8::1 -bind 2001:db8::1 -bootstrap-expect 3 -retry-join [2001:db8::1]:8301 -retry-join [2001:db8::2]:8301 -retry-join [2001:db8::3]:8301

Using consul-docker as our consul docker container (for client and server)

Consul client setup 

You'll need to cherry-pick this PR into your local build: https://github.com/hashicorp/consul/pull/1219.
The IPv6 address in the docker container will be random and we want to bind to the IPv6 address.
This patch looks for the first 'public' IPv6 address and uses this address to advertise.

So we start the client with:

consul agent -bind :: -join consul.service.consul

Gotcha's here:
bind :: actually binds to IPv4 and IPv6 addresses in the container, but because we advertise the IPv6 address the IPv4 address won't be used.

Other software

Registrator

We also use registrator to register our services in consul. So every time a container starts or stops, registrator handles the consul service registration process.

Also for registrator some extra fixes are needed to have IPv6 support. (not yet merged, see https://github.com/gliderlabs/registrator/pull/229)

Because we're running consul on IPv6 this means registrator also needs to connect to the IPv6 address.

registrator consul://server1.node.consul:8500

Registrator then can register other services that are running on the docker host, like e.g elasticsearch.

Registrator-netfilter

Besides main registrator we also run registrator-netfilter which automatically firewalls the IPv6 services in the container. The containers are no longer NATted but directly accessible, so they need to be firewalled.

Docker

A /64 is allocated for docker and a /80 is given to each docker host, running with the switches

--ipv6=true --fixed-cidr-v6=2001:db8::/80

Elasticsearch

ES is also run ipv6 only, using registrator, registrator-netfilter and consul.
You can find the relevant commands to give to docker below:

docker run --net bridge -e SERVICE_NAME=es -e SERVICE_9200_TAGS=http-data 
-e SERVICE_9300_TAGS=transport-data -e SERVICE_9200_IPV6=tcp -e SERVICE_9300_IPV6=tcp 
-e ADVERTISE_IPV6=yes

tmux memory usage on linux

So a while ago I switched from screen to tmux. My reason for switching was that GNU screen didn't work in my docker containers and tmux did ;-)

All was well for a few months and I was replacing screen with tmux everywhere. It did have some other niceties besides working in containers and seem to do its job.

Until

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
wim       1660  1.3 12.8 135056 131404 ?       Ss    2014 722:46 tmux -u

Notice anything special above ? Compare it with screen.

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
wim      29595  0.0  4.5  48784 46116 ?        Ss    2014   3:49 SCREEN -c mscreen

The tmux session has 8 open windows and 10000 history limit. (set -g history-limit 10000)
The screen session has 39 open windows and 10000 history limit (defscrollback 5000)

So, tmux seems to be using an awful lot of memory. Two times more than screen, for a 'lighter' session setup.

A quick google showed that other people were having the same issues

My first thought was, 'memoryleak', so I checked the code, but everything seemed to be free'd correctly.

I joined the #tmux channel on freenode for some help and got told that it's a specific glibc (linux) issue. Although the memory was free'd, Glibc wasn't releasing it back to the OS.

But you could force it by using malloc_trim(0). And maybe you could use specific glibc environment variables to control memory allocation behaviour to also emulate malloc_trim().

Too much time googling and testing was wasted, I couldn't get it too work, the memory wasn't getting released back to the OS.

So I made a small patch to tmux which
- calls malloc_trim(0) when a window gets destroyed
- also free's memory when you clear your history manually in a window (and also call malloc_trim())

The patch works for me but YMMV

I tried to get this patch into upstream tmux, but was told: 'It's up to glibc to decide how malloc works'.

PS: if you set history-limit 0, tmux actually uses less memory than screen (and doesn't grow), but ofcourse you don't have a scrollback ;-)

Rancid 3.2 alpha + git

Rancid lovers rejoice, a 3.2 alpha version is released with (at least) 2 interesting features.

- Git support: based on the patch by jcollie.

But with a 'small' difference, not one repository for all the groups, but a repository per group.
Maybe fine if your starting from scratch, but for my situation I like the one repository setup of the original patch.

You can find the latest version with the original setup of one repository for everything, together with some other minor patches on https://github.com/42wim/rancid/commits/mypatches3.1.99

- WLC support: Now you can backup your Cisco Wireless Lan Controllers configuration out of the box. One patch less to maintain. Hurrah!

I'm running Rancid in a Docker setup, so upgrading and testing was quite easy.
No issues found yet with this version.

Circumventing IPv6 feature parity: drop AAAA to specific IPs

Unless you've been living under a rock, you'll be aware that IPv6 usage has been increasing.

Yes, it even has come to this: mere mortals can use it at home. The audacity!

Unfortunately not all vendors (if any?) have feature parity, in our case a specific VPN product doesn't support IPv6.
The client will only receive an IPv4 address from the VPN server.

When the user at home starts it's VPN and asks for an internal resource (which also has an IPv6 address), it will try to connect to this resource using the IPv6 from his provider (he didn't receive one from the VPN server) which doesn't work, because this specific resource is firewalled for outside addresses.

Luckily the user has to use our DNS server to look up records (forced to do so by the vpn client)
Luckily we're using PowerDNS recursor which has support for LUA scripting which can modify DNS responses.

The script below gives normal answers to every host not coming from 10.100.0.0/15 or 10.0.0.1/32. Otherwise if the answer contains an AAAA, drop it, and return the rest.

More information about LUA scripting for PowerDNS can be found here: http://doc.powerdns.com/html/recursor-scripting.html

Winter is coming

Which means I have some more time, expect some IPv6 and Cisco related posts in the near future.

Compiling your own PuTTY-CAC with EID support

So we've got electronic ID's, (smartcards) but except for doing our taxes we're not using them so much.

Now under linux there are options to use them for SSH authentication, but these days I'm mostly using Putty on Windows, so I wanted it to work with this client. 

After some searches I found a possible candidate: Putty-cac : http://www.risacher.org/putty-cac/

It works with CAPI, the military uses it, it's opensource and based on Putty. Seems like a win-win-win-win. And for once it also is :-)

So I downloaded the source from https://github.com/risacher/putty-cac/archive/master.zip

compared it with the official putty source from http://svn.tartarus.org/sgt/putty/ to see if nothing suspicious was added to the code. There wasn't, so I could safely build the binary myself.

I remembered that Visual Studio Express was a free C++ compiler from Microsoft, so i download version 2010

So now just open the project and press build right? Wrong! The project was made in Visual studio 6 and apparently you can not convert from visual studio 6 to visual studio 2010. According to the internets you need to first install Visual studio 2008, convert there, save it, open it in Visual studio 2010, convert, save and build.

Here is an overview for those that want to do this:

Start visual C++ 2008

Open Project - c:\temp\putty-cac-master\windows\MSVC\putty.dsw

Convert and open project

Choose File - Save All

Start visual C++ 2010 (and close 2008 ;-)

Open Project - c:\temp\putty-cac-master\windows\MSVC\putty.sln

You'll get a wizard: Next - Next - Finish

Now when you try to build it, it won't. You'll need to add a define 

Open c:\temp\putty-cac-master\windows\winstuff.sh

Add  #define SECURITY_WIN32 at the top of the file

If you compile now you'll get linking errors. You'll need to add 'sc.c' and 'capi.c' to the 'source files'

Now you're finally ready to build your binary. Press build and enjoy your own build putty.

To actually use your EID with this, just follow the CAPI instructions on http://www.risacher.org/putty-cac/

Stuff you'll need to get this working:

Microsoft Windows SDK for Windows 7 and .NET Framework 4

http://www.microsoft.com/en-us/download/details.aspx?id=8279

Visual 2008 express

http://download.microsoft.com/download/E/8/E/E8EEB394-7F42-4963-A2D8-29559B738298/VS2008ExpressWithSP1ENUX1504728.iso

Visual 2010 express

http://go.microsoft.com/?linkid=9709949

PuTTY CAC source

https://github.com/risacher/putty-cac/archive/master.zip

Good luck!

DHCPv6: ISC MAC logging and Cisco relay agent configuration

DHCPv6 server MAC address logging

One of the differences between DHCPv6 and DHCPv4 is that it uses DUID as an identifier instead of a MAC address.

As you probably know, DUID's are for the majority of OS (ie Windows) based on a timestamp suffixed by a MAC address

For some of our internal systems we use a MAC as an identifier, for now we will also be  needing this for IPv6. The default ISC DHCPv6 daemon isn't logging a MAC address by default. One way to have it print out is by adding this to your DHCP config.

option dhcp6.macaddr code 193 = string;
option dhcp6.leased-address code 194 = string;
option dhcp6.macaddr = binary-to-ascii(16, 8, ":", suffix(option dhcp6.client-id, 6));
option dhcp6.leased-address = binary-to-ascii(16,16, ":", substring(suffix(option dhcp6.ia-na, 24),0,16));
log (info, concat ("Lease for ",config-option dhcp6.leased-address, " leased to ", config-option dhcp6.macaddr));

Above code will only work for DUID-LLT and DUID-LL (so not DUID-EN, but I don't know anyone using this at the moment)

More info over DUID on http://tools.ietf.org/html/rfc6355

Other great blogpost about DHCPv6 at http://ipv6friday.org/blog/2011/12/dhcpv6/

DHCPv6 relay configuration on Cisco equipment

Overview of the configuration we're using on our routers.

(1) We're using FE80:: as our IPv6 default gateway everywhere, seems to work for me for now ;) any best practices for this? (update: because of issues with linux and fe80:: (linux responds to fe80:: if it's specified on any link, this has now changed to FE80::1)

(2) Asking the hosts on the subnet to not do SLAAC please, we're asking you nicely.

(3) Letting the hosts know we're managing the config, that they must use DHCPv6.

(4) Just to be sure also set this flag, tell them to use DHCPv6 not only for getting an IP but also for getting e.g. DNS servers

(5) We're the boss on this subnet.

(6) relay DHCPv6 requests to our server and use (7) the loopback as a source for this.

We're using link-local addresses for routing, so if we don't specifiy an source-interface, the relay agents would try to use a link-local address which obviously can not be routed.

Interface vlan42
  ipv6 address FE80::1 link-local (1)
  ipv6 address 2001:0db8:100:4200::1/64
  ipv6 nd prefix default 2592000 604800 no-autoconfig (2)
  ipv6 nd managed-config-flag (3)
  ipv6 nd other-config-flag (4)
  ipv6 nd router-preference High (5)
  ipv6 dhcp relay destination 2001:0db8:0:40::547:1 (6)
  ipv6 dhcp relay source-interface Loopback0 (7)

interface Loopback0
 ipv6 address  2001:0db8:300::63/128

IPv6 logging and Cisco NCS

So you want to deploy IPv6 on your wireless network. 

You want to use SLAAC and you want logging of those SLAAC addresses.

Straight from the horses mouth:

Q: What are IPv6 private addresses and why are they important to track?

A: Private (also known as temporary) addresses are randomly generated by the client when SLAAC address assignment is in use. These addresses are often rotated at a frequency of a day or so, as to prevent host traceability that would come from using the same host postfix (last 64 bits) at all times. It is important to track these private addresses for auditing purposes such as tracing copyright infringement. Cisco NCS records all IPv6 addresses in use by each client and historically logs them each time the client roams or establishes a new session. These records can be configured at NCS to be held for up to a year.

Pasted from <http://www.cisco.com/en/US/products/ps10315/products_tech_note09186a0080bae506.shtml#faqs>

BUT Cisco NCS or PI doesn't make it easy for you, it keeps the addresses in the database somewhere, but as soon as the client has disassociated you can't search for the IPv6 address anymore.  Disassociated IPv4 addresses can be searched though.

So far for IPv6 parity.

Radius to the rescue ? 

But wait, we've got radius accounting, right? You'll see the Framed-IP-Address attribute and this will show you the IPv6 address, right ? RIGHT ?

As we all know, IPv6 is a very new protocol (only about 15 years old), so of course there isn't support for IPv6 in the Framed-IP-Address attribute. There is a draft proposing Framed-IPv6-Address (also very new, only 3 years old).

(See http://tools.ietf.org/html/draft-ietf-radext-ipv6-access-16 for more info)

So, no searching, no IPv6 address logging by Radius.

So far for IPv6 parity again..

But according to the documentation (see above) cisco is recording those addresses somewhere ...

Reports to the rescue!

The workaround is reports:

Go to the report launch pad - Client - Client Sessions

Create a new report:

- report by SSID (or your own favorite source)

- reporting criteria (all SSIDs)

- reporting period select last 7 days

Customize the report, where you can find the most important data field: "Global Unique", this will show you the IPv6 address. Now you can schedule this report weekly and you've got weekly CSV files containing all the necessary information of the users.

If anyone got a better workaround please share!

Cisco Prime Infrastructure NCS 1.2 and the Oracle issue

Continuing my story about NCS diskspace increase.

A few weeks everything went fine, until one day the webserver or the shell wasn't accessible anymore.
In the vmware server console we saw that all of the memory was in use (8GB). So we add some extra memory and rebooted the system. (At that time I didn't made the correlation with increasing the disk space a few weeks ago)

This didn't solve the problem.

But I could log in and memory usage seemed normal. First I went looking in /var/log , you can skip this step, nothing of value can be found here.

The interesting directories are /opt/CSCOlumos/logs/

Last file modified was hm-0-0.log. Containing:

ERROR [system] [HealthMonitorServer] HealthMonitorServer.initHealthMonitor: initHealthMonitor(): can not start DB
INFO  [database] [Thread-17] OracleSchemaUtil dbServerUp(): wcs errorCode = 1034

(Other logfiles were also giving oracle errors)
Closing in on the oracle DB.

Next target /opt/oracle (disclaimer: I know nothing about Oracle), after some searching I discovered the logfile /opt/oracle/diag/rdbms/wcs/wcs/trace/alert_wcs.log which seemed interesting.

ORA-19815: WARNING: db_recovery_file_dest_size of 107374182400 bytes is 100.00% used, and has 0 remaining bytes available.
ORA-19809: limit exceeded for recovery files
ORA-16038: log 1 sequence# 1018 cannot be archived
ORACLE Instance wcs - Archival Error
ARCH: Archival stopped, error occurred. Will continue retrying

This didn't look good. Apparently Oracle has it own internal idea of how big a disk is and how much free space is available. Because we still had 130Gb available.

/dev/mapper/smosvg-optvol
                     249204396  97436636 138909244  42% /opt

With the help of google and a colleague with the necessary oracle knowledge. This was the solution (increasing the recovery db size):

[root@ncs oracle]# su - oracle
[oracle@ncs ~]$ ls
base  coracleenv  dbPasswd.pwd  oracleenv  oraInventory  templates  utils
[oracle@ncs ~]$ . oracleenv
[oracle@ncs ~]$ sqlplus '/as sysdba'
SQL*Plus: Release 11.2.0.2.0 Production 
Copyright (c) 1982, 2010, Oracle.  All rights reserved.
Connected to an idle instance.

SQL> startup nomount
ORACLE instance started.

Total System Global Area 4275781632 bytes
Fixed Size                  2233336 bytes
Variable Size            2986347528 bytes
Database Buffers         1275068416 bytes
Redo Buffers               12132352 bytes
SQL> show parameter db_recovery_file_dest_size

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_recovery_file_dest_size           big integer 100G
SQL> alter system set db_recovery_file_dest_size=120G scope=both;

System altered.

SQL> show parameter db_recovery_file_dest_size;

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_recovery_file_dest_size           big integer 120G
SQL> shutdown
ORA-01507: database not mounted


ORACLE instance shut down.
SQL> quit

Still the error, but now some free space to do something

ORA-19815: WARNING: db_recovery_file_dest_size of 128849018880 bytes is 85.07% used, and has 19241095680 remaining bytes available.

Next ran ncs cleanup on the NCS CLI

ncs/admin# ncs cleanup
===================================================
Starting Cleanup: 
===================================================
Removing all files in backup staging directory
Removing all Matlab core related files
Removing all older log files
Cleaning older archive logs
Cleaning database backup and all archive logs
Cleaning database
Stopping database
Starting database
Starting database clean
Completed database clean
Stopping database
===================================================
Completed Cleanup
===================================================
ncs/admin#

And rebooted the system.

Oracle DB came up normally and NCS was happily running again.

New features added to WPTVdroid

New features recently added to WPTVdroid:

Channel guide (with current and next running show) added when you click long on the selected channel.

Channel preview, go to menu and select "Channels preview", this will briefly connect to every channel and will show you a screenshot of the current program. Just click on the screenshot to go to the selected channel.
(sometimes the screenshot will not be very clear/show artifacts.. haven't found a solution for this yet)
(this feature adds about 10MB to the WPTVdroid installation, if you don't have that much space don't use this version)

Downloads on http://blog.42.be/p/weepeetv.html

Any new features/ideas you would like to have added? Leave a comment or contact me @42wim

Increasing diskspace on Cisco Prime Infrastructure NCS 1.2 "the hard way?"

Somewhere last year PI NCS 1.2 was constantly complaining about disk full errors, although the the disk was only 62% used.

Apparently the complaining starts above 60% because it needs to be able to make a temporary backup on the same disk as it is running the database.

Our datacenter guy increased the disk (200GB) in vmware by increasing it by 100GB.

I wrongly thought this would have fixed the problem and that the system would automatically resize everything.

I was wrong.

So after running root_enable it was possible to login as root (it's just running a linux beneath it)

Below you can see the layout of the system with 62% in use for /opt where the database and temporary backups are located.

Filesystem           1K-blocks      Used Available Use% Mounted on

/dev/mapper/smosvg-rootvol
                       1967952    477860   1388512  26% /
/dev/mapper/smosvg-tmpvol
                       1967952     36460   1829912   2% /tmp
/dev/sda3               988116     17764    919348   2% /storedconfig
/dev/mapper/smosvg-recvol
                         95195      5664     84616   7% /recovery
/dev/mapper/smosvg-home
                         95195      5668     84612   7% /home
/dev/mapper/smosvg-optvol
                     147630356  86745492  53264668  62% /opt
/dev/mapper/smosvg-usrvol
                       5935604    904020   4725204  17% /usr
/dev/mapper/smosvg-varvol
                       1967952     90372   1776000   5% /var
/dev/mapper/smosvg-storeddatavol
                       3967680     74320   3688560   2% /storeddata
/dev/mapper/smosvg-altrootvol
                         95195      5664     84616   7% /altroot
/dev/mapper/smosvg-localdiskvol
                      29583412    204132  27852292   1% /localdisk
/dev/sda1               101086     12713     83154  14% /boot
tmpfs                  4021728   2044912   1976816  51% /dev/shm

How to fix this manually.
So we run fdisk, and we see that the extra 100GB is detected (/dev/sda 322.1GB)
We then add another (4) primary partition which consists of the missing 100GB

# fdisk /dev/sda

The number of cylinders for this disk is set to 39162.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): p

Disk /dev/sda: 322.1 GB, 322122547200 bytes
255 heads, 63 sectors/track, 39162 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          13      104391   83  Linux
/dev/sda2              14       25368   203664037+  8e  Linux LVM
/dev/sda3           25369       25495     1020127+  83  Linux

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)

p
Selected partition 4
First sector (409577175-629145599, default 409577175):
Using default value 409577175
Last sector or +size or +sizeM or +sizeK (409577175-629145599, default 629145599):
Using default value 629145599

Command (m for help): p

Disk /dev/sda: 322.1 GB, 322122547200 bytes
255 heads, 63 sectors/track, 39162 cylinders, total 629145600 sectors
Units = sectors of 1 * 512 = 512 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *          63      208844      104391   83  Linux
/dev/sda2          208845   407536919   203664037+  8e  Linux LVM
/dev/sda3       407536920   409577174     1020127+  83  Linux
/dev/sda4       409577175   629145599   109784212+  83  Linux

Also tag it 8e to be a linux LVM filesystem.
And write the partition table <scary>

Command (m for help): t
Partition number (1-4): 4
Hex code (type L to list codes): 8e
Changed system type of partition 4 to 8e (Linux LVM)

Command (m for help): p

Disk /dev/sda: 322.1 GB, 322122547200 bytes
255 heads, 63 sectors/track, 39162 cylinders, total 629145600 sectors
Units = sectors of 1 * 512 = 512 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *          63      208844      104391   83  Linux
/dev/sda2          208845   407536919   203664037+  8e  Linux LVM
/dev/sda3       407536920   409577174     1020127+  83  Linux
/dev/sda4       409577175   629145599   109784212+  8e  Linux LVM

Command (m for help): v
62 unallocated sectors

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.

Now reboot the system.
When it boots up, the partition can be used.
Use pvcreate to create the new physical volume.
vgdisplay will still show the old 194.22GB because it's not added yet to smosvg.

# pvcreate /dev/sda4
  Physical volume "/dev/sda4" successfully created

# vgdisplay
  --- Volume group ---
  VG Name               smosvg
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  12
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                11
  Open LV               11
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               194.22 GB
  PE Size               32.00 MB
  Total PE              6215
  Alloc PE / Size       6215 / 194.22 GB
  Free  PE / Size       0 / 0
  VG UUID               vJcFNu-qPIP-uoKY-KiIu-wKCd-Jj7l-e8cLaV

pvdisplay will show the 2 volumes.

# pvdisplay
  --- Physical volume ---
  PV Name               /dev/sda2
  VG Name               smosvg
  PV Size               194.23 GB / not usable 10.66 MB
  Allocatable           yes (but full)
  PE Size (KByte)       32768
  Total PE              6215
  Free PE               0
  Allocated PE          6215
  PV UUID               ugQzvR-HsdX-cK8u-Sylm-NYAm-FXFX-YXkb6R

  --- Physical volume ---
  PV Name               /dev/sda4
  VG Name               smosvg
  PV Size               104.70 GB / not usable 11.15 MB
  Allocatable           yes
  PE Size (KByte)       32768
  Total PE              3350
  Free PE               3350
  Allocated PE          0
  PV UUID               pEPFVV-AI3e-HLfo-fy5i-TyEN-eRCp-BJiXPS

Now add this volume to /opt and extend it by 50Gb And use resize2fs so that the kernel nows the size changed.

# lvextend /dev/mapper/smosvg-optvol /dev/sda4
# lvextend -L +50G /dev/mapper/smosvg-optvol
  Extending logical volume optvol to 195.34 GB
  Logical volume optvol successfully resized

# resize2fs /dev/mapper/smosvg-optvol
resize2fs 1.39 (29-May-2006)
Filesystem at /dev/mapper/smosvg-optvol is mounted on /opt; on-line resizing required
Performing an on-line resize of /dev/mapper/smosvg-optvol to 51208192 (4k) blocks.
The filesystem on /dev/mapper/smosvg-optvol is now 51208192 blocks long.

A df will noshow 47% in use for /opt
The annoying popups are now gone.

# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/smosvg-rootvol
                       1967952    477876   1388496  26% /
/dev/mapper/smosvg-tmpvol
                       1967952     36460   1829912   2% /tmp
/dev/sda3               988116     17764    919348   2% /storedconfig
/dev/mapper/smosvg-recvol
                         95195      5664     84616   7% /recovery
/dev/mapper/smosvg-home
                         95195      5668     84612   7% /home
/dev/mapper/smosvg-optvol
                     198417376  88100224 100077796  47% /opt
/dev/mapper/smosvg-usrvol
                       5935604    904020   4725204  17% /usr
/dev/mapper/smosvg-varvol
                       1967952     88744   1777628   5% /var
/dev/mapper/smosvg-storeddatavol
                       3967680     74320   3688560   2% /storeddata
/dev/mapper/smosvg-altrootvol
                         95195      5664     84616   7% /altroot
/dev/mapper/smosvg-localdiskvol
                      29583412    204132  27852292   1% /localdisk
/dev/sda1               101086     12713     83154  14% /boot
tmpfs                  4021728   2044888   1976840  51% /dev/shm

It seems like I fixed everything and that I was in the clear.

Which it was for a couple of weeks until ... to be continued in a later post.