tmux memory usage on linux

So a while ago I switched from screen to tmux. My reason for switching was that GNU screen didn't work in my docker containers and tmux did ;-)

All was well for a few months and I was replacing screen with tmux everywhere. It did have some other niceties besides working in containers and seem to do its job.

Until

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
wim       1660  1.3 12.8 135056 131404 ?       Ss    2014 722:46 tmux -u

Notice anything special above ? Compare it with screen.

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
wim      29595  0.0  4.5  48784 46116 ?        Ss    2014   3:49 SCREEN -c mscreen

The tmux session has 8 open windows and 10000 history limit. (set -g history-limit 10000)
The screen session has 39 open windows and 10000 history limit (defscrollback 5000)

So, tmux seems to be using an awful lot of memory. Two times more than screen, for a 'lighter' session setup.

A quick google showed that other people were having the same issues

My first thought was, 'memoryleak', so I checked the code, but everything seemed to be free'd correctly.

I joined the #tmux channel on freenode for some help and got told that it's a specific glibc (linux) issue. Although the memory was free'd, Glibc wasn't releasing it back to the OS.

But you could force it by using malloc_trim(0). And maybe you could use specific glibc environment variables to control memory allocation behaviour to also emulate malloc_trim().

Too much time googling and testing was wasted, I couldn't get it too work, the memory wasn't getting released back to the OS.

So I made a small patch to tmux which
- calls malloc_trim(0) when a window gets destroyed
- also free's memory when you clear your history manually in a window (and also call malloc_trim())

The patch works for me but YMMV

I tried to get this patch into upstream tmux, but was told: 'It's up to glibc to decide how malloc works'.

PS: if you set history-limit 0, tmux actually uses less memory than screen (and doesn't grow), but ofcourse you don't have a scrollback ;-)

Rancid 3.2 alpha + git

Rancid lovers rejoice, a 3.2 alpha version is released with (at least) 2 interesting features.

- Git support: based on the patch by jcollie.

But with a 'small' difference, not one repository for all the groups, but a repository per group.
Maybe fine if your starting from scratch, but for my situation I like the one repository setup of the original patch.

You can find the latest version with the original setup of one repository for everything, together with some other minor patches on https://github.com/42wim/rancid/commits/mypatches3.1.99

- WLC support: Now you can backup your Cisco Wireless Lan Controllers configuration out of the box. One patch less to maintain. Hurrah!

I'm running Rancid in a Docker setup, so upgrading and testing was quite easy.
No issues found yet with this version.

Circumventing IPv6 feature parity: drop AAAA to specific IPs

Unless you've been living under a rock, you'll be aware that IPv6 usage has been increasing.

Yes, it even has come to this: mere mortals can use it at home. The audacity!

Unfortunately not all vendors (if any?) have feature parity, in our case a specific VPN product doesn't support IPv6.
The client will only receive an IPv4 address from the VPN server.

When the user at home starts it's VPN and asks for an internal resource (which also has an IPv6 address), it will try to connect to this resource using the IPv6 from his provider (he didn't receive one from the VPN server) which doesn't work, because this specific resource is firewalled for outside addresses.

Luckily the user has to use our DNS server to look up records (forced to do so by the vpn client)
Luckily we're using PowerDNS recursor which has support for LUA scripting which can modify DNS responses.

The script below gives normal answers to every host not coming from 10.100.0.0/15 or 10.0.0.1/32. Otherwise if the answer contains an AAAA, drop it, and return the rest.

More information about LUA scripting for PowerDNS can be found here: http://doc.powerdns.com/html/recursor-scripting.html

Winter is coming

Which means I have some more time, expect some IPv6 and Cisco related posts in the near future.

Compiling your own PuTTY-CAC with EID support

So we've got electronic ID's, (smartcards) but except for doing our taxes we're not using them so much.

Now under linux there are options to use them for SSH authentication, but these days I'm mostly using Putty on Windows, so I wanted it to work with this client. 

After some searches I found a possible candidate: Putty-cac : http://www.risacher.org/putty-cac/

It works with CAPI, the military uses it, it's opensource and based on Putty. Seems like a win-win-win-win. And for once it also is :-)

So I downloaded the source from https://github.com/risacher/putty-cac/archive/master.zip

compared it with the official putty source from http://svn.tartarus.org/sgt/putty/ to see if nothing suspicious was added to the code. There wasn't, so I could safely build the binary myself.

I remembered that Visual Studio Express was a free C++ compiler from Microsoft, so i download version 2010

So now just open the project and press build right? Wrong! The project was made in Visual studio 6 and apparently you can not convert from visual studio 6 to visual studio 2010. According to the internets you need to first install Visual studio 2008, convert there, save it, open it in Visual studio 2010, convert, save and build.

Here is an overview for those that want to do this:

Start visual C++ 2008

Open Project - c:\temp\putty-cac-master\windows\MSVC\putty.dsw

Convert and open project

Choose File - Save All

Start visual C++ 2010 (and close 2008 ;-)

Open Project - c:\temp\putty-cac-master\windows\MSVC\putty.sln

You'll get a wizard: Next - Next - Finish

Now when you try to build it, it won't. You'll need to add a define 

Open c:\temp\putty-cac-master\windows\winstuff.sh

Add  #define SECURITY_WIN32 at the top of the file

If you compile now you'll get linking errors. You'll need to add 'sc.c' and 'capi.c' to the 'source files'

Now you're finally ready to build your binary. Press build and enjoy your own build putty.

To actually use your EID with this, just follow the CAPI instructions on http://www.risacher.org/putty-cac/

Stuff you'll need to get this working:

Microsoft Windows SDK for Windows 7 and .NET Framework 4

http://www.microsoft.com/en-us/download/details.aspx?id=8279

Visual 2008 express

http://download.microsoft.com/download/E/8/E/E8EEB394-7F42-4963-A2D8-29559B738298/VS2008ExpressWithSP1ENUX1504728.iso

Visual 2010 express

http://go.microsoft.com/?linkid=9709949

PuTTY CAC source

https://github.com/risacher/putty-cac/archive/master.zip

Good luck!

DHCPv6: ISC MAC logging and Cisco relay agent configuration

DHCPv6 server MAC address logging

One of the differences between DHCPv6 and DHCPv4 is that it uses DUID as an identifier instead of a MAC address.

As you probably know, DUID's are for the majority of OS (ie Windows) based on a timestamp suffixed by a MAC address

For some of our internal systems we use a MAC as an identifier, for now we will also be  needing this for IPv6. The default ISC DHCPv6 daemon isn't logging a MAC address by default. One way to have it print out is by adding this to your DHCP config.

option dhcp6.macaddr code 193 = string;
option dhcp6.leased-address code 194 = string;
option dhcp6.macaddr = binary-to-ascii(16, 8, ":", suffix(option dhcp6.client-id, 6));
option dhcp6.leased-address = binary-to-ascii(16,16, ":", substring(suffix(option dhcp6.ia-na, 24),0,16));
log (info, concat ("Lease for ",config-option dhcp6.leased-address, " leased to ", config-option dhcp6.macaddr));

Above code will only work for DUID-LLT and DUID-LL (so not DUID-EN, but I don't know anyone using this at the moment)

More info over DUID on http://tools.ietf.org/html/rfc6355

Other great blogpost about DHCPv6 at http://ipv6friday.org/blog/2011/12/dhcpv6/

DHCPv6 relay configuration on Cisco equipment

Overview of the configuration we're using on our routers.

(1) We're using FE80:: as our IPv6 default gateway everywhere, seems to work for me for now ;) any best practices for this? (update: because of issues with linux and fe80:: (linux responds to fe80:: if it's specified on any link, this has now changed to FE80::1)

(2) Asking the hosts on the subnet to not do SLAAC please, we're asking you nicely.

(3) Letting the hosts know we're managing the config, that they must use DHCPv6.

(4) Just to be sure also set this flag, tell them to use DHCPv6 not only for getting an IP but also for getting e.g. DNS servers

(5) We're the boss on this subnet.

(6) relay DHCPv6 requests to our server and use (7) the loopback as a source for this.

We're using link-local addresses for routing, so if we don't specifiy an source-interface, the relay agents would try to use a link-local address which obviously can not be routed.

Interface vlan42
  ipv6 address FE80::1 link-local (1)
  ipv6 address 2001:0db8:100:4200::1/64
  ipv6 nd prefix default 2592000 604800 no-autoconfig (2)
  ipv6 nd managed-config-flag (3)
  ipv6 nd other-config-flag (4)
  ipv6 nd router-preference High (5)
  ipv6 dhcp relay destination 2001:0db8:0:40::547:1 (6)
  ipv6 dhcp relay source-interface Loopback0 (7)

interface Loopback0
 ipv6 address  2001:0db8:300::63/128

IPv6 logging and Cisco NCS

So you want to deploy IPv6 on your wireless network. 

You want to use SLAAC and you want logging of those SLAAC addresses.

Straight from the horses mouth:

Q: What are IPv6 private addresses and why are they important to track?

A: Private (also known as temporary) addresses are randomly generated by the client when SLAAC address assignment is in use. These addresses are often rotated at a frequency of a day or so, as to prevent host traceability that would come from using the same host postfix (last 64 bits) at all times. It is important to track these private addresses for auditing purposes such as tracing copyright infringement. Cisco NCS records all IPv6 addresses in use by each client and historically logs them each time the client roams or establishes a new session. These records can be configured at NCS to be held for up to a year.

Pasted from <http://www.cisco.com/en/US/products/ps10315/products_tech_note09186a0080bae506.shtml#faqs>

BUT Cisco NCS or PI doesn't make it easy for you, it keeps the addresses in the database somewhere, but as soon as the client has disassociated you can't search for the IPv6 address anymore.  Disassociated IPv4 addresses can be searched though.

So far for IPv6 parity.

Radius to the rescue ? 

But wait, we've got radius accounting, right? You'll see the Framed-IP-Address attribute and this will show you the IPv6 address, right ? RIGHT ?

As we all know, IPv6 is a very new protocol (only about 15 years old), so of course there isn't support for IPv6 in the Framed-IP-Address attribute. There is a draft proposing Framed-IPv6-Address (also very new, only 3 years old).

(See http://tools.ietf.org/html/draft-ietf-radext-ipv6-access-16 for more info)

So, no searching, no IPv6 address logging by Radius.

So far for IPv6 parity again..

But according to the documentation (see above) cisco is recording those addresses somewhere ...

Reports to the rescue!

The workaround is reports:

Go to the report launch pad - Client - Client Sessions

Create a new report:

- report by SSID (or your own favorite source)

- reporting criteria (all SSIDs)

- reporting period select last 7 days

Customize the report, where you can find the most important data field: "Global Unique", this will show you the IPv6 address. Now you can schedule this report weekly and you've got weekly CSV files containing all the necessary information of the users.

If anyone got a better workaround please share!