DHCPv6: ISC MAC logging and Cisco relay agent configuration

DHCPv6 server MAC address logging

One of the differences between DHCPv6 and DHCPv4 is that it uses DUID as an identifier instead of a MAC address.

As you probably know, DUID's are for the majority of OS (ie Windows) based on a timestamp suffixed by a MAC address

For some of our internal systems we use a MAC as an identifier, for now we will also be  needing this for IPv6. The default ISC DHCPv6 daemon isn't logging a MAC address by default. One way to have it print out is by adding this to your DHCP config.

option dhcp6.macaddr code 193 = string;
option dhcp6.leased-address code 194 = string;
option dhcp6.macaddr = binary-to-ascii(16, 8, ":", suffix(option dhcp6.client-id, 6));
option dhcp6.leased-address = binary-to-ascii(16,16, ":", substring(suffix(option dhcp6.ia-na, 24),0,16));
log (info, concat ("Lease for ",config-option dhcp6.leased-address, " leased to ", config-option dhcp6.macaddr));

Above code will only work for DUID-LLT and DUID-LL (so not DUID-EN, but I don't know anyone using this at the moment)

More info over DUID on http://tools.ietf.org/html/rfc6355

Other great blogpost about DHCPv6 at http://ipv6friday.org/blog/2011/12/dhcpv6/

DHCPv6 relay configuration on Cisco equipment

Overview of the configuration we're using on our routers.

(1) We're using FE80:: as our IPv6 default gateway everywhere, seems to work for me for now ;) any best practices for this? (update: because of issues with linux and fe80:: (linux responds to fe80:: if it's specified on any link, this has now changed to FE80::1)

(2) Asking the hosts on the subnet to not do SLAAC please, we're asking you nicely.

(3) Letting the hosts know we're managing the config, that they must use DHCPv6.

(4) Just to be sure also set this flag, tell them to use DHCPv6 not only for getting an IP but also for getting e.g. DNS servers

(5) We're the boss on this subnet.

(6) relay DHCPv6 requests to our server and use (7) the loopback as a source for this.

We're using link-local addresses for routing, so if we don't specifiy an source-interface, the relay agents would try to use a link-local address which obviously can not be routed.

Interface vlan42
  ipv6 address FE80::1 link-local (1)
  ipv6 address 2001:0db8:100:4200::1/64
  ipv6 nd prefix default 2592000 604800 no-autoconfig (2)
  ipv6 nd managed-config-flag (3)
  ipv6 nd other-config-flag (4)
  ipv6 nd router-preference High (5)
  ipv6 dhcp relay destination 2001:0db8:0:40::547:1 (6)
  ipv6 dhcp relay source-interface Loopback0 (7)

interface Loopback0
 ipv6 address  2001:0db8:300::63/128

IPv6 logging and Cisco NCS

So you want to deploy IPv6 on your wireless network. 

You want to use SLAAC and you want logging of those SLAAC addresses.

Straight from the horses mouth:

Q: What are IPv6 private addresses and why are they important to track?

A: Private (also known as temporary) addresses are randomly generated by the client when SLAAC address assignment is in use. These addresses are often rotated at a frequency of a day or so, as to prevent host traceability that would come from using the same host postfix (last 64 bits) at all times. It is important to track these private addresses for auditing purposes such as tracing copyright infringement. Cisco NCS records all IPv6 addresses in use by each client and historically logs them each time the client roams or establishes a new session. These records can be configured at NCS to be held for up to a year.

Pasted from <http://www.cisco.com/en/US/products/ps10315/products_tech_note09186a0080bae506.shtml#faqs>

BUT Cisco NCS or PI doesn't make it easy for you, it keeps the addresses in the database somewhere, but as soon as the client has disassociated you can't search for the IPv6 address anymore.  Disassociated IPv4 addresses can be searched though.

So far for IPv6 parity.

Radius to the rescue ? 

But wait, we've got radius accounting, right? You'll see the Framed-IP-Address attribute and this will show you the IPv6 address, right ? RIGHT ?

As we all know, IPv6 is a very new protocol (only about 15 years old), so of course there isn't support for IPv6 in the Framed-IP-Address attribute. There is a draft proposing Framed-IPv6-Address (also very new, only 3 years old).

(See http://tools.ietf.org/html/draft-ietf-radext-ipv6-access-16 for more info)

So, no searching, no IPv6 address logging by Radius.

So far for IPv6 parity again..

But according to the documentation (see above) cisco is recording those addresses somewhere ...

Reports to the rescue!

The workaround is reports:

Go to the report launch pad - Client - Client Sessions

Create a new report:

- report by SSID (or your own favorite source)

- reporting criteria (all SSIDs)

- reporting period select last 7 days

Customize the report, where you can find the most important data field: "Global Unique", this will show you the IPv6 address. Now you can schedule this report weekly and you've got weekly CSV files containing all the necessary information of the users.

If anyone got a better workaround please share!

Cisco Prime Infrastructure NCS 1.2 and the Oracle issue

Continuing my story about NCS diskspace increase.

A few weeks everything went fine, until one day the webserver or the shell wasn't accessible anymore.
In the vmware server console we saw that all of the memory was in use (8GB). So we add some extra memory and rebooted the system. (At that time I didn't made the correlation with increasing the disk space a few weeks ago)

This didn't solve the problem.

But I could log in and memory usage seemed normal. First I went looking in /var/log , you can skip this step, nothing of value can be found here.

The interesting directories are /opt/CSCOlumos/logs/

Last file modified was hm-0-0.log. Containing:

ERROR [system] [HealthMonitorServer] HealthMonitorServer.initHealthMonitor: initHealthMonitor(): can not start DB
INFO  [database] [Thread-17] OracleSchemaUtil dbServerUp(): wcs errorCode = 1034

(Other logfiles were also giving oracle errors)
Closing in on the oracle DB.

Next target /opt/oracle (disclaimer: I know nothing about Oracle), after some searching I discovered the logfile /opt/oracle/diag/rdbms/wcs/wcs/trace/alert_wcs.log which seemed interesting.

ORA-19815: WARNING: db_recovery_file_dest_size of 107374182400 bytes is 100.00% used, and has 0 remaining bytes available.
ORA-19809: limit exceeded for recovery files
ORA-16038: log 1 sequence# 1018 cannot be archived
ORACLE Instance wcs - Archival Error
ARCH: Archival stopped, error occurred. Will continue retrying

This didn't look good. Apparently Oracle has it own internal idea of how big a disk is and how much free space is available. Because we still had 130Gb available.

/dev/mapper/smosvg-optvol
                     249204396  97436636 138909244  42% /opt

With the help of google and a colleague with the necessary oracle knowledge. This was the solution (increasing the recovery db size):

[root@ncs oracle]# su - oracle
[oracle@ncs ~]$ ls
base  coracleenv  dbPasswd.pwd  oracleenv  oraInventory  templates  utils
[oracle@ncs ~]$ . oracleenv
[oracle@ncs ~]$ sqlplus '/as sysdba'
SQL*Plus: Release 11.2.0.2.0 Production 
Copyright (c) 1982, 2010, Oracle.  All rights reserved.
Connected to an idle instance.

SQL> startup nomount
ORACLE instance started.

Total System Global Area 4275781632 bytes
Fixed Size                  2233336 bytes
Variable Size            2986347528 bytes
Database Buffers         1275068416 bytes
Redo Buffers               12132352 bytes
SQL> show parameter db_recovery_file_dest_size

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_recovery_file_dest_size           big integer 100G
SQL> alter system set db_recovery_file_dest_size=120G scope=both;

System altered.

SQL> show parameter db_recovery_file_dest_size;

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_recovery_file_dest_size           big integer 120G
SQL> shutdown
ORA-01507: database not mounted


ORACLE instance shut down.
SQL> quit

Still the error, but now some free space to do something

ORA-19815: WARNING: db_recovery_file_dest_size of 128849018880 bytes is 85.07% used, and has 19241095680 remaining bytes available.

Next ran ncs cleanup on the NCS CLI

ncs/admin# ncs cleanup
===================================================
Starting Cleanup: 
===================================================
Removing all files in backup staging directory
Removing all Matlab core related files
Removing all older log files
Cleaning older archive logs
Cleaning database backup and all archive logs
Cleaning database
Stopping database
Starting database
Starting database clean
Completed database clean
Stopping database
===================================================
Completed Cleanup
===================================================
ncs/admin#

And rebooted the system.

Oracle DB came up normally and NCS was happily running again.

Increasing diskspace on Cisco Prime Infrastructure NCS 1.2 "the hard way?"

Somewhere last year PI NCS 1.2 was constantly complaining about disk full errors, although the the disk was only 62% used.

Apparently the complaining starts above 60% because it needs to be able to make a temporary backup on the same disk as it is running the database.

Our datacenter guy increased the disk (200GB) in vmware by increasing it by 100GB.

I wrongly thought this would have fixed the problem and that the system would automatically resize everything.

I was wrong.

So after running root_enable it was possible to login as root (it's just running a linux beneath it)

Below you can see the layout of the system with 62% in use for /opt where the database and temporary backups are located.

Filesystem           1K-blocks      Used Available Use% Mounted on

/dev/mapper/smosvg-rootvol
                       1967952    477860   1388512  26% /
/dev/mapper/smosvg-tmpvol
                       1967952     36460   1829912   2% /tmp
/dev/sda3               988116     17764    919348   2% /storedconfig
/dev/mapper/smosvg-recvol
                         95195      5664     84616   7% /recovery
/dev/mapper/smosvg-home
                         95195      5668     84612   7% /home
/dev/mapper/smosvg-optvol
                     147630356  86745492  53264668  62% /opt
/dev/mapper/smosvg-usrvol
                       5935604    904020   4725204  17% /usr
/dev/mapper/smosvg-varvol
                       1967952     90372   1776000   5% /var
/dev/mapper/smosvg-storeddatavol
                       3967680     74320   3688560   2% /storeddata
/dev/mapper/smosvg-altrootvol
                         95195      5664     84616   7% /altroot
/dev/mapper/smosvg-localdiskvol
                      29583412    204132  27852292   1% /localdisk
/dev/sda1               101086     12713     83154  14% /boot
tmpfs                  4021728   2044912   1976816  51% /dev/shm

How to fix this manually.
So we run fdisk, and we see that the extra 100GB is detected (/dev/sda 322.1GB)
We then add another (4) primary partition which consists of the missing 100GB

# fdisk /dev/sda

The number of cylinders for this disk is set to 39162.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): p

Disk /dev/sda: 322.1 GB, 322122547200 bytes
255 heads, 63 sectors/track, 39162 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          13      104391   83  Linux
/dev/sda2              14       25368   203664037+  8e  Linux LVM
/dev/sda3           25369       25495     1020127+  83  Linux

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)

p
Selected partition 4
First sector (409577175-629145599, default 409577175):
Using default value 409577175
Last sector or +size or +sizeM or +sizeK (409577175-629145599, default 629145599):
Using default value 629145599

Command (m for help): p

Disk /dev/sda: 322.1 GB, 322122547200 bytes
255 heads, 63 sectors/track, 39162 cylinders, total 629145600 sectors
Units = sectors of 1 * 512 = 512 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *          63      208844      104391   83  Linux
/dev/sda2          208845   407536919   203664037+  8e  Linux LVM
/dev/sda3       407536920   409577174     1020127+  83  Linux
/dev/sda4       409577175   629145599   109784212+  83  Linux

Also tag it 8e to be a linux LVM filesystem.
And write the partition table <scary>

Command (m for help): t
Partition number (1-4): 4
Hex code (type L to list codes): 8e
Changed system type of partition 4 to 8e (Linux LVM)

Command (m for help): p

Disk /dev/sda: 322.1 GB, 322122547200 bytes
255 heads, 63 sectors/track, 39162 cylinders, total 629145600 sectors
Units = sectors of 1 * 512 = 512 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *          63      208844      104391   83  Linux
/dev/sda2          208845   407536919   203664037+  8e  Linux LVM
/dev/sda3       407536920   409577174     1020127+  83  Linux
/dev/sda4       409577175   629145599   109784212+  8e  Linux LVM

Command (m for help): v
62 unallocated sectors

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.

Now reboot the system.
When it boots up, the partition can be used.
Use pvcreate to create the new physical volume.
vgdisplay will still show the old 194.22GB because it's not added yet to smosvg.

# pvcreate /dev/sda4
  Physical volume "/dev/sda4" successfully created

# vgdisplay
  --- Volume group ---
  VG Name               smosvg
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  12
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                11
  Open LV               11
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               194.22 GB
  PE Size               32.00 MB
  Total PE              6215
  Alloc PE / Size       6215 / 194.22 GB
  Free  PE / Size       0 / 0
  VG UUID               vJcFNu-qPIP-uoKY-KiIu-wKCd-Jj7l-e8cLaV

pvdisplay will show the 2 volumes.

# pvdisplay
  --- Physical volume ---
  PV Name               /dev/sda2
  VG Name               smosvg
  PV Size               194.23 GB / not usable 10.66 MB
  Allocatable           yes (but full)
  PE Size (KByte)       32768
  Total PE              6215
  Free PE               0
  Allocated PE          6215
  PV UUID               ugQzvR-HsdX-cK8u-Sylm-NYAm-FXFX-YXkb6R

  --- Physical volume ---
  PV Name               /dev/sda4
  VG Name               smosvg
  PV Size               104.70 GB / not usable 11.15 MB
  Allocatable           yes
  PE Size (KByte)       32768
  Total PE              3350
  Free PE               3350
  Allocated PE          0
  PV UUID               pEPFVV-AI3e-HLfo-fy5i-TyEN-eRCp-BJiXPS

Now add this volume to /opt and extend it by 50Gb And use resize2fs so that the kernel nows the size changed.

# lvextend /dev/mapper/smosvg-optvol /dev/sda4
# lvextend -L +50G /dev/mapper/smosvg-optvol
  Extending logical volume optvol to 195.34 GB
  Logical volume optvol successfully resized

# resize2fs /dev/mapper/smosvg-optvol
resize2fs 1.39 (29-May-2006)
Filesystem at /dev/mapper/smosvg-optvol is mounted on /opt; on-line resizing required
Performing an on-line resize of /dev/mapper/smosvg-optvol to 51208192 (4k) blocks.
The filesystem on /dev/mapper/smosvg-optvol is now 51208192 blocks long.

A df will noshow 47% in use for /opt
The annoying popups are now gone.

# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/smosvg-rootvol
                       1967952    477876   1388496  26% /
/dev/mapper/smosvg-tmpvol
                       1967952     36460   1829912   2% /tmp
/dev/sda3               988116     17764    919348   2% /storedconfig
/dev/mapper/smosvg-recvol
                         95195      5664     84616   7% /recovery
/dev/mapper/smosvg-home
                         95195      5668     84612   7% /home
/dev/mapper/smosvg-optvol
                     198417376  88100224 100077796  47% /opt
/dev/mapper/smosvg-usrvol
                       5935604    904020   4725204  17% /usr
/dev/mapper/smosvg-varvol
                       1967952     88744   1777628   5% /var
/dev/mapper/smosvg-storeddatavol
                       3967680     74320   3688560   2% /storeddata
/dev/mapper/smosvg-altrootvol
                         95195      5664     84616   7% /altroot
/dev/mapper/smosvg-localdiskvol
                      29583412    204132  27852292   1% /localdisk
/dev/sda1               101086     12713     83154  14% /boot
tmpfs                  4021728   2044888   1976840  51% /dev/shm

It seems like I fixed everything and that I was in the clear.

Which it was for a couple of weeks until ... to be continued in a later post.