my journal of tech related projects, ideas, etc....

Octopress and the Eudyptula Challenge

I have been really slack with updating my Blog. Three posts last year and this is my first post this year. I have been quite busy with a new job, getting used to a new city, etc. I don’t feel too guilty.


A couple of weekends ago I converted my blog from Wordpress to Octopress. Octopress is a static blog generator which uses Jekyll. The conversion went fairly smoothly. I used exitwp to export my posts and switched to Disqus for comments. There are some formatting issues I need to fix, mainly with code blocks and links, however overall things look pretty good. I like being able to use markdown for my posts.

Octopress has a development server builtin which makes previewing content easy. Just run rake preview and this will spawn a Webrick server that watches your source files for changes and automatically regenerates the preview when required.

Apart from cleaning up the imported posts I would like to look at tweaking the theme a little. The default theme is quite nice and looks great on any size screen though it would be nice to make it a little distinctive.

Eudyptula Challenge

I came across the Eudyptula Challenge on Greg K. Hartman’s Google+ page. Not sure who is organising it though I am finding it fun. Unfortunately my C is quite rusty1. I’m currently on Task 15 and have been busy compiling kernels, writing modules and kernel patches. Kernel patches have included adding files to /proc/<PID>/ and adding a new syscall.

For one of the tasks I found myself wanting to be able to quickly boot a kernel with a minimal initramfs for testing. I wrote a couple of simple wrapper scripts for generating the initramfs and booting the kernel using kvm with console output going directly into the terminal for easy cut and paste. These scripts were based on a blog post by Stefan Hajnoczi and can be found here.

  1. Rusty as in, I am out of practice. Not that it looks like it was coded by Rusty.

Slow Network Performance to KVM Virtual Machine.

I’m currently using a KVM virtual machine as my primary file/media server. Since I have been using a virtual machine as my file server I have witnessed strange stalls in media playback when accessing media files via a NFS share. Media files would take a couple of seconds to load and occasionally playback would stall for anything from a second to 20 seconds. In addition scp transfers would also occasionally stall. With not much to go on I turned to Google which did turn up an old Redhat bug report from 20091. One of the solutions was to disable TCP offload in the KVM guest via:

mfs@wvm1 # ethtool -K eth0 tx off

Even though this was an old bug report, once I did this the stalls went away. Media playback is perfect. I haven’t tried any scp transfers though I am confident they will now complete without stalling as well. For the time being I will be turning off TCP offload on all of my KVM guests.

Upgrading Debian Squeeze to Wheezy on an ALIX2.

The last machine on my home network still running Debian Squeeze was my PC Engines ALIX2 based router. The ALIX2 doesn’t have any video output though it does send all bios output to the serial port. I also configured grub, the kernel and getty to all use the serial console for input/output by following the instructions in the Remote Serial Console HOWTO. While I did make sure I had a USB-to-serial adapter handy, I didn’t require it and managed to complete the upgrade via SSH.

I followed the Debian Wheezy upgrade instructions without incident. The only tricky item was the kernel:

Debian’s 686 kernel configuration has been replaced by the 686-pae configuration, which uses PAE (“Physical Address Extension”). If your computer is currently running the 686 configuration but does not have PAE, you will need to switch to the 486 configuration instead.

You can check for PAE support by checking /proc/cpuinfo. The Debian instructions provide a simple grep command you can copy and paste to look for the correct flag. The ALIX2 doesn’t support PAE so I removed the installed kernel and replaced it with kernel-image-486 as instructed. The grub config was automatically updated though I did double check. You can run update-grub to update grub if required.

After a reboot the system came up with the new kernel running Debian Wheezy. Debian Wheezy is still a couple of days away from official release however I have been running it for weeks on a laptop, a kvm host and a bunch of virtual machines without any issues.

Installing Bitmap Fonts on Fedora 17/18

For the second time in a month I have had to go searching to find how to install bitmap fonts on Fedora 17/18. This is covered in the Fedora documentation for older versions of Fedora though appears to have been dropped at some point. To install bitmap fonts do the following:

  1. Create a directory under /usr/share/fonts/ e.g. /usr/share/fonts/local

  2. Copy font files to this directory and run mkfontdir and mkfontscale

  3. Create a symlink /etc/X11/fontpath.d/local –> /usr/share/fonts/local

BTRFS Fileserver


For the last 9 months or so I have been using a HP N36L Microserver as a home Xen server running a mail relay for my local network. It wasn’t doing much else and I figured it was time to rectify that. I wanted to turn it into a backup file server as well as be able to use it as a test bench for things like lxc. The server comes standard with one sata drive with room for three more and a gig of ram. I upgraded the RAM to the maximum of 8GB and filled the drive bays with four 1TB drives I scrounged up. Two of the drives were canabalized from a couple of cheap USB 2 external drives. The server also has two PCIe half height slots so I filled them with an extra gigabit NIC and a USB 3 card.

I decided I wanted to use btrfs as the main filesystem. For anyone else thinking of using btrfs at this stage keep in mind that btrfs is still under heavy development. The btrfs wiki states:

Btrfs is under heavy development, but every effort is being made to keep the filesystem stable and fast. Because of the speed of development, you should run the latest kernel you can (either the latest release kernel from, or the latest -rc kernel.

For this reason I will be using the latest rc kernel and the latest btrfs-progs. For something different I went with Funtoo Linux as the operating system. I had installed and used Gentoo several years ago and had always wanted to check out Funtoo. I used sysrescuecd as my install medium booted from an external USB flash drive.

Arch Linux Net-tools Depreciated.

Arch Linux has depreciated the usage of net-tools in /etc/rc.conf in favor of iproute2. The new syntax in /etc/rc.conf makes it simple to setup a single interface though anything complicated is best done through netcfg. I didn’t really want to use netcfg for my main workstation as it’s network setup never changes and it seemed like overkill. Unfortunately I did need to configure IPv6 related settings and this was no longer easily done in rc.conf. The two items I needed to set were the IPv6 address and the interface MTU. (If I don’t set an MTU of 1280-1480 I have trouble with my IPv6 tunnel.)

I decided to remove the IPv6 address setting completely and rely on radvd running on my router. This took care of my workstation’s IPv6 address and default route. I really should have done this earlier.

I started patching the /etc/rc.d/network script to take an mtu variable but started to have second thoughts. Most users would not need it and it seemed a shame to complicate the network_up() function. There was a good chance the patch would not be accepted. In the end I just added the line:

ip link set dev eth0 mtu 1280

to /etc/rc.local. This worked fine.


Picked up a OCZ Vertex 2 160GB SSD recently with plans to use it as a boot/system drive in my main workstation. Unfortunately things did not go well. Partitioning the disk seemed to work fine using gdisk and I created an ext4 file system on it though it took a while to complete. Trying to mount the file system resulted in multiple errors of different types and trying to read the partition table also gave garbage output. A quick Google showed a few people with similar issues with SSDs when used with the on board SATA ports on my motherboard, an ASUS M2N which uses an nVidia nForce 430 MCP chipset. The board is quite dated by todays standards but hasn’t given me any trouble in the past.

I removed the SSD and placed it in a spare machine fitted with an Adapted AAR 1210 SATA controller. This time the drive performed much better. The partition table appeared fine and the creation of the ext4 file system was much faster. A run of bonnie++ completed with multiple errors when the file system was mounted with the “discard” option to make use of TRIM. Mounting the file system without the “discard” option resulted in a bonnie++ run without errors.

None of my machines have AHCI support which I would have liked to test. I’m not sure if AHCI is required for TRIM support or not. I’m guessing the drive is fine and I’m just having issues due to the age of my hardware. I’ll test the SSD on a more recent AHCI machine at work tomorrow. I may need to invest in a AHCI PCIe SATA controller for my main workstation or replace the motherboard.

It appears that SSDs are not always a simple drop in replacement for HDDs.


Recently replaced lighttpd with nginx on Debian squeeze. Used spawn-fcgi to manage PHP as php-fpm is not in squeeze yet. Had a little trouble with reliability until I set a couple of environment variables before spawning the PHP processes. Since using the script below to start php all has been well.

/usr/bin/spawn-fcgi -s /tmp/cgi-php.socket -u www-data -g www-data \
                    -P /var/run/ \
                    -- /usr/bin/php5-cgi

Next I would like to look at setting up varnish in front of nginx.


Having recently watched a couple of talks centered around the current state of IPv4 address space exhaustion I decided to look into setting up an IPv6 tunnel to my home network as my ISP does not seem to offer native IPv6 yet. I checked out a few of the free tunnel brokers available and settled on Hurricane Electric. As someone who knew nothing about IPv6 I found HE’s website to be a helpful resource. Another resource I found helpful was The Second Internet which is a freely available ebook. I also referred to the Linux IPv6 HOWTO. I enabled IPv6 in a couple of steps:

  1. Router.

  2. LAN

  3. Extras (e.g. Certification Tests and Enable Google IPv6 services)


Router setup was relatively straight forward. First use HE’s web interface to register an account and create a tunnel. You will need to know your local static IPv4 address for this. Once the tunnel appears in your account you can retrieve a list of commands to activate it on various platforms in the dropdown box at the bottom of the page. Linux users have two choices: Linux-route2 or Linux-net-tools. I chose Linux-route2 and simply copied the generated commands. As my router is behind a NAT device I had to replace the local IPv4 address with the address of the NAT device. The NAT device must also forward protocol 41 through to your endpoint. I had no trouble with a consumer level D-Link device. HE mentions both of these items in a bullet point.

Once these commands are executed I was able to use ping6 to ping the tunnel endpoints as well as IPv6 hosts on the internet.

$ ping6 -c 2
PING 56 data bytes
64 bytes from 2404:6800:8004::68: icmp_seq=1 ttl=54 time=473 ms
64 bytes from 2404:6800:8004::68: icmp_seq=2 ttl=54 time=473 ms

--- ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 473.126/473.322/473.519/0.715 ms


HE supplies a routed /64 network to use as your LAN subnet. If you require multiple subnets you can request a /48. I only required one subnet so I did not request the /48. As I only wanted one LAN client to have an IPv6 address while I am testing I did not use DHCP6 or radvd and just setup the interfaces manually. This step is not much different to manually setting up an IPv4 lan client except for the IPv6 addresses.

On the router you have to enable IPv6 packet forwarding and give the LAN interface one of the routed /64 IPv6 addresses:

# echo 1 > /proc/sys/net/ipv6/conf/all/forwarding
# ip addr add <<routed /64 prefix>>::1/64 dev eth1

On the LAN client you have to give the LAN interface one of the routed /64 addresses and set a default route:

# ip addr add <<routed /64 prefix>>::2/64 dev eth0
# ip route add ::/0 <<routed /64 prefix>>::1

Once this is done you should have IPv6 connectivity from the LAN client.


Is it working?

While it is easy to check that IPv6 is working via testing with ping6, traceroute6, dig, etc, I wanted to be able to check at a glance how much traffic was going through the IPv6 tunnel. Two tools I was already using on the router were handy to monitor IPv6 usage: bwm-ng and vnstat. bwm-ng gives you a snapshot of throughput of all interfaces while vnstat tracks usage over time.

bwm-ng v0.6 (probing every 0.500s), press 'h' for help
  input: /proc/net/dev type: rate
  /         iface                   Rx                   Tx                Total
               lo:           0.00 KB/s            0.00 KB/s            0.00 KB/s
             eth0:           0.00 KB/s            0.00 KB/s            0.00 KB/s
             eth1:           0.13 KB/s            0.31 KB/s            0.44 KB/s
             eth2:           0.00 KB/s            0.00 KB/s            0.00 KB/s
          he-ipv6:           0.00 KB/s            0.00 KB/s            0.00 KB/s
            total:           0.13 KB/s            0.31 KB/s            0.44 KB/s

$ vnstat -d -i he-ipv6

 he-ipv6  /  daily

         day         rx      |     tx      |    total    |   avg. rate
      02/10/11         0 KiB |       0 KiB |       0 KiB |    0.00 kbit/s
      02/11/11      6.97 MiB |    1.79 MiB |    8.76 MiB |    0.83 kbit/s
      02/12/11     91.47 MiB |    9.01 MiB |  100.48 MiB |    9.53 kbit/s
      02/13/11     91.89 MiB |   10.88 MiB |  102.78 MiB |    9.74 kbit/s
      02/14/11     10.17 MiB |    3.26 MiB |   13.43 MiB |    1.52 kbit/s
     estimated        11 MiB |       3 MiB |      14 MiB |

Google over IPv6

While Google allows access to Google Search on IPv6 other services are not served over IPv6 by default. Access to Gmail, etc over IPv6 normally requires registering with Google however thanks to HE you just need to use the supplied HE DNS resolvers to automatically gain access to Google’s IPv6 services.


HE offers an IPv6 certification test which takes you through different aspects of IPv6 configuration. I found this fun and educational and learnt quite a bit about IPv6 and DNS.


  • Having your LAN devices pop up on the internet has obvious security implications. Before enabling IPv6 for all LAN devices I will need to add IPv6 support to my firewall scripts. For the moment I have just made sure that services are only listening on IPv4 sockets and not IPv6.

  • Once security is taken care of I will install radvd and/or DHCP6 to take care of the LAN side of things.

  • At the moment I have to create the tunnel manually when I reboot my router. I should add the required stanza to /etc/network/interfaces to automate this.

GPT RAID1 Setup.

My home partition hit 90% recently and I decided it was time to upgrade my storage. I had been running 4x250GB hard disks in a RAID1/RAID10 setup and switched to a 2x1TB RAID1 setup. The drives were Western Digital Advanced Format drives with 4kB physical sectors and 512B logical sectors. It is important when partitioning these drives to ensure that partitions start on a logical sector number that is divisible by 8 otherwise performance will suffer as file system clusters will not be aligned to the underlying 4kB physical sector. If the partition is not aligned then writes turn into a read-modify-write. I also decided to use a GPT instead of an MBR partition table. I used gdisk to create boot, swap, root and home partitions:

$ sudo sgdisk -p /dev/sda
Disk /dev/sda: 1953525168 sectors, 931.5 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 0658FE0C-AE06-4FD9-8CCC-7DDB178BB0CA
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 1953525134
Partitions will be aligned on 2048-sector boundaries
Total free space is 3437 sectors (1.7 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048          264191   128.0 MiB   FD00  Linux RAID
   2          264192        12847103   6.0 GiB     8200  Linux swap
   3        12847104       147064831   64.0 GiB    FD00  Linux RAID
   4       147064832      1953523711   861.4 GiB   FD00  Linux RAID

I actually aligned the start of all my partitions to 1MiB (2048 sectors) which seems to be somewhat of an industry standard and was suggested by gdisk as well.

I had done some research into how to exactly boot Linux on a GPT disk and was happy to find that my boot loader, extlinux, works out of the box. Installing extlinux was as simple as:

# extlinux --raid --install /boot/extlinux
# sgdisk /dev/sda --attributes=1:set:2
# sgdisk /dev/sdb --attributes=1:set:2
# cat /usr/lib/syslinux/gptmbr.bin > /dev/sda
# cat /usr/lib/syslinux/gptmbr.bin > /dev/sdb

These commands install ldlinux.sys, set the bootable attribute for partition 1 of both disks and install the gptmbr.bin boot code into the first sector of both disks.

Once my file systems were copied onto my new disk setup I recreated my initrd images with a new /etc/mdadm.conf and edited /etc/fstab and /boot/extlinux/extlinux.conf. The Arch Linux Wiki article Installing with Software RAID or LVM proved handy for assembling the arrays.

I now have plenty of disk space again:

$ pydf / /boot /home
Filesystem Size Used Avail Use%                               Mounted on
/dev/md1    63G  31G   29G 48.6 [#############..............] /         
/dev/md0   124M  17M  101M 13.6 [####.......................] /boot     
/dev/md2   861G 373G  488G 43.3 [############...............] /home

A quick check with Bonnie++ showed that disk performance was as expected which confirmed that I hadn’t made any mistakes with partition alignment.