my journal of tech related projects, ideas, etc....

CentOS 7 and XFS

CentOS 7 has been out for a while now and brings a couple of notable new features. Two of the main changes to previous versions is systemd and XFS as the default file system. If you are like me you probably just did a default install of CentOS 7 to see what it is like. If you find you like it you may want to adjust the partitioning/LVM layout from the default after the fact. The default partitioning scheme is to allocate half the disk to the home LV and the other half of the disk to the root LV. In my case I didn’t want a separate home LV and I wanted to shrink the root LV. Removing the home LV is easy:

# umount /home
# lvremove /dev/centos/home

In addition, remove the /etc/fstab entry for /home.

Shrinking the root LV is a little more difficult as, unlike ext4, it is not possible to shrink an XFS file system, mounted or otherwise. What we can do is create a new root LV and copy the existing root file system to it. First create our new LV and format it:

# lvcreate -L 8G centos -n root_new
# mkfs.xfs /dev/centos/root_new

While it is possible to copy the root file system while the system is live, it is much easier to do this next step from a livecd such as SystemRescue CD.

First mount the file systems:

# mkdir /mnt/{root,root_new}
# mount /dev/centos/root /mnt/root
# mount /dev/centos/root_new /mnt/root_new

Then copy the root file system to the new LV. Note the flags passed to rsync! -A and -X ensure that ACLs and extended attributes are copied. Without the later you will have issues with SELinux.

# rsync -axvAX /mnt/root/ /mnt/root_new

Then unmount the file systems and move the original root LV out of the way and rename the new root LV:

# umount /mnt/root
# umount /mnt/root_new
# lvrename centos root root_old
# lvrename centos root_new root

You should now be able to reboot. Delete the root_old LV once your are confident your new root file system is complete and functioning correctly.

Enabling SPDY in Nginx on Debian Wheezy

The default nginx packages on Debian Wheezy are 1.2.1 which don’t include SPDY support. There are a couple of options here:


Install nginx from source

This is an option though less than ideal. I prefer to have all software installed and managed using apt. If you want to go this way there are several howtos available as well as nginx’s own documentation. One advantage to installing from source is that you can tailor the build to include only the features you require.

Use the wheezy-backports repo

Debian backports are a solid option. This is an official repo and will give you access to nginx 1.4.6. If you simply want a version of nginx that includes SPDY support this is probably the best option.

Use the dotdeb repo

Dotdeb are a third party repo that provide updated packages common in LAMP setups. They currently have nginx 1.4.7 available.

Use the nginx repo

Nginx also provides deb packages for the stable and mainline nginx builds. If you don’t just want SPDY support and either require or prefer a later version this will give you the cutting edge. Currently nginx 1.6 and 1.7 are available direct from nginx.

Decisions, Decisions

To add SPDY support to this site I used the nginx repos. This is a non production site and I’m willing to use cutting edge software. One thing to watch when using the nginx repos is that the included /etc/nginx/nginx.conf does not include the line include /etc/nginx/sites-enabled/*;. If you are using the sites-{available,enabled} directories, backup your /etc/nginx/nginx.conf file or add this line in manually. If you choose to backup your existing config note that it may require tweaking to get it working in the new version of nginx.

Once you have a SPDY capable version of nginx installed, enabling SPDY is a matter of modifying your listen directives:

server {
    listen 443 ssl spdy;

To check that SPDY is working use

Octopress and the Eudyptula Challenge

I have been really slack with updating my Blog. Three posts last year and this is my first post this year. I have been quite busy with a new job, getting used to a new city, etc. I don’t feel too guilty.


A couple of weekends ago I converted my blog from Wordpress to Octopress. Octopress is a static blog generator which uses Jekyll. The conversion went fairly smoothly. I used exitwp to export my posts and switched to Disqus for comments. There are some formatting issues I need to fix, mainly with code blocks and links, however overall things look pretty good. I like being able to use markdown for my posts.

Octopress has a development server builtin which makes previewing content easy. Just run rake preview and this will spawn a Webrick server that watches your source files for changes and automatically regenerates the preview when required.

Apart from cleaning up the imported posts I would like to look at tweaking the theme a little. The default theme is quite nice and looks great on any size screen though it would be nice to make it a little distinctive.

Eudyptula Challenge

I came across the Eudyptula Challenge on Greg K. Hartman’s Google+ page. Not sure who is organising it though I am finding it fun. Unfortunately my C is quite rusty1. I’m currently on Task 15 and have been busy compiling kernels, writing modules and kernel patches. Kernel patches have included adding files to /proc/<PID>/ and adding a new syscall.

For one of the tasks I found myself wanting to be able to quickly boot a kernel with a minimal initramfs for testing. I wrote a couple of simple wrapper scripts for generating the initramfs and booting the kernel using kvm with console output going directly into the terminal for easy cut and paste. These scripts were based on a blog post by Stefan Hajnoczi and can be found here.

  1. Rusty as in, I am out of practice. Not that it looks like it was coded by Rusty.

Slow Network Performance to KVM Virtual Machine.

I’m currently using a KVM virtual machine as my primary file/media server. Since I have been using a virtual machine as my file server I have witnessed strange stalls in media playback when accessing media files via a NFS share. Media files would take a couple of seconds to load and occasionally playback would stall for anything from a second to 20 seconds. In addition scp transfers would also occasionally stall. With not much to go on I turned to Google which did turn up an old Redhat bug report from 20091. One of the solutions was to disable TCP offload in the KVM guest via:

mfs@wvm1 # ethtool -K eth0 tx off

Even though this was an old bug report, once I did this the stalls went away. Media playback is perfect. I haven’t tried any scp transfers though I am confident they will now complete without stalling as well. For the time being I will be turning off TCP offload on all of my KVM guests.

Upgrading Debian Squeeze to Wheezy on an ALIX2.

The last machine on my home network still running Debian Squeeze was my PC Engines ALIX2 based router. The ALIX2 doesn’t have any video output though it does send all bios output to the serial port. I also configured grub, the kernel and getty to all use the serial console for input/output by following the instructions in the Remote Serial Console HOWTO. While I did make sure I had a USB-to-serial adapter handy, I didn’t require it and managed to complete the upgrade via SSH.

I followed the Debian Wheezy upgrade instructions without incident. The only tricky item was the kernel:

Debian’s 686 kernel configuration has been replaced by the 686-pae configuration, which uses PAE (“Physical Address Extension”). If your computer is currently running the 686 configuration but does not have PAE, you will need to switch to the 486 configuration instead.

You can check for PAE support by checking /proc/cpuinfo. The Debian instructions provide a simple grep command you can copy and paste to look for the correct flag. The ALIX2 doesn’t support PAE so I removed the installed kernel and replaced it with kernel-image-486 as instructed. The grub config was automatically updated though I did double check. You can run update-grub to update grub if required.

After a reboot the system came up with the new kernel running Debian Wheezy. Debian Wheezy is still a couple of days away from official release however I have been running it for weeks on a laptop, a kvm host and a bunch of virtual machines without any issues.

Installing Bitmap Fonts on Fedora 17/18

For the second time in a month I have had to go searching to find how to install bitmap fonts on Fedora 17/18. This is covered in the Fedora documentation for older versions of Fedora though appears to have been dropped at some point. To install bitmap fonts do the following:

  1. Create a directory under /usr/share/fonts/ e.g. /usr/share/fonts/local

  2. Copy font files to this directory and run mkfontdir and mkfontscale

  3. Create a symlink /etc/X11/fontpath.d/local –> /usr/share/fonts/local

BTRFS Fileserver


For the last 9 months or so I have been using a HP N36L Microserver as a home Xen server running a mail relay for my local network. It wasn’t doing much else and I figured it was time to rectify that. I wanted to turn it into a backup file server as well as be able to use it as a test bench for things like lxc. The server comes standard with one sata drive with room for three more and a gig of ram. I upgraded the RAM to the maximum of 8GB and filled the drive bays with four 1TB drives I scrounged up. Two of the drives were canabalized from a couple of cheap USB 2 external drives. The server also has two PCIe half height slots so I filled them with an extra gigabit NIC and a USB 3 card.

I decided I wanted to use btrfs as the main filesystem. For anyone else thinking of using btrfs at this stage keep in mind that btrfs is still under heavy development. The btrfs wiki states:

Btrfs is under heavy development, but every effort is being made to keep the filesystem stable and fast. Because of the speed of development, you should run the latest kernel you can (either the latest release kernel from, or the latest -rc kernel.

For this reason I will be using the latest rc kernel and the latest btrfs-progs. For something different I went with Funtoo Linux as the operating system. I had installed and used Gentoo several years ago and had always wanted to check out Funtoo. I used sysrescuecd as my install medium booted from an external USB flash drive.

Arch Linux Net-tools Depreciated.

Arch Linux has depreciated the usage of net-tools in /etc/rc.conf in favor of iproute2. The new syntax in /etc/rc.conf makes it simple to setup a single interface though anything complicated is best done through netcfg. I didn’t really want to use netcfg for my main workstation as it’s network setup never changes and it seemed like overkill. Unfortunately I did need to configure IPv6 related settings and this was no longer easily done in rc.conf. The two items I needed to set were the IPv6 address and the interface MTU. (If I don’t set an MTU of 1280-1480 I have trouble with my IPv6 tunnel.)

I decided to remove the IPv6 address setting completely and rely on radvd running on my router. This took care of my workstation’s IPv6 address and default route. I really should have done this earlier.

I started patching the /etc/rc.d/network script to take an mtu variable but started to have second thoughts. Most users would not need it and it seemed a shame to complicate the network_up() function. There was a good chance the patch would not be accepted. In the end I just added the line:

ip link set dev eth0 mtu 1280

to /etc/rc.local. This worked fine.


Picked up a OCZ Vertex 2 160GB SSD recently with plans to use it as a boot/system drive in my main workstation. Unfortunately things did not go well. Partitioning the disk seemed to work fine using gdisk and I created an ext4 file system on it though it took a while to complete. Trying to mount the file system resulted in multiple errors of different types and trying to read the partition table also gave garbage output. A quick Google showed a few people with similar issues with SSDs when used with the on board SATA ports on my motherboard, an ASUS M2N which uses an nVidia nForce 430 MCP chipset. The board is quite dated by todays standards but hasn’t given me any trouble in the past.

I removed the SSD and placed it in a spare machine fitted with an Adapted AAR 1210 SATA controller. This time the drive performed much better. The partition table appeared fine and the creation of the ext4 file system was much faster. A run of bonnie++ completed with multiple errors when the file system was mounted with the “discard” option to make use of TRIM. Mounting the file system without the “discard” option resulted in a bonnie++ run without errors.

None of my machines have AHCI support which I would have liked to test. I’m not sure if AHCI is required for TRIM support or not. I’m guessing the drive is fine and I’m just having issues due to the age of my hardware. I’ll test the SSD on a more recent AHCI machine at work tomorrow. I may need to invest in a AHCI PCIe SATA controller for my main workstation or replace the motherboard.

It appears that SSDs are not always a simple drop in replacement for HDDs.


Recently replaced lighttpd with nginx on Debian squeeze. Used spawn-fcgi to manage PHP as php-fpm is not in squeeze yet. Had a little trouble with reliability until I set a couple of environment variables before spawning the PHP processes. Since using the script below to start php all has been well.

/usr/bin/spawn-fcgi -s /tmp/cgi-php.socket -u www-data -g www-data \
                    -P /var/run/ \
                    -- /usr/bin/php5-cgi

Next I would like to look at setting up varnish in front of nginx.