Terminator Keyboard Shortcuts

Ctrl-Shift-E: will split the view vertically.

Ctrl-Shift-O: will split the view horizontally.

Ctrl-Shift-P: will focus be active on the previous view.

Ctrl-Shift-N: will focus be active on the next view.

Ctrl-Shift-W: will close the view where the focus is on.

Ctrl-Shift-Q: will exit terminator.

Ctrl-Shift-X: will focus active window and  enlarge it

Linux, Storage

GlusterFS Fuse Hanging on CentOS 7

Having strange GlusterFS hanging when using the native FUSE client on CentOS? This was a bit of a bitch, actually. It was hard to reproduce. Eventually, the only semi-regular way to repro it was to create lots of small files from multiple servers at the same time.

The Behavior

It would still be mounted but hang. The only indication of things being a problem would be a console hang when trying to df or use the filesystem.

The kern.log also shows that there’s long waits waiting for either the application running on top, or the fuse client itself.

Note: I was actually able to make the NFS client hang, but we don’t want to use the NFS client due to losing the graceful failover features etc. Performance has been reported to be an issue with the fuse client, but I was able to tune this pretty well. I don’t want to go into that here.

The Solution

The base CentOS 7 kernel is pretty old. I mean, it’s still updated, but it’s still 3.10.0-327.10.1 as of June 2016. Instead of compiling our own kernel, I grabbed the RPMs from Elrepo (

Installed this, after many days of troubleshooting, testing, and tuning, this solved the issue. No more lock-ups or fop STAT / LOCK issues.

I didn’t want to go main-line 4.6 kernel, so I opted for the 4.5.4-1 stable kernel. You should also be aware these are VMs running under VMware.

Here’s a quick hacked together Ansible playbook to handle the upgrade and verification for you via yum.


  • Glusterfs Server – 3.7.11-1 (April 18 2016)
  • Glusterfs Fuse Client – 3.7.11-1
  • Old kernel 3.1.0-327.10.1.el7
  • New kernel 4.5.4-1.el7.elrepo

– hosts: all
sudo: true
kernel_version: “4.5.4-1.el7.elrepo”

– name: Read Kernel Version
command: ‘uname -r’
register: result

– name: Has kernel upgrade already completed
fail: msg=”Kernel version already {{ kernel_version }}”
when: “‘{{ kernel_version }}’ in result.stdout”

– name: Uninstall Existing Kernel Packages
yum: pkg={{item}} state=absent disable_gpg_check=yes
– kernel-headers
– kernel-tools
– kernel-tools-libs

– name: Install Existing Kernel Packages
yum: pkg={{item}} update_cache=yes state=installed disable_gpg_check=yes
– kernel-ml-{{ kernel_version }}
– kernel-ml-devel-{{ kernel_version }}
– kernel-ml-headers-{{ kernel_version }}
– kernel-ml-tools-{{ kernel_version }}
– kernel-ml-tools-libs-{{ kernel_version }}
– kernel-ml-tools-libs-devel-{{ kernel_version }}

– name: Set Boot Time Option for Kernel
command: “grub2-set-default 0”

– name: Change grub2 configs
command: “grub2-mkconfig -o /boot/grub2/grub.cfg”

– name: Read Kernel Version
command: ‘uname -r’
register: result
ignore_errors: True

– name: Print Kernel Version
debug: var=result.stdout_lines

– name: Restart server
sudo: true
command: “{{ item }}”
async: 0
poll: 0
– “shutdown -r +1″
ignore_errors: true

– name: Wait for server to reboot
wait_for: >
host={{ inventory_hostname }}
timeout={{ 5 * 60 }}
delegate_to: localhost

– name: Read Kernel Version
command: ‘uname -r’
register: result
ignore_errors: True

– name: Print Kernel Version
debug: var=result.stdout_lines

– name: Did kernel upgrade fail
fail: msg=”Kernel does not match {{ kernel_version }} actual kernel is result.stdout”
when: “‘{{ kernel_version }}’ not in result.stdout”

Linux, Sysadmin

SIMPLFIED: Your own mail server (Postfix, Dovecot, SpamAssasin, ClamAV, Roundcube, SSL, Admin Panel)

The Stack

I was about to do the usual dance of setting up a mail system comprising of:

  • Postfix (outbound)
  • Dovecot (inbound – pop and imap)
  • MySQL (MariaDB for mailboxes etc.)
  • SpamAssassin (anti-spam)
  • ClamAV (anti-virus)
  • Roundcube (webmail)
  • Nginx (for the webmail)
  • SSL keys for everything
  • DNS – DKIM/SPF/etc
  • Fail2ban
  • IPTables firewall additions

It’s pretty intense setting all this up, so it’s recommended that you build using a configuration management tool like Ansible.

I took a look to see if there was a more modern way to do all this..

Enter iRedmail

iRedmail is an open source package that installs/configures all of the above and an administration panel iRedAdmin! The panel allows for user and domain administration but the pro version is what handles aliases and so on. It’s very expensive an unnecessary. You can make the same changes via the DB with one liners. If you have a server like Jenkins, you could easily push-button automate this. If/when I do, I’ll update this post.

You still need to add all the DNS entries (SPF, DKIM, etc) and sort out your SSL certs, plus integration with a smarthost (like Sendgrid).

Go here: and follow the instructions for your OS. It’s really *that* easy. It will even install the database server for you. 🙂

Add SendGrid (or another smart host / relay service) support

Postfix: /etc/postfix/

However, the local AV instance expects an unencrypted connection, so you need to account for this. Remember to restart the service after the config change with (debian/ubuntu) /etc/init.d/postfix restart.

# Sendgrid smarthost
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = static:USERNAME:PASSWORD
smtp_sasl_security_options = noanonymous
smtp_tls_security_level = encrypt
header_size_limit = 4096000
relayhost = []:587

Postfix: /etc/postfix/

Amavisd integration.
smtp-amavis unix – – – – 2 smtp
-o smtp_data_done_timeout=1200
-o smtp_send_xforward_command=yes
-o disable_dns_lookups=yes
-o max_use=20
-o smtp_tls_security_level=none inet n – – – – smtpd
-o content_filter=
-o mynetworks_style=host
-o mynetworks=
-o local_recipient_maps=
-o relay_recipient_maps=

< snip >

Linux, MySQL

How to restore debian-sys-maint MySQL Maria user

I had an issue where I wiped this user out accidentally in a galera cluster on Ubuntu 14.04. This is how to add ‘debian-sys-maint’@’localhost’ user back into your MySQL server and then enjoy all the ‘benefits’ once more..

You can then verify the user with SELECT * from mysql.user\G

Make sure that the password in /etc/mysql/debian.conf matches the password below as THE_PASSWORD.

use mysql;
INSERT INTO `user` (
Hardware, HomeLab, Linux

LXLE on 2007 Sony Vaio VGN-TZ170N

The laptop and the OS

I have an old beat up but super nice back-in-the-day Sony Vaio VGN-TZ170N that has been sitting in a drawer for a few years. This thing was pretty much a flagship machine back when it was launched in 2007, it carried an MSRP of $2,399.99.

I figure that since I never get rid of anything and living in San Francisco means that space (for normal people) is a luxury that we don’t have (let’s say family pressure is the driver), it’s time to sell this (at least try to) to someone who will appreciate it more than me.

Here is the eBay link:

Here’s a quick rundown of the machine:

  • CPU: Intel Core 2 Duo U7500 (1.06GHz) x64
  • Screen: 11.1″ WXGA (1366×768) XBRITE-HiColor (wat)
  • RAM: 2GB DDR2
  • Storage: 100 GB PATA 4200 RPM (1.8″ HDD, I believe!)
  • Optical: Built in DVD-RW drive!
  • GFX: Intel GMA950
  • Chipset: Mobile Intel 945GMS Express.
  • VRAM: Shared
  • Comms: Modem (v92), gigabit LAN, wireless LAN (Intel PRO 4965AGN), bluetooth, i400 firewire, 2 USB, 1 VGA, 1 docking conector, 1 headphone, 1 microphone
  • Expansion: 1x ExpressCard /34 slot
  • Built in webcam, and SD (plus Sony memory stick) card reader, fingerprint sensor.
  • Weight: 2.7 lbs
  • Dimensions: 10.9″ x 7.8″ x 0.8-1.17″ (this thing is SMALL – unbelievable how much is packed into it, even today never mind 2007).
  • Windows Vista Business COA on the underside.

Looks like you can even pick up a replacement (or additional 6 cell battery for $22 now.
While the RAM slot is accessible from the underside, it doesn’t look possible to upgrade it beyond the 2GB limit (maybe there is a 4G DDR2 SODIMM that would work? I don’t have one to try).

The revival?

First, I grabbed a copy of Vista Business and installed it via the internal optical drive (nice!). This took *forever*, and was over 4 CDs. Once I got into Windows after waiting about 2 days to install Windows updates, I then went on a quest to kill all the background processes that jam the CPU at 100%.

End result? Machine back in the drawer for another 2 years.

Lightweight Linux

I knew that there were various distributions of Linux that are aimed at low spec, or low performance systems. Since I’ve always been an Ubuntu guy, I decided to start there. The aim of this post isn’t to review the top options of lightweight Linux – I simply don’t have enough time. I can say that at some point I must have installed Kubuntu 14.04, I powered it up, there it was – works great. Uses a bit too much RAM for my liking (about half gone at boot), I started looking into other options to prep this for eBay.

The machine’s hard drive is pretty slow, and I’m also burning these to CD, so I can’t really be assed spending a long time trying them all out. Here’s a non-scientific conclusion of what I considered over the course of an hour or two:


I did look at Lubuntu as a more lightweight replacement for Kubuntu, and that’s what started this whole thing..

Super lightweight and looks ideal for purpose. However, the choice of applications preinstalled are typically considered lighter versions. For example, no LibreOffice.

Generally, the consensus seems to be that it feels a bit too lightweight or barebones. This doesn’t freak me out, as I’d install what I want to, but for eBay purposes, I moved on.


Elementary OS

Very Mac OSX like. However, I’ve read that it’s heavy on RAM. When you only have 2GB of RAM to play with and an old 4200 RPM drive, I think disk swapping is something that needs to be avoided..



Looks like a nice out of the box desktop, but the website put me off. Just sayin..


LXLE (Linux Extra Life Extension)

This was actually my first choice, after looking at Lubuntu!

It’s basically touting itself as Lubuntu but with everything you need configured and customized out of the box to be a nice workstation/desktop. I downloaded this one, 64bit (I know, only 2G ram..)

It’s basically Lubuntu but with the best/useful up-to-date applications, beautiful aesthetics, and a polished user interface. It’s supposed to be a drop-in replacement for Windows, as a desktop.

It did annoy me that the base image was 1.47G and not within CD-R limits. However, old PC might mean old Internet. I just would have rather had a 2 CD option versus 1 DVD.


Ubuntu MATE

Lots of good reviews and buzz around this. I had already settled on LXLE and cut a disc, I didn’t fancy a reinstall. I actually really like the look and sound of this OS.



Download & Preparation

First, I had to figure out a way to burn the ISO to disc. Let’s use the Sony and the Kubuntu installation.

Since I pre-downloaded it on my Macbook Air, I uploaded it to the NAS and grabbed it from there. I have to say it was much slower downloading it from the NAS than it was uploading it (and my Air doesn’t have ac wireless).

Downloading LXLE ISO via Firefox from my Synology Diskstation web interface

Approximately a third of the way done, the system looks OK. Interestingly, CPU software interrupts seems pretty high.

Looking further into this, it looks like it’s the timer (IRQ 0), there’s not much I can do about that. I guess this is just related to using an old machine, and not really worth tuning.

I do note that the system fan has kicked in, Firefox is taxing the machine. No real slowdowns noticed, though.


Eventually, a cup of tea later, it’s time to write the ISO to disc. I’ve not really had much of a requirement for optical discs or drives for quite a few years now, so I’ve still had the same pack of DVD-Rs for easily over 5 years. Interestingly, they all seem to have some weirdness to them now. I’m not sure if you can see this, but here’s a photo:

Over 5 years old, still on the original spindle. They all look like this.

I’m not holding out much hope for a 9 year old machine being able to burn a 5+ year old disc, but it’s worth a shot.

Kubuntu: Burning LXLE on 9 year old laptop with 5+ year old media

Looks like it worked fine. Verified OK, and disc verification in LXLE menu was OK too. Man, I don’t even want to sell this. 🙂


Took about 5 minutes to load past the loading screen. Be free, Be open! .. Wifi connected fine.


  • 20 minutes to get to “Installation is complete” message.

We’re in business!



Overall it’s pretty snappy. LibreOffice worked ok, I was able to do some basic things in terminal and used Seamonkey to perform a run (so Flash works OK too).

Hardware (out of the box)

  • Audio: OK
  • Wifi: OK
  • Bluetooth: Looks OK in dmesg, didn’t test further
  • Optical drive: OK
  • GSM modem: Looks OK in dmesg, didn’t test further
  • LAN: looks OK in dmesg, didn’t test further
  • iee1394/firewire: Looks OK in lspci, didn’t test further.
  • Fingerprint reader: Looks OK in lsusb, didn’t test further
  • Camera: Looks OK in lsusb, no driver installed.
  • Blue FN key presses; some work OK
    • Volume up/down OK
    • Mute OK, doesn’t un-mute
    • Brightness does nothing
    • Page Up/Page down OK
    • Home/End OK
    • Didn’t test the others.
  • Right hand side of touchpad for scrolling works well.


CPU gets high, pretty much any time you do anything with the browser. It doesn’t seem to slow down the OS really from a user experience perspective, though. Right now, I guess anti-virus updates happened, and it’s using 100% of a single core doing regular updates. Overall, a nice little compact Linux laptop.

img_9966 over wireless.

Battery life test will be difficult, it’s going to take a while to finish charging. I’m not really sure if it even still holds a charge.

Browsing generally feels OK, and modern. Youtube plays a music video smooth with no skips, although CPU is high. Works full screen too.


Now I didn’t expect Netflix to work, and I wasn’t disappointed. Before I started streaming, just after I logged in, RAM usage spiked enough to start swapping ~100MB. Once I tried to start streaming, it threw an error saying HTML5 support was needed. I guess Seamonkey doesn’t have support / correct support. Quick Google seems to show this as a known problem. I launched the Lubuntu Software Center, and grabbed Firefox from there. No dice, same issue.

Battery Test

Charging the battery took a long time; It’s 88% now and wants 27 minutes to fill. It was 0% (and off for years) when I started this. I switched out for a 16V 4a adapter I have from the 16V 2.2a and it only dropped to 25 minute charge time from 27 – I guess it just takes a while to charge. From 89% to empty, it’s giving an estimate of 3h 51m remaining. It looks like this thing still holds a charge!

Usage Log

  • Started on battery at 17:54
  • Had a game of DreamChess .. still 89%
  • Game of CriticalMass still 89%
  • I am now editing the remainder of this blog post from the laptop itself via Seamonkey. Works well, the screen is really nice.
  • Down to 87% remaining and an estimate of 3 hours.
  • Still only using 20% cpu and 450MB RAM.
  • Have 4 desktop spaces open, with just the Seamonkey browser on one half of the screen. The keyboard is still pretty nice to type on, except for me having a missing tab key cap.
  • Pandora is playing in another tab now, through the speakers at 50%. CPU 16.3%, RAM 553MB (27%) – pretty impressive memory footprint, really. 86% battery.
  • Went back up the blog post and edited the section adding in the blue FN key info. Down to 84%. Pandora must be dragging the battery down a bit now.
  • Opened the software center to see what games I can download..
  • Installed:
    • Freeciv
    • Freecol
    • Descent 1 Rebirth
    • Freedoom
    • Freedroid
    • Freedroid RPG
  • Even while installing packages, Pandora didn’t skip, and I was still able to type and edit this post.
  • Pandora wanted me to sign in, so I launched the built-in Guayadeque Music Player to stream some radio instead.
  • 78% after the software all installed.
  • Freedoom: silky smooth
  • Freedroid RPG – what is this game? I love it.
  • FreeCol ground the machine to a halt. I guess Java is a no-no. Rebooting, going to try again. 75% battery when I get back into desktop.
  • FreeCol starts but when I try to click past the introduction video, it sorta just hangs. I gave it a good 60 seconds to sort itself out before I got impatient and rebooted again.
  • It’s now 18:36 and I’m back on the Macbook to edit the blog.
  • FreeCiv.. works great, still 73%
  • Descent 1 doesn’t start
  • Heading back to the software store to find some more intensive games. Installing quite a lot actually, 3G of stuff downloaded, 5G installed. I killed the radio at the start of installation too.
  • After installing and playing a bunch of games.. generally what you could say “heavy use”, at 20:20, there’s 25% battery (estimate 1h 4m remaining).

This slideshow requires JavaScript.

Battery Results

  • 89% to 25% battery – 2h 25m of heavy use.
  • Charge estimate of 25% to full – 3h 18m
  • Not bad at all for a laptop nearly a decade old.

Concluding Stats

  • Boot time: 1m 20s (usable as soon as desktop appears with no slowdown)
  • Shutdown time:
  • Wifi performance: 23Mb down/50Mb up
  • Usable for: basic web browsing, remote administration, Linux development, open source gaming, portability, those times where you need old port connectivity and optical drive in a nice tiny package – almost impossible to find now…

…and we’ll call it a day.