Having strange GlusterFS hanging when using the native FUSE client on CentOS? This was a bit of a bitch, actually. It was hard to reproduce. Eventually, the only semi-regular way to repro it was to create lots of small files from multiple servers at the same time.
The Behavior
It would still be mounted but hang. The only indication of things being a problem would be a console hang when trying to df or use the filesystem.
The kern.log also shows that there’s long waits waiting for either the application running on top, or the fuse client itself.
Note: I was actually able to make the NFS client hang, but we don’t want to use the NFS client due to losing the graceful failover features etc. Performance has been reported to be an issue with the fuse client, but I was able to tune this pretty well. I don’t want to go into that here.
The Solution
The base CentOS 7 kernel is pretty old. I mean, it’s still updated, but it’s still 3.10.0-327.10.1 as of June 2016. Instead of compiling our own kernel, I grabbed the RPMs from Elrepo (http://elrepo.org/tiki/tiki-index.php).
Installed this, after many days of troubleshooting, testing, and tuning, this solved the issue. No more lock-ups or fop STAT / LOCK issues.
I didn’t want to go main-line 4.6 kernel, so I opted for the 4.5.4-1 stable kernel. You should also be aware these are VMs running under VMware.
Here’s a quick hacked together Ansible playbook to handle the upgrade and verification for you via yum.
Versions
Glusterfs Server – 3.7.11-1 (April 18 2016)
Glusterfs Fuse Client – 3.7.11-1
Old kernel 3.1.0-327.10.1.el7
New kernel 4.5.4-1.el7.elrepo
– hosts: all
sudo: true
vars:
kernel_version: “4.5.4-1.el7.elrepo”
tasks:
– name: Read Kernel Version
command: ‘uname -r’
register: result
– name: Has kernel upgrade already completed
fail: msg=”Kernel version already {{ kernel_version }}”
when: “‘{{ kernel_version }}’ in result.stdout”
– name: Wait for server to reboot
wait_for: >
host={{ inventory_hostname }}
port=20848
state=started
delay=90
timeout={{ 5 * 60 }}
delegate_to: localhost
– name: Read Kernel Version
command: ‘uname -r’
register: result
ignore_errors: True
– name: Print Kernel Version
debug: var=result.stdout_lines
– name: Did kernel upgrade fail
fail: msg=”Kernel does not match {{ kernel_version }} actual kernel is result.stdout”
when: “‘{{ kernel_version }}’ not in result.stdout”
I was about to do the usual dance of setting up a mail system comprising of:
Postfix (outbound)
Dovecot (inbound – pop and imap)
MySQL (MariaDB for mailboxes etc.)
SpamAssassin (anti-spam)
ClamAV (anti-virus)
Roundcube (webmail)
Nginx (for the webmail)
SSL keys for everything
DNS – DKIM/SPF/etc
Fail2ban
IPTables firewall additions
It’s pretty intense setting all this up, so it’s recommended that you build using a configuration management tool like Ansible.
I took a look to see if there was a more modern way to do all this..
Enter iRedmail
iRedmail is an open source package that installs/configures all of the above and an administration panel iRedAdmin! The panel allows for user and domain administration but the pro version is what handles aliases and so on. It’s very expensive an unnecessary. You can make the same changes via the DB with one liners. If you have a server like Jenkins, you could easily push-button automate this. If/when I do, I’ll update this post.
You still need to add all the DNS entries (SPF, DKIM, etc) and sort out your SSL certs, plus integration with a smarthost (like Sendgrid).
Go here: http://www.iredmail.org/ and follow the instructions for your OS. It’s really *that* easy. It will even install the database server for you. 🙂
Add SendGrid (or another smart host / relay service) support
Postfix: /etc/postfix/main.cf
However, the local AV instance expects an unencrypted connection, so you need to account for this. Remember to restart the service after the config change with (debian/ubuntu) /etc/init.d/postfix restart.
I had an issue where I wiped this user out accidentally in a galera cluster on Ubuntu 14.04. This is how to add ‘debian-sys-maint’@’localhost’ user back into your MySQL server and then enjoy all the ‘benefits’ once more..
You can then verify the user with SELECT * from mysql.user\G
Make sure that the password in /etc/mysql/debian.conf matches the password below as THE_PASSWORD.
I have an old beat up but super nice back-in-the-day Sony Vaio VGN-TZ170N that has been sitting in a drawer for a few years. This thing was pretty much a flagship machine back when it was launched in 2007, it carried an MSRP of $2,399.99.
I figure that since I never get rid of anything and living in San Francisco means that space (for normal people) is a luxury that we don’t have (let’s say family pressure is the driver), it’s time to sell this (at least try to) to someone who will appreciate it more than me.
Built in webcam, and SD (plus Sony memory stick) card reader, fingerprint sensor.
Weight: 2.7 lbs
Dimensions: 10.9″ x 7.8″ x 0.8-1.17″ (this thing is SMALL – unbelievable how much is packed into it, even today never mind 2007).
Windows Vista Business COA on the underside.
Looks like you can even pick up a replacement (or additional 6 cell battery for $22 now.
While the RAM slot is accessible from the underside, it doesn’t look possible to upgrade it beyond the 2GB limit (maybe there is a 4G DDR2 SODIMM that would work? I don’t have one to try).
The revival?
First, I grabbed a copy of Vista Business and installed it via the internal optical drive (nice!). This took *forever*, and was over 4 CDs. Once I got into Windows after waiting about 2 days to install Windows updates, I then went on a quest to kill all the background processes that jam the CPU at 100%.
End result? Machine back in the drawer for another 2 years.
Lightweight Linux
I knew that there were various distributions of Linux that are aimed at low spec, or low performance systems. Since I’ve always been an Ubuntu guy, I decided to start there. The aim of this post isn’t to review the top options of lightweight Linux – I simply don’t have enough time. I can say that at some point I must have installed Kubuntu 14.04, I powered it up, there it was – works great. Uses a bit too much RAM for my liking (about half gone at boot), I started looking into other options to prep this for eBay.
The machine’s hard drive is pretty slow, and I’m also burning these to CD, so I can’t really be assed spending a long time trying them all out. Here’s a non-scientific conclusion of what I considered over the course of an hour or two:
Lubuntu
I did look at Lubuntu as a more lightweight replacement for Kubuntu, and that’s what started this whole thing..
Super lightweight and looks ideal for purpose. However, the choice of applications preinstalled are typically considered lighter versions. For example, no LibreOffice.
Generally, the consensus seems to be that it feels a bit too lightweight or barebones. This doesn’t freak me out, as I’d install what I want to, but for eBay purposes, I moved on.
Very Mac OSX like. However, I’ve read that it’s heavy on RAM. When you only have 2GB of RAM to play with and an old 4200 RPM drive, I think disk swapping is something that needs to be avoided..
This was actually my first choice, after looking at Lubuntu!
It’s basically touting itself as Lubuntu but with everything you need configured and customized out of the box to be a nice workstation/desktop. I downloaded this one, 64bit (I know, only 2G ram..)
It’s basically Lubuntu but with the best/useful up-to-date applications, beautiful aesthetics, and a polished user interface. It’s supposed to be a drop-in replacement for Windows, as a desktop.
It did annoy me that the base image was 1.47G and not within CD-R limits. However, old PC might mean old Internet. I just would have rather had a 2 CD option versus 1 DVD.
Lots of good reviews and buzz around this. I had already settled on LXLE and cut a disc, I didn’t fancy a reinstall. I actually really like the look and sound of this OS.
First, I had to figure out a way to burn the ISO to disc. Let’s use the Sony and the Kubuntu installation.
Since I pre-downloaded it on my Macbook Air, I uploaded it to the NAS and grabbed it from there. I have to say it was much slower downloading it from the NAS than it was uploading it (and my Air doesn’t have ac wireless).
Downloading LXLE ISO via Firefox from my Synology Diskstation web interface
Approximately a third of the way done, the system looks OK. Interestingly, CPU software interrupts seems pretty high.
Looking further into this, it looks like it’s the timer (IRQ 0), there’s not much I can do about that. I guess this is just related to using an old machine, and not really worth tuning.
I do note that the system fan has kicked in, Firefox is taxing the machine. No real slowdowns noticed, though.
Eventually, a cup of tea later, it’s time to write the ISO to disc. I’ve not really had much of a requirement for optical discs or drives for quite a few years now, so I’ve still had the same pack of DVD-Rs for easily over 5 years. Interestingly, they all seem to have some weirdness to them now. I’m not sure if you can see this, but here’s a photo:
Over 5 years old, still on the original spindle. They all look like this.
I’m not holding out much hope for a 9 year old machine being able to burn a 5+ year old disc, but it’s worth a shot.
Kubuntu: Burning LXLE on 9 year old laptop with 5+ year old media
Looks like it worked fine. Verified OK, and disc verification in LXLE menu was OK too. Man, I don’t even want to sell this. 🙂
Installation
Took about 5 minutes to load past the loading screen. Be free, Be open! .. Wifi connected fine.
20 minutes to get to “Installation is complete” message.
We’re in business!
Usability
Overall it’s pretty snappy. LibreOffice worked ok, I was able to do some basic things in terminal and used Seamonkey to perform a speedtest.net run (so Flash works OK too).
Hardware (out of the box)
Audio: OK
Wifi: OK
Bluetooth: Looks OK in dmesg, didn’t test further
Optical drive: OK
GSM modem: Looks OK in dmesg, didn’t test further
LAN: looks OK in dmesg, didn’t test further
iee1394/firewire: Looks OK in lspci, didn’t test further.
Fingerprint reader: Looks OK in lsusb, didn’t test further
Camera: Looks OK in lsusb, no driver installed.
Blue FN key presses; some work OK
Volume up/down OK
Mute OK, doesn’t un-mute
Brightness does nothing
Page Up/Page down OK
Home/End OK
Didn’t test the others.
Right hand side of touchpad for scrolling works well.
Software
CPU gets high, pretty much any time you do anything with the browser. It doesn’t seem to slow down the OS really from a user experience perspective, though. Right now, I guess anti-virus updates happened, and it’s using 100% of a single core doing regular updates. Overall, a nice little compact Linux laptop.
speedtest.net over wireless.
Battery life test will be difficult, it’s going to take a while to finish charging. I’m not really sure if it even still holds a charge.
Browsing generally feels OK, and modern. Youtube plays a music video smooth with no skips, although CPU is high. Works full screen too.
Now I didn’t expect Netflix to work, and I wasn’t disappointed. Before I started streaming, just after I logged in, RAM usage spiked enough to start swapping ~100MB. Once I tried to start streaming, it threw an error saying HTML5 support was needed. I guess Seamonkey doesn’t have support / correct support. Quick Google seems to show this as a known problem. I launched the Lubuntu Software Center, and grabbed Firefox from there. No dice, same issue.
Battery Test
Charging the battery took a long time; It’s 88% now and wants 27 minutes to fill. It was 0% (and off for years) when I started this. I switched out for a 16V 4a adapter I have from the 16V 2.2a and it only dropped to 25 minute charge time from 27 – I guess it just takes a while to charge. From 89% to empty, it’s giving an estimate of 3h 51m remaining. It looks like this thing still holds a charge!
Usage Log
Started on battery at 17:54
Had a game of DreamChess .. still 89%
Game of CriticalMass still 89%
I am now editing the remainder of this blog post from the laptop itself via Seamonkey. Works well, the screen is really nice.
Down to 87% remaining and an estimate of 3 hours.
Still only using 20% cpu and 450MB RAM.
Have 4 desktop spaces open, with just the Seamonkey browser on one half of the screen. The keyboard is still pretty nice to type on, except for me having a missing tab key cap.
Pandora is playing in another tab now, through the speakers at 50%. CPU 16.3%, RAM 553MB (27%) – pretty impressive memory footprint, really. 86% battery.
Went back up the blog post and edited the section adding in the blue FN key info. Down to 84%. Pandora must be dragging the battery down a bit now.
Opened the software center to see what games I can download..
Installed:
Freeciv
Freecol
Descent 1 Rebirth
Freedoom
Freedroid
Freedroid RPG
Even while installing packages, Pandora didn’t skip, and I was still able to type and edit this post.
Pandora wanted me to sign in, so I launched the built-in Guayadeque Music Player to stream some radio instead.
78% after the software all installed.
Freedoom: silky smooth
Freedroid RPG – what is this game? I love it.
FreeCol ground the machine to a halt. I guess Java is a no-no. Rebooting, going to try again. 75% battery when I get back into desktop.
FreeCol starts but when I try to click past the introduction video, it sorta just hangs. I gave it a good 60 seconds to sort itself out before I got impatient and rebooted again.
It’s now 18:36 and I’m back on the Macbook to edit the blog.
FreeCiv.. works great, still 73%
Descent 1 doesn’t start
Heading back to the software store to find some more intensive games. Installing quite a lot actually, 3G of stuff downloaded, 5G installed. I killed the radio at the start of installation too.
After installing and playing a bunch of games.. generally what you could say “heavy use”, at 20:20, there’s 25% battery (estimate 1h 4m remaining).
This slideshow requires JavaScript.
Battery Results
89% to 25% battery – 2h 25m of heavy use.
Charge estimate of 25% to full – 3h 18m
Not bad at all for a laptop nearly a decade old.
Concluding Stats
Boot time: 1m 20s (usable as soon as desktop appears with no slowdown)
Shutdown time:
Wifi performance: 23Mb down/50Mb up
Usable for: basic web browsing, remote administration, Linux development, open source gaming, portability, those times where you need old port connectivity and optical drive in a nice tiny package – almost impossible to find now…