Updates from November, 2019 Toggle Comment Threads | Keyboard Shortcuts

  • duncanbowring 18:39 on November 18, 2019 Permalink | Reply
    Tags: BigTable, GCP   

    Google Cloud BigTable Performance Observations 

    I was handling a project that involved a migration from HDD cluster to SSD cluster, with a multi-terabyte sequential backup file in GCS. My basic conclusion is that ROI dropped drastically beyond around 70 nodes in the BT cluster, during this type of activity.

    • Bigtable node vertical CPU maxes out at around 5.6 million row writes per second (~550MB/sec), therefore there was little to no difference in the hot node CPU from 70-90 nodes, even with 90 import workers.
    • Dropping to 70 nodes from 90 only resulted in a dip of around 20MB/sec (300k row writes per sec), and $58.74 ph to $45.74 ph. Sure adds up over the course of a month.

    • As you can see from the following graph, there is an upper limit on sequential import CPU, presumably because partitioning isn’t helping here? I’m unsure why, since there are supposed to be 90 workers operating. I’m guessing there is an upper CPU limit on whatever is orchestrating/managing the inbound flow. Perhaps a load balancer that we cannot see. You can see that increasing node count does affect the overall processing CPU, but has little impact on the hottest node. I’m not really sure where the bottleneck is.



  • duncanbowring 20:40 on January 31, 2019 Permalink | Reply  


    No more messing about trying to find these clients, if you can’t grab them from a hypervisor. Here are the direct links to vmware.com for the executable downloads:

    vSphere 4.1

    VMware vSphere Client v4.1 : VMware-viclient-all-4.1.0-258902.exe
    VMware vSphere Client v4.1 Update 1 : VMware-viclient-all-4.1.0-345043.exe
    VMware vSphere Client v4.1 Update 2 : VMware-viclient-all-4.1.0-491557.exe
    VMware vSphere Client v4.1 Update 3 : VMware-viclient-all-4.1.0-799345.exe

    vSphere 5.0

    VMware vSphere Client v5.0 : VMware-viclient-all-5.0.0-455964.exe
    VMware vSphere Client v5.0 Update 1 : VMware-viclient-all-5.0.0-623373.exe
    VMware vSphere Client v5.0 Update 1b : VMware-viclient-all-5.0.0-804277.exe
    VMware vSphere Client v5.0 Update 2 : VMware-viclient-all-5.0.0-913577.exe

    vSphere 5.1

    VMware vSphere Client v5.1 : VMware-viclient-all-5.1.0-786111.exe
    VMware vSphere Client 5.1.0a : VMware-viclient-all-5.1.0-860230.exe
    VMware vSphere Client 5.1.0b : VMware-viclient-all-5.1.0-941893.exe
    VMware vSphere Client 5.1 Update 1 : VMware-viclient-all-5.1.0-1064113.exe
    VMware vSphere Client 5.1 Update 1b : VMware-viclient-all-5.1.0-1235233.exe

    vSphere 5.5

    VMware vSphere Client v5.5 : VMware-viclient-all-5.5.0-1281650.exe
    VMware vSphere Client 5.5 Update 3: VMware-viclient-all-5.5.0-3024345.exe

    vSphere 6

    VMware vSphere Client 6.0: VMware-viclient-all-6.0.0-2502222.exe
    VMware vSphere Client 6.0 Update 3: VMware-viclient-all-6.0.0-5112508.exe

  • duncanbowring 22:48 on January 30, 2019 Permalink | Reply
    Tags: cygwin, , rsync,   

    Cygwin Rsync Windows Permissions 


    OK, stop this happening:

    Open /etc/fstab and add noacl to the entry:

    none /cygdrive cygdrive binary,noacl,posix=0,user 0 0

    Save it. Close all cygwin terminals and start new terminal.

    Fix the bad stuff that already exists at the destination:

    Change Owner to local Administrator. Save and Apply to Children.
    Edit permissions, fix them up.. then advanced, Advanced.. apply to children (inherit from the new top level one you added).
    After this.. disable read only, apply to children.
    You will be able to delete this mess now.


    • psychocod3r 13:44 on February 27, 2019 Permalink | Reply

      I don’t think administrative programs like that are meant to run in Cygwin. The security models used by Linux and Windows are so fundamentally different that it can be near impossible to translate between the two. You’re probably better off just running stuff like this natively in Linux. I tried running pro-ftpd in Cygwin once and it was a nightmare, mostly because there was no documentation available to indicate what specific accounts I needed to create for running the server within Cygwin. Programs that run in Cygwin are not actually Linux programs; they’re Windows programs with a few extra layers of abstraction on top. You have to remember that even though a Cygwin program may feel like it’s running on a Unix system (since that’s what an emulator does after all), it’s really running entirely in the context of Windows and its security model.

      • duncanbowring 13:54 on February 27, 2019 Permalink | Reply

        Hi there. There were a couple of scenarios where I had to do this on Windows. I agree with you, fundamentally.

  • duncanbowring 10:58 on January 12, 2019 Permalink | Reply  

    Dell R210 II BIOS Update Bricked – Fixed 

    Broke my old Dell R210 II doing a BIOS update. Managed to fix it by flashing the chip directly.

    In the case of the R210 II, I used:

    CH341A USB Programmer (https://www.amazon.com/gp/product/B01I1EU9LG)

    SOIC8 Chip Clip (https://www.amazon.com/gp/product/B00V9QNAC4)

    CH341A Programmer software (free)

    The toughest part was finding the 8MB flash binary within the Dell downloads.

    The flash image start header is: FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF5AA5F00F .. searching for that [using a hex editor] and then selecting 0x800000 (8MB) length (which is the end of file here). I dumped it to a new file, then flashed that new .bin to the BIOS. First I tried to use the OEM EXE, but it was a weird length. I ended up extracting the installer and using the file that was in the payload directory.

    Result! Back in business.

  • duncanbowring 13:17 on March 23, 2017 Permalink | Reply
    Tags: fragment identifier, javascript, web   

    How to redirect a URL fragment – hashtag in the URL 

    If you have address similar to http://host/#/blah, only ‘host’ is sent to the webserver – #/blah is only handled in the browser. This is called a fragment identifier.

    You can get around it by adding javascript to the page(s) you want to redirect as follows, changing up the catch-all for specifics:

    var redirectFragment = window.location.hash.replace(/^#/,”);
    if ( ” !== redirectFragment ) {
    window.location = ‘http://www.google.com’;

  • duncanbowring 04:21 on December 1, 2016 Permalink | Reply  

    MySQL Query Table Sizes 

    Return a table with the table sizes in descending order of the selected database. Run this from mysql console. It will let you compare actual table size versus disk size (before you optimize table to reclaim space).

         table_schema as `Database`, 
         table_name AS `Table`, 
         round(((data_length + index_length) / 1024 / 1024), 2) `Size in MB` 
    FROM information_schema.TABLES 
    ORDER BY (data_length + index_length) DESC;
    • hasforkvadorni 01:45 on March 16, 2017 Permalink | Reply

      Excellent site. Plenty of useful info here. I’m sending it to several pals ans also sharing in delicious. And obviously, thank you to your effort!

  • duncanbowring 16:01 on April 29, 2016 Permalink | Reply
    Tags: Dovecot, iRedmail, , Mailing, Postfix, ,   

    SIMPLFIED: Your own mail server (Postfix, Dovecot, SpamAssasin, ClamAV, Roundcube, SSL, Admin Panel) 

    The Stack

    I was about to do the usual dance of setting up a mail system comprising of:

    • Postfix (outbound)
    • Dovecot (inbound – pop and imap)
    • MySQL (MariaDB for mailboxes etc.)
    • SpamAssassin (anti-spam)
    • ClamAV (anti-virus)
    • Roundcube (webmail)
    • Nginx (for the webmail)
    • SSL keys for everything
    • DNS – DKIM/SPF/etc
    • Fail2ban
    • IPTables firewall additions

    It’s pretty intense setting all this up, so it’s recommended that you build using a configuration management tool like Ansible.

    I took a look to see if there was a more modern way to do all this..

    Enter iRedmail

    iRedmail is an open source package that installs/configures all of the above and an administration panel iRedAdmin! The panel allows for user and domain administration but the pro version is what handles aliases and so on. It’s very expensive an unnecessary. You can make the same changes via the DB with one liners. If you have a server like Jenkins, you could easily push-button automate this. If/when I do, I’ll update this post.

    You still need to add all the DNS entries (SPF, DKIM, etc) and sort out your SSL certs, plus integration with a smarthost (like Sendgrid).

    Go here: http://www.iredmail.org/ and follow the instructions for your OS. It’s really *that* easy. It will even install the database server for you. 🙂

    Add SendGrid (or another smart host / relay service) support

    Postfix: /etc/postfix/main.cf

    However, the local AV instance expects an unencrypted connection, so you need to account for this. Remember to restart the service after the config change with (debian/ubuntu) /etc/init.d/postfix restart.

    # Sendgrid smarthost
    smtp_sasl_auth_enable = yes
    smtp_sasl_password_maps = static:USERNAME:PASSWORD
    smtp_sasl_security_options = noanonymous
    smtp_tls_security_level = encrypt
    header_size_limit = 4096000
    relayhost = [smtp.sendgrid.net]:587

    Postfix: /etc/postfix/master.cf

    Amavisd integration.
    smtp-amavis unix – – – – 2 smtp
    -o smtp_data_done_timeout=1200
    -o smtp_send_xforward_command=yes
    -o disable_dns_lookups=yes
    -o max_use=20
    -o smtp_tls_security_level=none inet n – – – – smtpd
    -o content_filter=
    -o mynetworks_style=host
    -o mynetworks=
    -o local_recipient_maps=
    -o relay_recipient_maps=

    < snip >

Compose new post
Next post/Next comment
Previous post/Previous comment
Show/Hide comments
Go to top
Go to login
Show/Hide help
shift + esc
%d bloggers like this: