Archive for the ‘Articles’ Category

Script day: persistent memoize in bash

Thursday, September 10th, 2015

One type of task that I often find myself implementing as a bash script, is to periodically generate some data and display or operate on it – maybe through a cron job, watch or simply a loop. Sometimes part of the process is an expensive computation (could be network based, IO intensive or simply subject to throttling by another entity). The way to deal with issues like that in modern programming languages is a caching technique known as “memoization” (based on the word “memorandum”) in the results of an expensive call is retained in memory after the first time, and returned for future calls instead of running the expensive calculation. We also need to clear the cache every once in a while, but that’s another issue.

So, how to implement in bash?


Script Day: Cloud-init for MS-Windows, The Poor Man’s Version

Thursday, August 20th, 2015

Cloud-init is a Linux technology that allows easy setup and automation of virtual machines. The concept is very simple – the VM infrastructure provides some way of setting some custom data for each virtual machine (many providers call this “user data”), and when the operating system starts the cloud-init service reads that configuration, loads a bunch of modules to handle various parts and let them configure the system. As a user it is very convenient – you write a setup scenario using the variety of tools offered by cloud-init, you can store the scenario in a source control to allow to develop the scenario further, then just launch a bunch of machines with the specified scenario and watch them configure themselves.

The situation is much worse on the MS-Windows side of the fence: want to have an MS-Windows server configured and ready to go? Start a virtual machine, connect to is using RDP and Next, Next, Finish until your fingers are sore. Need to deploy a new version? either retrofit an existing image (again, manually) and risk deployment side effects, or do the whole process again from scratch.

Here’s a script to try to help a bit with the problem – at least on Amazon Web Services: a poor man’s cloud-init-like for MS-Windows server automation.


Fix another ‘curl|sh’ bogus installation – Heroku

Friday, June 19th, 2015

The Heroku toolbelt (which I don’t remember if its mentioned in the “curlpipesh” tumbler mentioned by Amir in his response to my “Fix RVM” post), is a CLI to manage applications on the Heroku PaaS platform. As is common (and horrible) in this day and age they also offer a ‘curl|sh’ type install on their home page.

While the Debian/Ubuntu specific installer is not entirely horrible – it basically adds the Heroku Toolbelt debian repository to the APT source list, updates the package list and installs it, the “standalone” version is as horrible as it can get: download an unsigned binary from the internet, get root permissions and then do something.

For users of Fedora and other distributions, or just Ubuntu users who don’t like installing external repositories on their system, here is a simpler method to get the Heroku Toolbelt running on your system without root permissions and downloading scripts off the internet:


How to circumvent the free Wi-Fi content filter, for fame & profit

Monday, May 18th, 2015

I’m very grateful for the free Wi-Fi on the train, the coffee shop or the municipal free Wi-Fi, but the content filter they have on their proxies is sometimes really weird – for example it may blocks one of my favorite podcasts website (the Jupiter Broadcasting network) under the category “streaming media” even though they don’t actually host their video files, but they do let through YouTube and Facebook (where most cat videos are posted these days). So apparently Israeli Rail has an aversion to streaming media so they won’t let me send an email to a small podcast, but I can watch all the cat videos I want. Weird. Also, most VPN services are blocked by default, so no help will be coming from that way(1).

So, to fix that, here’s a small workaround using an external proxy – this is rather simple, but it does assume you have all kinds of tools that most users won’t have just lying around – but if you’re a Linux geek you should do just fine.


  1. I’ve checked the OpenVPN ports are blocked, as well as all web-based proxies I could find, such as FoxyProxy and Hola. I’ve encountered in the past a weird VPN software that does not use standard UDP or TCP sockets, but instead using GRE packets and I have no idea if that would work, but I’m assuming it won’t as well. []

Fix RVM “run script from the internet to install”

Friday, May 8th, 2015

On Wednesday I complained about the latest UN*X fad of installing software by running scripts from the internet, without any regard to how your operating system handles software installation.

Docker, that I complained about last time, at least has a script that takes into account the local software management solution (uses apt for Ubuntu, yum for Fedora, etc), but RVM – the Ruby Version Manager which is a popular tool among rubyists everywhere, just downloads a bunch of executable stuff (granted, most of it are scripts, but the difference is lost on most people) into arbitrary location on your file system. At least it doesn’t install system software, oh wait – it does.

While I can’t help with RVM’s desire to install system level software (that it actually needs because one of the things you want RVM to do for you is to compile ruby versions from source), I can try to help you figure out how to install RVM where you want it and use it how you want it.


Docker and the horrible “one line installation” fad

Wednesday, May 6th, 2015

One of the weird things that sane (or some would say “old skool”) system administrators complain about lately is that with the rising popularity of UN*X systems (mostly Mac OS X and Linux) in the world, and in particular in the software development world, people using UN*X system want less and less to understand how to manage their systems and the culmination is the

to install this complicated system level software, just copy and paste simple wget command to your terminal

with Docker being the most horrible example of that behavior. No sane person (who understand UN*X) will ever think that installing Docker by feeding the content of a URL to bash is a good idea, but for some reason this is the documented and recommended way by the Docker people. Other examples are abound, but lets concentrate on fixing the Docker scenario.


Microsoft open-sourced MSBuild

Thursday, March 26th, 2015

The Microsoft initiative to open source the .Net platform (which the MSBuild tool is a part of) has been talked about a lot in the past (though I have something to say about this as well, probably later in this post), but the fanfare has died down quite a bit since the last announcement. One might say that the reason they didn’t open source the entire thing at once was so Microsoft can space out the announcement and synthetically generate continued buzz about their platform, but knowing how these things usually work, its much more likely that because preparing a project for open-source is difficult and time consuming and a project as large as .Net doubly so (or a thousand times so), so it makes sense to do so in parts.

But to the question at hand – what does an open source MSBuild means to you? (more…)

Never let a Mac user design your website, here’s why:

Tuesday, February 17th, 2015

Well, at least until the entire world uses high DPI screens. Lets see an example:

Pulsing button (video)

This is a “call to action” effect on a button – it pulses slightly to get attention. This has proven to be really effective at improving “conversion” (web term to “getting you to do that thing I want you to do”). But even if you are not a designer at heart its easy to see that the text in the button is pulsing at different speeds and this creates a really jarring effect.


Pollution Efficiency of Alternative Fuel Cars

Tuesday, November 25th, 2014

Every now and then, when discussing EVs (electric vehicles) or other alternative fuel cars(1), that offer basically zero pollution (in the form of green house gases and other poisonous gases) someone always raises the argument that these cars just “move the pollution upstream” to the electric power plants and you are still polluting just the same.

But this argument only works if the power production “upstream” really is polluting as much as an onboard gas engine – which really sounds weird to me, because as you increase the success of the engine – from something that had to be small enough to fit in a car to something that basically only had the limit of required to be cost effective – surely you can have large efficiency gains, even if burning the same fuel?

So here are some numbers:


  1. like the new “hydrogen powered” cars []

Script day – Amazon AWS Signature Version 4 With Bash

Wednesday, November 19th, 2014

As anyone who works with the Amazon Web Services API knows, when you submit requests to an AWS service you need to sign the request with your secret key – in order to authenticate your account. The AWS signing process has changed through the years – an earlier version (I think version 1) I implemented in a previous blog post: upload files to Amazon S3 using Bash, with new APIs and newer versions of existing APIs opt in to use the newer signing process.

The current most up to date version of the signing process is known as Signature Version 4 Signing Process and is quite complex, but recently I had the need to use an AWS API that requires requests to be signed using the version 4 process in a bash script(1), so it was time to dust off the old scripting skills and see if I can get this much much much more elaborate signing process to work in bash – and (maybe) surprisingly it is quite doable.

With no further ado, here is the code:


  1. I’m trying to use SQS to send change notifications from a FreeBSD jail running on a FreeNAS server – a place were I’m uncomfortable installing the AWS CLI tool or the SDK. This also help explains all the FreeBSD compatibility written into the code []

Spam prevention powered by Akismet