Archive for the ‘Programming’ Category

Polymer Runtime Application Configuration

When creating web applications, we often need to have some parts of the application configuration for the deployment environment – usually web service API endpoints have different URLs in different environments, such as using a production web service in production and a locally hosted web service during development.

Such a feature is implemented in many web frameworks and building tools for web applications, such as Gulp or Grunt. Unfortunately, when building applications using Google’s Polymer SDK, there are no such features available – reviewing the Polymer documentation one, there isn’t even any mention of how one handles such mundane tasks as configuring API URLs, except hard-coding them(1).

Developers have tried to solve this problem in different ways, from adding “environments” feature for Polymer’s internal build tool; abusing “behavior modules”; or using “app globals” custom element with complex code to share application-level state. None of these features work well or elegantly (except maybe the environments feature, if it ever gets implemented).

Here is the solution I came up with – with many thanks to Daniel Tse that described part of the implementation in this article – just using the core Polymer elements iron-ajax and iron-meta and without any custom code. Its not the most elegant thing that can be done, but it is relatively simple and works well. Its main down-side is that the application configuration is not embedded in the application during build time but loaded from an external file when the application loads – this may even be a required feature in some scenarios but its not the generally accepted practice.

(more…)


  1. all iron-ajax examples, as an obvious example, use hard-coded URLs []

Script day: persistent memoize in bash

One type of task that I often find myself implementing as a bash script, is to periodically generate some data and display or operate on it – maybe through a cron job, watch or simply a loop. Sometimes part of the process is an expensive computation (could be network based, IO intensive or simply subject to throttling by another entity). The way to deal with issues like that in modern programming languages is a caching technique known as “memoization” (based on the word “memorandum”) in the results of an expensive call is retained in memory after the first time, and returned for future calls instead of running the expensive calculation. We also need to clear the cache every once in a while, but that’s another issue.

So, how to implement in bash?

(more…)

Script Day: Cloud-init for MS-Windows, The Poor Man’s Version

Cloud-init is a Linux technology that allows easy setup and automation of virtual machines. The concept is very simple – the VM infrastructure provides some way of setting some custom data for each virtual machine (many providers call this “user data”), and when the operating system starts the cloud-init service reads that configuration, loads a bunch of modules to handle various parts and let them configure the system. As a user it is very convenient – you write a setup scenario using the variety of tools offered by cloud-init, you can store the scenario in a source control to allow to develop the scenario further, then just launch a bunch of machines with the specified scenario and watch them configure themselves.

The situation is much worse on the MS-Windows side of the fence: want to have an MS-Windows server configured and ready to go? Start a virtual machine, connect to is using RDP and Next, Next, Finish until your fingers are sore. Need to deploy a new version? either retrofit an existing image (again, manually) and risk deployment side effects, or do the whole process again from scratch.

Here’s a script to try to help a bit with the problem – at least on Amazon Web Services: a poor man’s cloud-init-like for MS-Windows server automation.

(more…)

Fix another ‘curl|sh’ bogus installation – Heroku

The Heroku toolbelt (which I don’t remember if its mentioned in the “curlpipesh” tumbler mentioned by Amir in his response to my “Fix RVM” post), is a CLI to manage applications on the Heroku PaaS platform. As is common (and horrible) in this day and age they also offer a ‘curl|sh’ type install on their home page.

While the Debian/Ubuntu specific installer is not entirely horrible – it basically adds the Heroku Toolbelt debian repository to the APT source list, updates the package list and installs it, the “standalone” version is as horrible as it can get: download an unsigned binary from the internet, get root permissions and then do something.

For users of Fedora and other distributions, or just Ubuntu users who don’t like installing external repositories on their system, here is a simpler method to get the Heroku Toolbelt running on your system without root permissions and downloading scripts off the internet:

(more…)

Fix RVM “run script from the internet to install”

On Wednesday I complained about the latest UN*X fad of installing software by running scripts from the internet, without any regard to how your operating system handles software installation.

Docker, that I complained about last time, at least has a script that takes into account the local software management solution (uses apt for Ubuntu, yum for Fedora, etc), but RVM – the Ruby Version Manager which is a popular tool among rubyists everywhere, just downloads a bunch of executable stuff (granted, most of it are scripts, but the difference is lost on most people) into arbitrary location on your file system. At least it doesn’t install system software, oh wait – it does.

While I can’t help with RVM’s desire to install system level software (that it actually needs because one of the things you want RVM to do for you is to compile ruby versions from source), I can try to help you figure out how to install RVM where you want it and use it how you want it.

(more…)

Microsoft open-sourced MSBuild

The Microsoft initiative to open source the .Net platform (which the MSBuild tool is a part of) has been talked about a lot in the past (though I have something to say about this as well, probably later in this post), but the fanfare has died down quite a bit since the last announcement. One might say that the reason they didn’t open source the entire thing at once was so Microsoft can space out the announcement and synthetically generate continued buzz about their platform, but knowing how these things usually work, its much more likely that because preparing a project for open-source is difficult and time consuming and a project as large as .Net doubly so (or a thousand times so), so it makes sense to do so in parts.

But to the question at hand – what does an open source MSBuild means to you? (more…)

Script day – Amazon AWS Signature Version 4 With Bash

As anyone who works with the Amazon Web Services API knows, when you submit requests to an AWS service you need to sign the request with your secret key – in order to authenticate your account. The AWS signing process has changed through the years – an earlier version (I think version 1) I implemented in a previous blog post: upload files to Amazon S3 using Bash, with new APIs and newer versions of existing APIs opt in to use the newer signing process.

The current most up to date version of the signing process is known as Signature Version 4 Signing Process and is quite complex, but recently I had the need to use an AWS API that requires requests to be signed using the version 4 process in a bash script(1), so it was time to dust off the old scripting skills and see if I can get this much much much more elaborate signing process to work in bash – and (maybe) surprisingly it is quite doable.

With no further ado, here is the code:

(more…)


  1. I’m trying to use SQS to send change notifications from a FreeBSD jail running on a FreeNAS server – a place were I’m uncomfortable installing the AWS CLI tool or the SDK. This also help explains all the FreeBSD compatibility written into the code []

Script Day: Upload Files to Amazon S3 Using Bash

Here is a very simple Bash script that uploads a file to Amazon’s S3. I’ve looked for a simple explanation on how to do that without perl scripts or C# code, and could find none. So after a bit of experimentation and some reverse engineering, here’s the simple sample code:

(more…)

Powershell still sucks

Powershell is a great command line shell, if you all you know is cmd and batch. There are so many things it is missing when trying to compete with current Unix shells such as Bash, and while some of them have semi-working workarounds, many are sorely missing.

My pet peeves are:

  • A decent pager. “more” is basically at the same stage it was when I started working in MS-DOS 3.30, and it is nowhere near the functionality of “less”(1).
  • Persistent history. I’ve seen some workarounds but couldn’t get any of them to work properly.

Both of these features have been available to me since I started working with Linux in 1995, and it is really difficult living without them in MS-world. A decent terminal emulator will be nice too – the Powershell box has advantage over the cmd.exe box in that it is blue – other then that they are both in the same sorry state that the “dos box” of Windows 3 fame was at. I’m using “Console 2” to get some useful work done, but it too leaves much to be desired.

Also, startup is so.. fscking.. slow.. Starting Powershell on a brand new machine (with no per-session user scripts) can take as much as 3 seconds. Those are minutes of my life everyday that I would never get back.


  1. and I’m not talking about the built-in editor, just being able to “page up” would have been nice []

Canonical announced a new display server – Mir, and it is good for the consumer

Canonical have last week announced that they are developing their own display server to replace the ubiquitous X display server, a project called Mir, and the shit storm has begun anew(as what happened after Unity, Ubuntu Touch and other Canonical announcements). Contrary to popular belief, I think that this happening is a good thing for the Linux community in whole.

There are many reasons why I think this is good, most are not really concrete technical things, but I can list a few:

  • X11 is showing its age. There were some internal efforts to modernize it (e.g. kdrive which have mostly merged into the existing code) and some external efforts to replace it (Fresco and Wayland to name a few), but none have made much of an impact on the current state of Linux display.
  • From first look, Mir is a modern code base written in C++11 and Boost, which I like.
  • Diversity is generally a good thing.

If we go over the last point in a bit more depth, I think we can see why Mir would generally be a good thing for Linux developers and users and why people should stop being negative.

(more…)


Spam prevention powered by Akismet