Archive for the ‘Programming’ Category

Script Day: Automatically backup your EC2 instance using snapshots

The following script I install as a cron job on Amazon AWS virtual machines I deploy, to allow them to backup themselves automatically. The script uses the EC2 management utilities that are normally available on “Amazon Linux” installations (and can be easily installed on other Linux distributions) to create EBS snapshots of the current mounted root EBS volume(1).

  1. I don’t expect this script to work for instances that have an instance-stored root device, but I don’t expect to encounter these any more []

Script Day: find the oldest file in a directory structure

This piece of script came in handy when I wrote a utility that “recycles” space on a logging partition: before log rotation archives the current log file, we move some old log files (depending on some archive freshness policy) to a remote storage that archives older files.

The problem is that the “old archive storage” also has limited disk space and I got fed up managing the archive by hand. The solution I came up is to scan the hierarchy of  log files in the storage (logs are stored hierarchically according to origin and type) and delete old files until I have enough room to move some newer files in. That way the “old archive storage” is always kept full and keeps as much back-log as possible and does this automatically.

The piece of code that determines which files we want to delete works like this:

  1. Use find to list all the files in the directory structure
  2. Pipe it to perl and collect all the file names in a list
  3. Use perl’s sort operator to compare the modification times of each file in the list and show them in the order (i.e. oldest first)
  4. Use head to get just the first file

So it looks like this:

find /mnt/httpd_back/ -type f | perl -nle 'next unless -f; push @files, $_; END { foreach $file (sort { @a=stat($a); @b=stat($b); $a[9] <=> $b[9] } @files) { print $file; }}' | head -n1

Note: normally we use head to get some initial output and terminate the process early before it does more costly work – when head has enough data it terminates the pipe sending SIGPIPE to the upstream process and that usually terminates the process that generates the data. In this case – and in all other cases involving sort – the upstream process buffers all the data in its own memory before outputting anything, so it can sort everything, and using head here is just a filter to get what I want and does not actually save me from doing all the work. I could have easily done the same thing inside the perl script itself by replacing the block of  print $file; with print $file; last; – this has the same effect as using head, because head will send SIGPIPE to perl after getting the first print and will terminate it. Deciding which way you want to go is probably more about readability of the code and I prefer my original version because its easier to read to non-perl specialists.

I can then just remove that file, see if I have enough room to move in the newer log file and if no – repeat the process.

This would work well, I believe, but it may be inefficient if I find a bunch of small files and I want to copy in a large file. So what I did next is to take advantage of the fact that all the log files I have are named using the following simple format:


and that allows me to easily find all the log files that record the same day and eliminate them at the same time. Subsequent moving of additional files will likely succeed because I cleared out all the log files of an entire day. If not, I can always go and clear up another day’s worth of logs.

Enhanced by Zemanta

What kind of personality question you’d be?

While filling in a silly personality questionnaire (you know the annoying type: If you’d have been X, what type of X would you be?), I figure out the most computer geeky personality question ever!

It was obviously way too geeky for the task at hand, so here it is now for you to answer – or better yet: invent your a worst geeky personality question:

If you’d have been a design pattern (or an anti-pattern), which design pattern you’d be and why?

I, for laughs, I’d say I would have been a Duff’s device – complicated and impossible to debug 😉

Enhanced by Zemanta

More Internet Explorer Funny Behaviors

Sometimes, when you try to develop things that work in Internet Explorer, you get to a point where you can just scratch your head and wonder “what the he*$ where they thinking of?!?”. This is one of those cases:

This is in Internet Explorer 8 when set to IE7 mode, but this is very faithful to the original as I’ve tested it on a real IE7 and it behaves the same. What happens here is that when you scroll the page down, all absolutely positioned elements (the two “combo boxes” which are a custom UI widget and the text “Dimensions:… at the top right corner”) get pulled down a few pixels. When you scroll up, they get put back in the correct place.


Kohana 3 RHEL/CentOS RPMs

As I have not found any available, here is my build for Kohana – the PHP development framework – for RHEL 5 based operating systems.

You can find Kohana RPM for the current stable release 3.0.5 here, and the source RPM is available here in case you want to rebuild it yourself (and you might, details follow). New releases to correspond with new releases from Kohana will be updated there as needed.

This package is built on a CentOS 5.4 machine, with pretty much default settings.


Yet another programing language

Google have declared a new programming language – Go. Contrary to how the title might read, this post is not a tirade about the abundance of programming language – I just like this sort of expressions (and Google could have easily named Go as YAPL instead 😉 ). On the contrary – I’m all for a large selection of good programming language to choose from, and a programming language coming from there, like any other Google product released, definitely warrants a more serious test then what I can offer in this rather quick post.

But I actually found out about Go from a friend that asked about it, didn’t mention Google, and gave me a link to the home page at where Google is not mentioned. So my review below was not influenced by any Google hype, and on the face of it, after reading through about half of the tutorial and a couple of other documents, I don’t like it.


How many lines should a function have?

This question is probably one of the most debatable in programming, ever, and I wasn’t much surprised to find it also on Stack Overflow(1).

Actually, I wasn’t surprised to find about a dozen different questions in the same gist, but here is the one that I really liked, especially some of the more interesting answers: How many lines should a method typically have?.


  1. The best programming questions and answers site – if you’re a programmer by trade or hobby and you’re not familiar with Stack Overflow, then go register. []

Script day: grep in jar (or zip) files

Here is another script I wrote for work and I thought it will be interesting enough to share:

Say you want to check which JAR files (or ZIP files for that matter, as Java ARchive files are just ZIP files with a different extension) contain files that contain some text. grep is the obvious answer, but how to grep files in JARs?


The Insanity of Upstream

Sometimes the Java community, or more specifically the people that write Java open source software, drive me nuts!

For the past couple of week I’ve been trying to build a new version for the Jetty package based on the current Jetty6 package from JPackage(1), and in the process combating its hellish dependency tree and the way open source Java projects build opon each other in a complicated, confusing and often circular manner.


  1. an excellent excellent project that is operated by talented people in what I can only guess is what little free time they have []

Script day – Shutting down multiple servers at once

A system administrator in my company recently approached me with a problem – how to shutdown multiple Linux servers at the same time from a central location. Apparently this is something that people in the MS-Windows world use all kinds of applications, like the Remote Shutdown Tool from Microsoft (though I don’t understand how they handle the authentication – this tools doesn’t seem to require any authentication so it appears that any person with network access can shutdown any computer).

Anyway, apparently searching the web for “Linux remote shutdown” yields no useful results (or so I’ve been told), but frankly – when you have standard UN*X tools at your fingertips, a remote shutdown tool is simply typing ssh root@server shutdown -r now at your local console. But still, for people who want a “tool” – read on.


Spam prevention powered by Akismet