Archive for the ‘Articles’ Category

Code Spaces break-in lessons: using your infrastructure provider for backup is a single point of failure

Summary of the events of the Code Space break-in: Code Spaces was hosting their services on Amazon Web Services VPS infrastructure. An attacker managed to gain access to their AWS administration console account and after his demands for ransom were not answered, proceeded to delete all the data in the account.

The disaster recovery plan for Code Spaces was based on having machine images and data backups stored in AWS, everything was gone, and Code Spaces basically had to shutdown.


What to do when your virtual private server is really slow

Ok, this is not a real solution for all types of problems – just a tip, that worked for me today, to try out if you can’t figure out what the problem is.

I’m running a VM on Amazon EC2, and looking at top, I saw that most of the CPU time was spent either in “steal/guest” or “IRQ“. Now steal/guest is kernel speak for “I wanted to allocate some CPU time for progams, but the hypervisor stole it” – which is not surprising on a a virtualization solution, but if it happens all the time then that means that your physical host is constantly loaded by other VMs that take as much CPU time as they can. The second item “IRQ” is time the kernel spends at handling interrupt requests from the hardware. This shouldn’t consume a significant amount of time unless the hardware has a problem – another good indication that you want to move your VPS to another physical host.


Script Day: Upload Files to Amazon S3 Using Bash

Here is a very simple Bash script that uploads a file to Amazon’s S3. I’ve looked for a simple explanation on how to do that without perl scripts or C# code, and could find none. So after a bit of experimentation and some reverse engineering, here’s the simple sample code:


Script Day: SSH to a host behind a NAT

I use SSH daily to work with different remote services, and its always a very straight-forward process… unless the remote server you want to work with is on LAN somewhere behind NAT1. When you need to access such an internal server, the only option is to SSH into the firewall2, and then SSH again to your server of choice.

But there’s a better way, and you don’t even have to fiddle with the firewall server!

(this is not actually a script, though minimal text editing is required)

The solution is actually quite simple: set up an alias in your .ssh/config file that you can use to call the remote server when you are outside the LAN (if you are inside the LAN its better to access it directly), and for that alias we will set up a ProxyCommand that will tell SSH to first access the firewall server and open a tunnel to the target LAN server.

It looks like this:

Host remote-alias
ProxyCommand ssh firewall-user@firewally-server nc lan-server 22

This set up works best if your access to the firewall-user account is without password or passphrase (using an SSH private key that is either without passphrase or already loaded in the agent), then the login is as streamlines as a direct access – but the worst is that you’d need to type in two passwords.

Enhanced by Zemanta
  1. router that does Network Address Translation so the servers address is not accessible from outside the LAN []
  2. or some other server that has legs both inside and outside the LAN – I’m using a DNATed server, what most off-the-shelf routers incorrectly call “DMZ“ []

הקלות הבלתי נסבלת של מספר ת”ז באינטרנט

באופן מעצבן יותר ויותר (ככל שהפילוט של המאגר הביומטרי נמשך ומאיים להביא עלינו אסון), כל מיני אתרי אינטרנט טריוויאלים לחלוטין דורשים מאיתנו האזרחים לספק להם את הזיהוי היחודי שלנו במאגר (הביומטרי וגם זה שלא) של הממשלה (הווה אומר “מספר תעודת הזהות”).

אני לא בטוח בכלל מה הזכות של חברות מסחריות אקראיות לדרוש מאיתנו את הפרט הזה, אבל אני די בטוח שאני לא חייב לספק אותו, וזה ממש מעצבן שהם דורשים את הזהות שלי כתנאי למכירה של כרטיס קולנוע, או שאר דברים טריוויאלים.

בגלל שאני בטוח שגם אותכן זה מעצבן, ובגלל שאני חושב שכל הפרקטיקה מטופשת (אפילו אם היא חוקית, ויש חשד קל שהיא לא), הנה לשרותכם – מחשב מספרי תעודות זהות ישראלית:

לשם הנוחות העמוד הנ”ל ידידותי לטלפונים חכמים והכתובת הישירה אליו היא:

עוד נקודה: הוצע לי ששימוש במספר ת”ז לא שלכן באתר של ספק שרות זה עבירה על החוק – סוג של גניבת זהות. אני לא חושב שזה המצב – ואני אשמח לדעתכן בנושא, אבל לשם הזהירות:

  1. אין בבנית התוכנה הנ”ל משום המלצה להציג מספרי זהות שאינם שלכם לספקי שרות ואחרים. אם אתם בוחרים לעשות זאת זה על אחריותכם בלבד.
  2. אם אתם בכל זאת בוחרים להציג מספרי זהות כך, אני ממליץ לבחור מספרים שלא סביר שנמצאים בשימוש, כמו מספרים שמתחילים בשלוש פעמים אפס או שלוש פעמים תשע

Recovering InnoDB Tables In MySQL 5

The following procedure can be used to recover InnoDB database tables from a backup of a MySQL server that had the “innodb_file_per_table” setting but all the metadata (in the “ibdata*” files) was lost – for example due to a server crash.

The process involves two steps:

  1. Recover the table structure from the .frm files
  2. Recover the data from the .ibd files (InnoDB tablespace)

There is a lot of copying the backed up files over and over to the MySQL datadir, so its useful to have the backup available on the database server machine. In my setup the backup for the databases was copied to the directory “backup” under the database’s datadir, so – for example – for the table somedb.sometable there exist files somedb/backup/sometable.frm and somedb/backup/sometable.ibd.

Additionally the process for recovering the table structures creates a lot of superfluous metadata in the InnoDB data files, so after the first stage I’m going to destroy the InnoDB data files and let the InnoDB engine re-generate them – as a result any existing InnoDB tables will be destroyed. This is important so I’ll iterate: using the procedure detailed here will destroy any existing and working InnoDB databases! So this procedure is useful to recover a destroyed database server to a new server or as a temporary measure on a temporary server to be able to dump the data to SQL files that will later be loaded into an existing server.

There likely a way to do this which is less heavy handed – for example, check out this article from Percona’s MySQL blog, but for my purpose this is enough.


How to fix “No bootable device” after Ubuntu installation

When installing Ubuntu on a disk partitioned with a GUID partition table (GPT), the Ubuntu installer doesn’t set the boot flag correctly on the partitions, and on computers with older bioses (like my Intel ICH9 board), the BIOS will not recognize the disk as bootable and will refuse to start the boot loader, claiming that “No bootable device found”.

The solution is to set the boot flag on the boot partition (the real one in the GPT) as well as on the “protective MBR partition”:

  1. Start your computer with the Ubuntu live CD again, and select “try Ubuntu before installing”
  2. When the Ubuntu desktop loads, open the dash and start “gparted”
  3. When gparted starts, select your boot drive and from that select your boot partition – usually the first Linux partition on the drive, and right click it and select “flags” in the menu. In the flags dialog check “boot”.
  4. Open the dash again and start “terminal”
  5. In the terminal type “sudo fdisk /dev/sda” (or whatever your boot drive is). You’d get a bunch of warning about the disk using a GPT scheme – ignore them. type “a” to set the bootable flag and choose partition “1”. Finally type “w” to write out your changes.

Now its safe to reboot and your old computer should have no problem starting the boot loader now.

Script Day: Automatically backup your EC2 instance using snapshots

The following script I install as a cron job on Amazon AWS virtual machines I deploy, to allow them to backup themselves automatically. The script uses the EC2 management utilities that are normally available on “Amazon Linux” installations (and can be easily installed on other Linux distributions) to create EBS snapshots of the current mounted root EBS volume1.

  1. I don’t expect this script to work for instances that have an instance-stored root device, but I don’t expect to encounter these any more []

Nvidia Optimus on Ubuntu 12.04

Nvidia Optimus is a neat solution to the problem of power consumption vs. 3D performance in notebook computers – the computer comes with two graphics chips, an “integrated graphics package” – the Intel GMA embedded graphics and a “discrete graphics package” – the Nvidia chip. The setup works by running your normal windowing UI on the integrated graphics, only powering on the discrete graphics when you want to play a 3D game or something like that.

Recently I had the fortune to work on a Lenovo T420 laptop1 that has this setup, and it works very well on MS-Windows with the Nvidia Optimus driver – you get battry life around 10 hours with the 9 cell battery.

Unfortunately I spend most of my day in Linux and because Nvidia does not support Optimus on Linux, you have the poor choice of either running everything on the Nvidia chip – decreasing battery life to around 4 hours if you’re careful, or disabling the Nvidia chip completely.

The solution comes from the Bumblebee project – a software suite to handle the switching between the Nvidia discrete graphics and the embedded Intel chip.

The setup is pretty simple to understand (though I suspect under the hood there are many problems to be solved): A service runs and waits for users to ask for 3D accelerated graphics. When a user starts a program using the special command optirun, the service loads the Nvidia driver, starts an X server using the discrete graphics (with the display disconnected from the actual screen) and runs the specified program on that “background” X server. Then it copies the visuals from the program that is rendered using the discrete graphics to a window on the main X server. When the program terminates, the service closes the secondary X server, removes the driver and powers down the graphics card – putting us back into the ~10 hours battery life.

The Bumblebee software had some problems in the past, but the current version – 3.0 – looks very good. There are a few seconds of delay when you launch the application (setting up the driver and X takes some time), but performance is about what you’d expect when running directly on the hardware. All this without any configuration – that is if you are running on the stable Ubuntu version.

As I can’t leave well enough alone, and whenever someone says “alpha”, I say – “I wants”, I’m running the current Ubuntu 12.04 alpha (which is not so bad – due to be released in a couple of months, it works very well). And of course Bumblebee doesn’t work properly here.

So this is what I had to do to get it running:


  1. with an amazing 1600×900 screen, what they call “HD+”. Truly a work of art []

LDAP authentication on Fedora 16 (and why it sucks)

In my company we (still) use an Active Directory domain controller to manage central authentication1, which is not set up very well – no SSL and the Kreberos setup is not done properly. This makes gives much trouble to modern Linuxes (e.g. not Ubuntu. yes – I’m looking at you Shuttleworth.)  such as Fedora, as Fedora have done away with NSS/PAM based LDAP authentication and instead relies on SSS – which I have yet managed to get working or even find a tutorial on how to set it up properly.

So if you still want to authentication your Fedora installation against the company’s Active Directory – and can’t/won’t rely on Winbind’s notably flaky behaviour, you can always install NSS/PAM ldap authentication manually. Unfortunately its not as easy as it sounds, and as I learned the hard way – one must pay careful attention to SELinux. So here’s the recipe:


  1. there are a lot of MS-Windows workstation, so it kinds of makes sense – but we are planning to phase it out in favour of OpenLDAP, so don’t worry about it []

Spam prevention powered by Akismet