Code Spaces break-in lessons: using your infrastructure provider for backup is a single point of failure

Summary of the events of the Code Space break-in: Code Spaces was hosting their services on Amazon Web Services VPS infrastructure. An attacker managed to gain access to their AWS administration console account and after his demands for ransom were not answered, proceeded to delete all the data in the account.

The disaster recovery plan for Code Spaces was based on having machine images and data backups stored in AWS, everything was gone, and Code Spaces basically had to shutdown.

It is obvious that backing up your VPS hosted data to the VPS provider’s file storage – a very common use case – is not enough. If your cloud infrastructure account is compromised, or sometimes failures just happen, then your entire business will be compromised.

The most basic of disaster recovery plans always call for off-site backup, and in this case storing the backups on the same provider is still on-site – even if the provider offers different services at different geographical areas. A single event can still take out the entire system.

A reliable disaster recovery plan for AWS hosted systems must include backup of the important data to S3, and then also backing up the backup to a different location, either a self-hosted data server or another cloud infrastructure provider (which must be set up with a different credentials set then the first one).

The disaster recovery plan must, obviously, also include being able to recover from failure using the off-site data, to a different infrastructure provider. If your AWS system is compromised, its not far fetched to create a new AWS account and recover to that, but if Amazon has suffered a serious failure, you should be able to recover your system to Rackspace or Azure. This is simply a common sense business continuity plan.

When planning for disaster recovery, you must consider the failure of any system, and failing to secure access to your cloud provider administration console is just one of these systems that can fail.

Leave a Reply