Implementing revolving backups on AWS EC2

dgildeh's picture

A few weeks ago we decided to switch on hourly backups on SambaJAM and I went in search of some good automation scripts to do this...I was very disappointed.



Like a lot of new start-ups, we've been taking advantage of Amazon Web Services to host our servers and data, and if you're building stuff in the cloud it is a great service to use. The problem is they still haven't implemented an easy way to do revolving backups if you're storing data on Elastic Block Storage (EBS) volumes. By revolving backups I basically mean I keep a certain period of backups (say 2 weeks) and any backups older than that get deleted/overwritten to ensure I stay within AWS's 500 snapshot limit and only keep what I need to recover data in a disaster recovery scenario. Right now AWS provide EC2 command line tools to create snapshots of your volumes but it doesn't have any way of deleting snapshots automatically so I can only keep the last 400 snapshots, and delete any that are older.



So when I didn't find good enough tools, I did it myself and have put the final scripts on this page for anyone to use. For the first timers out there (like I was a few weeks back) I've put all the information you need to get started and use the scripts properly. They should work on any Linux distro, but for our purposes they've only been tested on Ubuntu 9.04. Also the scripts were originally written in for the EU region, but have been tested successfully with minor changes in the US region - so hopefully you shouldn't have any problems using them regardless of your location!



To begin with, you need to follow the tutorial here by Eric Hammond, and make sure your volume is formatted using XFS. This allows you to use ec2-consistent-snapshot program and lock both the MySQL database AND the filesystem temporarily while making snapshots for consistent backups. For your convenience I've attached a simple installation script (install_ec2_const_snapshot.sh) so you can quickly install the required program for the next part.



Next the other two files are the ones that execute ec2-consistent-snapshot and run a PHP script to delete snapshots automatically:

  • backup_ebs.sh - This is the main script called by a cron job that will execute both ec2-consistent-snapshot and the PHP script to start revolving backups. You call it using the following command:

    ./backup_ebs.sh vol-XXXXXXX 60

    This will take a snapshot of vol-XXXXXX (the volume Id) and only keep the last 60 snapshots for that volume. You need to also edit the file to change the following parameters:
    • Environment Variables: As Cron will not load all the necessary environment variables, we export them at the top of the file to ensure they are set correctly for the ec2-tools to execute properly. To set them to your values just type "echo $JAVA_HOME" to see the value on your system.
    • AWS_ACCESS_KEY_HERE: Replace this with your AWS Access Key you can get from your account security settings page.
    • AWS_SECRET_KEY_HERE: Replace this with your AWS Secret Key you can get from your account security settings page.
    • MYSQL_USERNAME_HERE: Your MySQL root user username.
    • MYSQL_PASSWORD_HERE: Your MySQL root user password.
    • AWS_REGION_HERE: For EU servers this will currently be eu-west-1, for US ue-east-1, us-west-1.



  • remove_old_snapshots.php - This is a PHP script that takes the number part of the argument you passed into backup_ebs.sh and uses the local EC2 command line tools to fetch a list of all the snapshots for the volume you declared, and if its greater than N, will delete the oldest ones. You need to edit the file to change the following parameters at the top:
    • AWS_ACCESS_KEY: Replace this with the file path location to your AWS private key certificate on the server. This is the file you can get from your AWS security page starting with pk-...
    • AWS_SECRET_ACCESS_CERT: Replace this with the file path location to your AWS secret key certificate on the server. This is the file you can get from your AWS account security page starting with cert-...
    • AWS_REGION_HERE: For EU servers this will currently be eu-west-1, for US ue-east-1, us-west-1.






The PHP script is based on Alvin Kreitman's PHP script here which I liked because it works by keeping the last N snapshots (as opposed to time period like the next one) but didn't like because you have to code the array of volumes and parameters in the PHP file. The other script I looked at was ec2-deleted-snapshots PHP script, but I really didn't like the fact I had to download all the AWS PHP libraries (lots of files) and the fact it works with time periods as opposed to N number of snapshots. So I've taken the best of both which is the final result. There are also some perl/Ruby scripts out there that claim to do the same thing but I'm not a perl/Ruby guy, so stuck to simple PHP.



Unlike other solutions you only need these 3 files and a cron job to get up and running with revolving backups on Amazon EC2. In order to run the cron job its as simple as editing the cron tab (you can read full information here for Ubuntu):



(as root) crontab -e

Add the following two lines to the crontab file for hourly backups:

# Hourly backup script, runs half past the hour every day

30 * * * * /bin/sh /pathToScript/backup_ebs.sh vol-XXXXXXXX 400


I hope this saves some time for people, please post any questions below, and if you're interested in seeing the final result of these scripts why not checkout our excellent collaboration platform SambaJAM! (Shameless plug)

NOTE: For security reasons, the files have been renamed to .txt files with underscores, ensure you rename them back to the file names above before running on your server for them to work properly.

AttachmentSize
install_ec2_const_snapshot.sh343 bytes
backup_ebs.sh808 bytes
remove_old_snapshots.php4.59 KB

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

I figured out my issue

On Ubuntu 10.04 with PHP 5.3.2,

/usr/bin/php -f /opt/Scripts/backup/remove_old_snapshots.php -- args -v $1 -n $2

needs to be

/usr/bin/php /opt/Scripts/backup/remove_old_snapshots.php -v $1 -n $2

Can't get remove_old_snapshots.php to work

I can't get remove_old_snapshots.php to work.

I've event tried running it as a stand alone and every time I get errors. I am running Ubuntu 10.04 with PHP 5.3

Anybody else have problems running it?

Error on the PHP file

I'm using Ubuntu 10.04 and have everything installed, but I'm getting the variables all wrong somehow.

Here's what I'm using from the command line to test this out. Eventually, I'll make this a cron job.
What am I doing wrong.

$ sudo /opt/scripts/backup/backup_ebs.sh vol-8cc356e5 -N=21
Backing up volume vol-8cc356e5
snap-a0236ec9
PHP Notice: Undefined index: v in /opt/scripts/backup/remove_old_snapshots.php on line 28
PHP Notice: Undefined index: v in /opt/scripts/backup/remove_old_snapshots.php on line 31
PHP Notice: Undefined index: n in /opt/scripts/backup/remove_old_snapshots.php on line 34
ERROR: Please provide number of snapshots to keep (-n option)
Completed

I've tried all of the following with the same error:
$ sudo /opt/scripts/backup/backup_ebs.sh vol-8cc356e5 -N=21
$ sudo /opt/scripts/backup/backup_ebs.sh vol-8cc356e5 -n=21
$ sudo /opt/scripts/backup/backup_ebs.sh vol-8cc356e5 -21
$ sudo /opt/scripts/backup/backup_ebs.sh vol-8cc356e5 21

Thanks for this script.

Update the script to include new install instructions

I just wanted to point out that Eric Hammond has created a new version of the ec2-consistent-snapshot tool, and it's much easier to install. You might want to update your script.

Here's the full details:
http://cloud.ubuntu.com/2011/02/new-release-of-ec2-consistent-snapshot-a...

From the article - "we can save lots of trouble installing ec2-consistent-snapshot by adding a dependency on the new libnet-amazon-ec2-perl package in Ubuntu instead of forcing people to install the Net::Amazon::EC2 Perl package through CPAN"

dgildeh's picture

Thanks

Thanks David, will definately take a look and update

-----------------------------------
David Gildeh - Director SambaStream

Without MySQL?

I would like to run this without having MySQL running. We have just moved our MySQL databases to RDS so we don't need to backup our local MySQL databases anymore.

What changes do I need to make so that we don't deal with the MySQL parts of the script.

I have an attached EBS volume which has all the apache files and logs, etc. I want to make sure and create a snapshot of that volume every day.

Thanks.

never mind. Its working thx

never mind. Its working thx for you steps.

Passing args to remove_old_snapshots

Hi David,

Thanks very much for this tutorial. It has been great. Under Ubuntu 10.10 with php 5.3.3, I found I was getting the same error as Mahrob did back in May 2010. After removing "-- args" I have much better success.

I have since rewritten my implementation of the backup script to use variables defined in the script (rather than passed by argument) to store the volume name and number of snapshots and I'm passing it to the purge script as follows:


php /path/to/purge_script.php -v $VOL_TO_BACKUP -n $NUM_OF_SNAPS_TO_KEEP

This is working reliably when run manually but haven't yet automated it with cron.

Cheers,
Sean

security

This is not a secure setup. An EC2 instance is not secure. Putting your AWS Certificate and private key on an EC2 instance exposes all your instances and your entire AWS account if this one instance is compromised. The problem is limited to the PHP script, not the backup_ebs.sh script.

dgildeh's picture

You're right but without them

You're right but without them you can't get revolving backups working! For us there's two things:

1. Security which you mentioned
2. Ensuring our customer data is regularly backed up

Without these scripts we can't ensure the latter, so as long as you're confident your instance is secure I guess the benefits outweigh the potential issues

-----------------------------------
David Gildeh - Director SambaStream

In

In http://www.sambastream.com/sites/sambastream.com/files/backup_ebs.sh_.txt
the word "args" should not be there on the php line I think.

dgildeh's picture

Not sure - I seem to remember

Not sure - I seem to remember without it I couldn't get it to run. I guess if it works its OK to stay ;)

-----------------------------------
David Gildeh - Director SambaStream

arguments are not being passed

Hey David,

This is just what I was looking for. Thanks a ton for putting it together.

I can run remove_old_snapshots.php from the command line and it works great. When I run backup_ebs.sh, however, the volume and number don't seem to be getting passed through:


ubuntu@domU-12-31-39-03-44-F2:~/backup$ sudo ./backup_ebs.sh vol-xxxxxxxx 60
Backing up volume vol-xxxxxxxx
snap-xxxxxxxx
PHP Notice: Undefined index: v in /home/ubuntu/backup/remove_old_snapshots.php on line 28
PHP Notice: Undefined index: v in /home/ubuntu/backup/remove_old_snapshots.php on line 31
PHP Notice: Undefined index: n in /home/ubuntu/backup/remove_old_snapshots.php on line 34
ERROR: Please provide number of snapshots to keep (-n option)
Completed
ubuntu@domU-12-31-39-03-44-F2:~/backup$

Any ideas? I'm a shell/PHP hack, and I'm stumped. Any thoughts are appreciated. Thanks!

Ben

what about Debian?

Hello David,
do you think all this wonderful tool is deployable on a Debian machine? I really wouldn't like to rebuild all my architecture, but if it's necessary.....

Thank You
Emanuele Luchetti @ www.tuotempo.com

dgildeh's picture

Apart from the install script

Apart from the install script which may be Ubuntu specific I see no reason why the scripts can't run on Debian as long as the right dependencies are installed.

-----------------------------------
David Gildeh - Director SambaStream

Thanks for sharing!

Nice work David,

Your script(s) worked in on the first try. A warm feeling has now enveloped me.

Regards,
Mike

dgildeh's picture

Glad to know! This issue was

Glad to know! This issue was bugging me for ages which is why I went ahead to publish it! Hope AWS provides a proper out of the box solution eventually

-----------------------------------
David Gildeh - Director SambaStream

Error while running the script

Hi David,

Thanks for the wonderful work!

While running the script, I am getting the following error and would really appreciate if you could help me identifying the issue.
===================
C:\remove_old_snapshots -v -n=1
PHP Notice: Undefined index: v in remove_old_snapshots.php on line 27
PHP Notice: Undefined index: v in remove_old_snapshots.php on line 30
PHP Notice: Undefined index: n in remove_old_snapshots.php on line 33
ERROR: Please provide number of snapshots to keep (-n option)
===================

Thanks
Mahrob

dgildeh's picture

You're missing the Volume ID

Hi Mahrob,

Looking at the command you put: C:\remove_old_snapshots -v -n=1 After the '-v' you need to put the volume Id. The full command should be:

C:\remove_old_snapshots -v vol-XXXXXXXX -n=1

Hope this helps,

David

-----------------------------------
David Gildeh - Director SambaStream

Nice Script

Seems a very usefull script. I am new to AWSs myself, but I will most likely use your script if I want to take regular backups while keeping a maximum number of snapshots.

Thanks!

Delete is not working from cron job

Hi,
its very nice. it works very good from terminal on ubuntu server. When i run a cron job the backup of EBS is working, but it is not deleting the sanpshopts.

iam unable to trace the problem.

how to configure the cron job to delete the old snapshots.

Thanks
C.Kiran Kumar

dgildeh's picture

SOLVED!

Hi Kiran,

It took me ages to solve this issue too but I got to the bottom of it! Basically my script was missing the environment variables being set for ec2-tools to run properly and the way the php executable was called was not escaping the arguments with "-- args".

I've updated the cron expression and files above and have successfully retested everything on my system so this problem should be solved now.

Thanks,

David

-----------------------------------
David Gildeh - Director SambaStream

I have a problem with the the

I have a problem with the the backup_ebs.sh. The snapshot is created but de remove_old_snapshots.php script has errors. Running the remove_old_snapshots.php works fine but not in the backup_ebs.sh. Can somebody see a solution to solve this ?

root@backup:~# crontab -l
# m h dom mon dow command
00 * * * * /usr/local/sbin/other_scripts/backup_ebs.sh vol-7053dc19 5

root@backup:~# tail -f /var/log/backup.log

********** Starting backup for volume vol-7053dc19...
PHP Deprecated: Comments starting with '#' are deprecated in /etc/php5/cli/conf.d/imagick.ini $
============
Input parameters:
Volumes to prune:
Array
(
[0] => vol-7053dc19
)
Keep Quantity = 5
============
sh: ec2-describe-snapshots: not found
PHP Warning: array_multisort(): Argument #1 is expected to be an array or a sort flag in /usr/$
The current array of snapshots ordered by date and volume:
Array
(
)
==============
** Searching for volume vol-7053dc19...
WARNING: Could not find volume vol-7053dc19
********** Ran backup: Sat Apr 16 04:00:02 PDT 2011

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.

More information about formatting options

Latest Tweets

SambaStream Newsletter

Follow our Product Updates and Company News with our Monthly Newsletter

Syndicate content
Site map SambaStream | Creating the Future Workplace News