Hola! I just started a new project on GitHub, I’m calling it ec2-extra-tools. It’s basically a set of extra command line tools that I find useful on Amazon EC2. Right now, it only has 2, but I will add more in the future and encourage other people to collaborate on the project.
Here’s what’s included so far:
- ec2-identify-instance: returns the current EC2 instance id
- ec2-identify-volume: returns the EBS volume id of a device
I needed ec2-identify-volume to automate EBS snapshots, so I thought I’d release my work on GitHub.
Here’s what I added in /etc/crontab to automate EBS snapshots:
0 2 * * * root ec2-create-snapshot `ec2-identify-volume /dev/sdj` >> /var/log/ebs_snapshots
The great thing about this line is that I don’t need to know the volume id beforehands. If a new volume is attached at /dev/sdj, I don’t have to change my crontab file either. In my situation, this is very convenient.
Hope you find ec2-extra-tools useful!
27/10/2008 at 3:57 am Permalink
The tools sounds great.
But when I look at your snapshot code I get a bit worried. A snapshot taken at a random point in time of a mounted EBS volume can be very corrupted. The volume may or may not re-mount without errors, the data in files may or may not have been written.
I would (at the least) start with a “sync” of the volume as a small start to making the volume consistent. But for best results the volume needs to be “frozen” for a moment.
If the content is all read only and changes are only made to whole files, for example a collection of .php files, this may just work. But for random access files you will see partial updates and broken links inside data files.
Caution, you have a great idea here just adding a backup to the cron. But I would add a few more steps, to get a good snapshot that would remount the next time you need it for a volume.
Edward M. Goldberg
e.m.g.
29/10/2008 at 2:08 am Permalink
Ed,
Indeed. I wouldn’t do this on something like a database storage device, but for my purpose it was enough. On our MySQL master, we use XFS, which is much better than ext3 for this purpose.
25/11/2008 at 10:32 pm Permalink
Ed, might want to look at ec2-xfs-snapshot in my ec2-extra-tools: http://github.com/cmer/ec2-extra-tools
It does the freezing and plenty of nifty tricks to automate snapshots of XFS volumes. Works great in production too
18/02/2009 at 9:28 am Permalink
Hi, Carl
> I wouldn’t do this on something like a database storage device,
> but for my purpose it was enough
Then, you should say so in the script.
People, use your script, are all newbies so they cannot tell it’s safe or not.
>On our MySQL master, we use XFS, which is much better than ext3 for this purpose.
Ouch !! yes, u can take a snapshot of the volume, but from MySQL’s perspective it could be inconsistent.
> It does the freezing and plenty of nifty tricks to automate snapshots of XFS volumes
I took a look at ur code but it doesn’t seem to do “plenty of nifty tricks”.
> Works great in production too
You’re just lucky.
I’m so sure you will get your data corrupted someday.
18/02/2009 at 2:51 pm Permalink
jack: we’ve been using these scripts in production for 6 months on our MySQL servers (which are very high traffic) and we haven’t had corrupted data. It’s just a matter of knowing how to use them, which, of course, I can’t take the responsibility of.
16/04/2009 at 8:48 pm Permalink
Hi Carl, great job!
For European instances you have to add –region eu-west-1 (or –region eu-east-1) on line 12 of ec2-identify-volume otherwise it will return nil.
Maybe you should add a parameter on your ruby script…
17/04/2009 at 6:31 am Permalink
Carl,
I would love to add a few of my “tricks of the trade” to your project.
1) Next free device for attaching a new EBS Volume.
2) Format (only if needed) then mount a Volume.
3) Patch the “Whole File Tree” from a mounted Volume, with a “sparse” tree.
I have been working on these programs for a while. The mount script is a very careful mounter for Volumes. Never damages volumes that have data. Formats as needed.
We should talk more….
Will you be at the MySQL show next week?
Call me: Skype: EdwardMGoldberg
Edward M. Goldberg
http://myCloudWatcher.com/
17/04/2009 at 12:38 pm Permalink
Ed,
You are more than welcome to fork the project on GitHub and make the changes you think are valuable. I’ll be happy to merge back!
Won’t be at MySQL Conf this year. Cheers!