what is amazon S3
Amazon has started to monetize their technology platform as part of their Amazon Web Services (AWS). I know very little about AWS in general, but I have spent some time looking into one of their initial services: Amazon Simple Storage Service (Amazon S3). This service provides a simple web service interface to store data for a small fee. The storage service is supposed to be highly scalable, reliable and fast. You pay for what you use. They charge $.15 per GB/per month for storage and they also charge you for data transfer at $.20 per GB/per month. The service is tailored towards developers and is not intended to be used as a consumer level backup product.
using S3 for personal backups
I started looking into the service to get familiar with Amazon's services. I wanted to get a feel for their level of quality, ease of use and how practical they would be to leverage from my companies own products. S3 is something that's directly interesting to me. Personally and professionally I manage a bit of data that I never want to lose. I have applications, source code, documentation, customer data and servers that need to be backed up at work. At home I have photos, videos and personal documents that need to be backed up. Ultimately I want to write code that hooks up with Amazon's services, but for this post I want to just talk about my experiences in trying to find already written code to backup my stuff.
my current backup situation
I have a mix of machines that I need to have data backed up on. The majority of the machines are Windows-based, with two that run Linux. In my basement I have one server with half a terabyte of mirrored drive space. That machine is only used as a file server and a backup server. At different intervals throughout the week, I have my other servers and desktops (from multiple locations) sending their backup data to this machine. To make that happen I install OpenSSH and a copy of rsync on each client machine. I then use a scheduled script to securely synchronize specific client folders with the server. This has been working fine for me. It's not the easiest thing to setup, but once it's done for a machine, managing it is fairly simple and restoring is as easy as plucking the files or folders of my backup server. Since SSH and rsync are cross-platform, I can use the same technique from my Linux machines.
how does S3 fit in
To prevent data loss, I depend on the fact that my data is always in two places at one time (except my photos which exist only on the file server) and that one of those places uses a RAID for storage. Ideally, I would also like to have my most sensitive data archived away where I can't accidentally screw it up. I've looked into some commercial offsite backup products but they are fairly expensive for the amount of data I want to secure (ideally close to 200GB). I would be fine with paying the higher prices if it was my only option. Fortunately Amazon S3 seems to be a low cost option. My short term goal is to keep my rsync backups going and just synchronizing my backup server with S3. Maybe at some point I will bypass my rsync step altogether and sync directly to S3.
what am i looking for
For my purposes, I would like to find software that can allow me to automatically synchronize my backup server with S3. Here are my current factors that I'm using to choose a solution:
- The software has to be automate-able, thru scripts or some other method.
- I won't need to access the data very often except to verify everything is working correctly, so the software does not have to have a UI or be very feature-ful.
- I would like to have the data stored on the S3 service in a generic format so I'm not tied to my initial choice of software. If my software is no longer usable, I should be able to retrieve my data using other software.
- The software only has to run on Windows.
- Ideally the software would be open-source.
- Ideally there would be no additional fees on top of what Amazon charges.
what i've found
Unfortunately there are limited sources of information on S3. One of the most helpful was a blog post by Jeremy Zawodny called A List of Amazon S3 Backup Tools. Check out the comments for a good view into what's available. Amazon S3 Tools on elastic8.com. Another resource was Amazon's Solution Catalog. I haven't had much luck in getting everything I want and didn't have much luck in evaluating the ones I've tried. Below are the apps I've tried with my initial findings. I will follow up this post in a couple weeks to after I've done some more research.
- JungleDisk - This service looks very promising and is free. It basically adds S3 as a virtual drive on your computer. You can access it like any other drive (theoretically). It's not open-source and stores data in a proprietary format, but they do have open-source code available to browse and download your data if something ever happens to them. I've tried a couple times but I could not get JungleDisk to work on Windows Server 2003. I've gotten it to work on Windows XP though and that worked fine.
- S3 Backup - A Windows client that allows you to access S3 like in a manner similar to Windows Explorer. The client has jobs built in that allow you to run backups automatically (no scheduling yet). The software is written by one person and has frequent updates. For some reasons the beta downloads have a built-in timeout which is unusual. The author is not sure what portion of the software will be open-sourced (if any) and plans to eventually charge for part of it. I was able to browse and upload some data, but got errors when trying to create backup jobs. I'm sure the job error will be worked out in future releases, but because of it's somewhat undefined future I will have to put this app on the back burner.
- S3Drive - Another virtual drive app like JungleDisk but from an independent developer. The software depends on the .Net Framework 2.0. To optimize the browsing thru Windows Explorer the developer chose to split the files up into chunks when storing them and uses an index file to make his flat S3 file storage appear as a folder structure. It took a couple tries to get the virtual drive to work, but once it was setup it worked very well. I saw that some people were experiencing memory leaks and communication problems with larger files (>300MB) but I have had no problems using smaller files. I've also read somewhere that there is or plans to be source code available.
- NS3 Manager - This is a free Windows client that enables you to manage data on the S3 service. It's limited in its features (upload, download and delete only) but does store data in a generic format. It can also be used to browse the proprietary formats of the other tools mentioned here. I have not been able to find a way to automate it yet and am not sure if there's any development going on. If not something I will use for my backups, it's still a valuable too just to see what's in your data store.
- #Sh3ll - A command shell for S3. I like this app. It's not fancy and has no UI. It's not easy to use. Event the name is hard (pronounced Sharp-Shell). And as you may have guessed from the name, it depends on the 1.1 or 2.0 .Net Framework. It reads and writes to the S3 store in a generic format and can be automated. It's not as straightforward as I would have liked. Since it's actually a shell, you have to pipe commands into it to script it. I guess that is done because calls into the service require a bit of context between calls (for example, you would need to specify the bucket, username and password for each invocation). I've had no problems so far with it. The nice thing is that the source is available (actually comes in the download) and I can change it as I see fit.
conclusions
I have not gone as far as doing my actual backups to S3 since I'm still in the process of evaluating the tools and the service itself. My current thoughts are to use a combination of two tools I mentioned above. For automation I can use #Sh3ll which let's me put my files on S3 in generic form. To browse and download my files with a UI I can use NS3 Manager. It would be nice to get the source of NS3, but since there is no proprietary format I'm not locked into using it. I'll let you know as I discover more.