Friday, June 15, 2007

Third App Migrated to Amazon EC2

A couple of days ago, I moved another application to Amazon Web Services. This is my third application I've setup on their infrastructure and I have to say, I think I'm addicted to the throwaway nature of Amazon Elastic Compute Cloud. I've got the whole long term, persistent storage thing nailed and can do quick recoveries so I am no longer worried about servers crashing... and boy that's a good feeling.

On our previous dedicated servers, there's always a worry about having a server meltdown because there is no quick way to get a new one up and running. It could take a day or more. Now I just launch new servers for fun. ;)

The big lessons learned:

  1. Automate absolutely everything. So when something goes wrong, you run a script or two to get a new instance running. And practice, practice, practice doing emergency recoveries (I've had to recover from crashed instances several times now so I learnt this lesson the hard way).
  2. Scale early. As soon as you notice some performance or memory limitations, start adding more instances to spread out the load. Need performance, put your database on a separate server from app. Need redundancy, replicate your database on two servers. Need memory, fire up some more instances just for caching (memcached, ehcache, jbosscache).
  3. Keep your database as small as possible. Store as much data as possible on Amazon Simple Storage System (S3), NOT in your database and just store the S3 key to the data in your database. Consider putting any blob or large text fields in S3. This will make it much easier and faster to manage database backups, plus your database will perform better.
Are you an EC2 user? Let me know your experiences.

Monday, June 04, 2007

Amazon EC2 Ephemeral Storage (/mnt) and MySQL

Amazon Elastic Compute Cloud (EC2) includes 160 GB of local storage. This storage is split into two partitions:

  1. ~10 GB for your VM/AMI image instance which includes the OS and software.
  2. ~147 GB for your "ephemeral storage" which is where you should put your data or anything that is going to need a lot of space. This storage is in /mnt.
So in order to let your MySQL database grow, you'll need to put the data files in /mnt and the easiest way to do this is to make a symbolic link from the default MySQL data storage location (usually /var/lib/mysql) to a directory on /mnt. Be sure to do this immediately after installing MySQL so you won't run into any issues.

Here is a step by step how to do it:
  1. mkdir /mnt/mysql
    1. Creates your new data directory
  2. chown mysql:mysql /mnt/mysql
    1. Gives mysql ownership to the directory
  3. /etc/init.d/mysql stop
    1. Stop MySQL before moving data files
  4. mv /var/lib/mysql/* /mnt/mysql
    1. Move your data files
  5. rmdir /var/lib/mysql
    1. Delete the old directory
  6. ln -s /mnt/mysql mysql
    1. Create a symlink to your new directory on /mnt
  7. /etc/init.d/mysql start
    1. Restart MySQL
That's all she wrote. One last very important thing to remember: This storage is NOT reliable. If your instance is terminated or crashes, your data is lost forever! So always be sure to do regular backups to S3.