Monday, June 04, 2007

Amazon EC2 Ephemeral Storage (/mnt) and MySQL

Amazon Elastic Compute Cloud (EC2) includes 160 GB of local storage. This storage is split into two partitions:

  1. ~10 GB for your VM/AMI image instance which includes the OS and software.
  2. ~147 GB for your "ephemeral storage" which is where you should put your data or anything that is going to need a lot of space. This storage is in /mnt.
So in order to let your MySQL database grow, you'll need to put the data files in /mnt and the easiest way to do this is to make a symbolic link from the default MySQL data storage location (usually /var/lib/mysql) to a directory on /mnt. Be sure to do this immediately after installing MySQL so you won't run into any issues.

Here is a step by step how to do it:
  1. mkdir /mnt/mysql
    1. Creates your new data directory
  2. chown mysql:mysql /mnt/mysql
    1. Gives mysql ownership to the directory
  3. /etc/init.d/mysql stop
    1. Stop MySQL before moving data files
  4. mv /var/lib/mysql/* /mnt/mysql
    1. Move your data files
  5. rmdir /var/lib/mysql
    1. Delete the old directory
  6. ln -s /mnt/mysql mysql
    1. Create a symlink to your new directory on /mnt
  7. /etc/init.d/mysql start
    1. Restart MySQL
That's all she wrote. One last very important thing to remember: This storage is NOT reliable. If your instance is terminated or crashes, your data is lost forever! So always be sure to do regular backups to S3.

4 comments:

David said...

Travis,
David (of typica "fame") here. I've done this by editing the /etc/my.cnf file. It allows you to change the data and temp dir. I put the temp dir on the ephemeral store also because we had some huge temp tables that filled up root. In general, we had to improve our queries, but this was just a safety net.
This is what my /etc/my.cnf file looks like;

[mysqld]
basedir=/usr/local/mysql
datadir=/mnt/mysql_data
tmpdir=/mnt/mysql_data/tmp/

Travis Reeder said...

Thanks David, that's a good alternative (perhaps even better).

spinron said...

From Amazon's EC2 API docs:

"If an instance reboots (intentionally or unintentionally), the data on the ephemeral store will survive. If
the underlying drive fails or the instance is terminated, the data will be lost."

The ephemeral store can survive normal reboots, but not hardware (i.e.: disk) related failures. It' a little less flaky then you think, but only a little...

Sunil & Abinash said...

I also did this exact thing. I took mysql to mnt directory via the softlink. However, still I was having sda1 filled > 80%. I then saw the /mnt/usr directory is the problem maker. My next step was to link that too to mnt. [ usr -> /mnt/usr ]. When I did it I found I have broken many relative links.. So next step was to fix them. I have detailed my experiences at http://sunilabinash.vox.com/library/post/building-an-amazon-instance-with-apache-php-tomcat-mysql-with-hadoop-s3.html

Once my server hung. I usually take incremental backups in each 30mins interval to S3 and eod master backup. There was some issue in the script and it had not run for a week. Suddently the EC2 instance hung. I felt so helpless as i was about to loose the production data. I rebooted the machine. Thank god, it survived the crash.

Cheers,
Sunil & Abinash
http://sunilabinash.vox.com
Software for developing world