Sunday, May 25, 2008

Warming Up the Hibernate Second Level Cache

I recently ran into a problem where Hibernate took forever to load up a result set because it was executing two additional queries for every object in the list to get it's @ManyToOne references. With thousands of objects returned, this meant thousands of individual sql select queries! I am using the JPA EntityManager throughout these examples.

eg:

public class MyEntity{
....
@ManyToOne
public MyEntity2 getMyEntity2(){
...
}

@ManyToOne
public MyEntity3 getMyEntity3(){
...
}

}

I had to use these many-to-one references so fiddling with lazy loading wouldn't help my situation. I was already caching the result collection after the first load, but it still was taking minutes to load up the first time and there was less than a thousand results, what gives? But 1000 * 2 extra sql select queries is 2000 individual queries which is just not good. Even the Hibernate second level cache didn't help because it still needs the first load.

There must be some way to make this startup faster right??

So I starting thinking, "how can I get these things to load up in the fewest number of sql queries?". Then it struck me, I'll warm up the cache by querying for all of the ManyToOne objects on startup, for instance:

em.createQuery("select me2 from MyEntity2").getResultList();

and

em.createQuery("select me2 from MyEntity3").getResultList();

Now when I run the original query for MyEntity, the result is almost immediate. I turned ~2000 database hits into 3!

More Reading:

Tuesday, May 20, 2008

Automated, Real-time Hard Drive Backup

As I've said before, I'm a big advocate of backup systems because I've gone through the hell of hard drive failures. A 2007 study by The Computer Science Department of Carnegie Mellon University states that:

We find that in the field, annual disk replacement rates typically exceed 1%, with 2-4% common and up to 13% observed on some systems.
With statistics like that, you better believe it will happen to you some day.

I currently use Mozy for remote backup in case I throw my computer out the window because Vista has me so frustrated. Remote backups are great and a must have in my eyes (they also give you that warm fuzzy feeling), but they are not perfect. For instance, it could take weeks to actually recover your 50 gigs of data from a remote service. You will eventually get it all, but it won't be a quick recovery.

Want zero down time? Enter RAID. Accordance Systems has a really cool backup device called the ARAID that automatically and transparently backs up your hard drive by mirroring it. Now mirroring hard drives isn't anything new, but the cool part about the ARAID device is that it acts like a normal hard drive on your computer and you can simply add it to your box without having to do anything crazy like reformatting your hard drive or installing raid controllers to get it running. And if a hard drive fails, you won't be waiting weeks to recover, in fact you won't even have to stop working because the mirrored drive takes over.

Here's the skinny on how it works:
  1. You shutdown your computer (or course)
  2. Replace your master hard drive with the ARAID unit
  3. Put your master hard drive in the top bay of the ARAID
  4. Now you start up your computer and it runs like normal as if the ARAID weren't there
  5. Now you insert a second drive into the bottom tray (yes, you can do this while the drive is
  6. The ARAID will now synchronize the master drive to the second drive on the fly
  7. Now you have a fully functioning mirrored RAID 1 array where your data is always backed up to the second drive so if one drive fails, the other takes over with no downtime.
This is pretty cool stuff, especially because they made it so easy to setup. I would definitely recommend using something like this along with a remote backup solution just in case your office burns to the ground.