Space Program Blog

Deleting SVN Directories from a Project

2008-11-04T17:59:00.001-08:00

Sometimes you just need to get rid of those pesky subversion folders because you can’t export or something so here’s the quick and easy way to do it in windows:

Be sure you can see hidden files and folders (Advanced folder settings)
Open Windows Explorer in the root folder of your project
Explorer
In the search bar (or Ctrl+F for older versions of windows), type “.svn”
Sort by name
Delete all the ones that are folders (not the ones that are files)

Done.

Callable and the Case of the Disappearing Exceptions

2008-10-07T00:35:00.001-07:00

If you have a class that implements Callable, but you don't plan on using the Future that is returned after you call Executor.submit(), you may lose any exceptions that are thrown. They literally disappear into the void. No log whatsoever.

So it is good practice to treat it a bit like you would with a Runnable and wrap the code in your call method with a try/catch to print the exception or log it somewhere.

How To Give a User Root Access on Linux

2008-10-02T16:02:00.001-07:00

NOTE: I take no responsibility for using the following and do not recommend using it.

But for those that want it:

usermod -G root username

This changes the user with ‘username’ to the root group and thus gives root access.

And remember, as Uncle Ben says, “with great power comes great responsibility”.

Using JAXB Without Annotations

2008-10-01T13:16:00.001-07:00

The latest JAXB included with JDK 1.6 is really a great addition to the JDK making it easy to send objects around in XML format without requiring any third party libraries. All you have to do is put an @XmlRootElement annotation on a class and then call:

marshaller.marshal(Object, OutputStream);

Example class using @XmlRootElement:

@XmlRootElement
class MyClass {
  String name;
  String content;
}

To read the object back in, you simply call:

MyClass object = unmarshaller.unmarshal(InputStream);

But what if you are building a library that doesn’t know what the class is and doesn’t have the @XmlRootElement annotation? Well it turns out that it’s almost just as easy without it you just need to let JAXB know what class you expect.

First get the JAXBContext like this:

JAXBContext context = JAXBContext.newInstance(object.getClass());

To marshal:

marshaller.marshal(new JAXBElement(new QName(object.getClass().getSimpleName()), object.getClass(), object), out);

To unmarshal:

JAXBElement o = unmarshaller.unmarshal(new StreamSource(inputStream), expectedType);
MyClass object = o.getValue();

That’s it.

Warming Up the Hibernate Second Level Cache

2008-05-25T17:53:00.000-07:00

I recently ran into a problem where Hibernate took forever to load up a result set because it was executing two additional queries for every object in the list to get it's @ManyToOne references. With thousands of objects returned, this meant thousands of individual sql select queries! I am using the JPA EntityManager throughout these examples.

eg:

public class MyEntity{
....
@ManyToOne
public MyEntity2 getMyEntity2(){
...
}

@ManyToOne
public MyEntity3 getMyEntity3(){
...
}

}

I had to use these many-to-one references so fiddling with lazy loading wouldn't help my situation. I was already caching the result collection after the first load, but it still was taking minutes to load up the first time and there was less than a thousand results, what gives? But 1000 * 2 extra sql select queries is 2000 individual queries which is just not good. Even the Hibernate second level cache didn't help because it still needs the first load.

There must be some way to make this startup faster right??

So I starting thinking, "how can I get these things to load up in the fewest number of sql queries?". Then it struck me, I'll warm up the cache by querying for all of the ManyToOne objects on startup, for instance:

em.createQuery("select me2 from MyEntity2").getResultList();

and

em.createQuery("select me2 from MyEntity3").getResultList();

Now when I run the original query for MyEntity, the result is almost immediate. I turned ~2000 database hits into 3!

More Reading:

Enabling the hibernate second level cache

Automated, Real-time Hard Drive Backup

2008-05-20T15:41:00.000-07:00

As I've said before, I'm a big advocate of backup systems because I've gone through the hell of hard drive failures. A 2007 study by The Computer Science Department of Carnegie Mellon University states that:

We find that in the field, annual disk replacement rates typically exceed 1%, with 2-4% common and up to 13% observed on some systems.

With statistics like that, you better believe it will happen to you some day.

I currently use Mozy for remote backup in case I throw my computer out the window because Vista has me so frustrated. Remote backups are great and a must have in my eyes (they also give you that warm fuzzy feeling), but they are not perfect. For instance, it could take weeks to actually recover your 50 gigs of data from a remote service. You will eventually get it all, but it won't be a quick recovery.

Want zero down time? Enter RAID. Accordance Systems has a really cool backup device called the ARAID that automatically and transparently backs up your hard drive by mirroring it. Now mirroring hard drives isn't anything new, but the cool part about the ARAID device is that it acts like a normal hard drive on your computer and you can simply add it to your box without having to do anything crazy like reformatting your hard drive or installing raid controllers to get it running. And if a hard drive fails, you won't be waiting weeks to recover, in fact you won't even have to stop working because the mirrored drive takes over.

Here's the skinny on how it works:

You shutdown your computer (or course)
Replace your master hard drive with the ARAID unit
Put your master hard drive in the top bay of the ARAID
Now you start up your computer and it runs like normal as if the ARAID weren't there
Now you insert a second drive into the bottom tray (yes, you can do this while the drive is
The ARAID will now synchronize the master drive to the second drive on the fly
Now you have a fully functioning mirrored RAID 1 array where your data is always backed up to the second drive so if one drive fails, the other takes over with no downtime.

This is pretty cool stuff, especially because they made it so easy to setup. I would definitely recommend using something like this along with a remote backup solution just in case your office burns to the ground.

How To Set Up Dynamic DNS for your Amazon EC2 Instance

2008-03-06T11:11:00.001-08:00

Here's a step by step to setting up dynamic dns for you EC2 instance. This is required because EC2 does not have static IPs (yet).

First of all, you need a dynamic dns account. You can get one at www.dyndns.com or my personal favorite, www.zoneedit.com.
After you sign up, be sure to add your domains to their system.
Download dyndns client
1. run: wget http://cdn.dyndns.com/ddclient.tar.gz
2. The download includes Installation Instructions in the README, but the step by step is below
tar -zxvf ddclient.tar.gz
cd ddclient-X.Y
cp ddclient /usr/sbin/
mkdir /etc/ddclient
nano -w /etc/ddclient/ddclient.conf

Then fill in config file:

FOR ZONEEDIT

# Basic configuration file for ddclient
#
# /etc/ddclient.conf
daemon=600
pid=/var/run/ddclient.pid
use=web, web=checkip.dyndns.org/, web-skip='IP Address'
login=[USERNAME
password=[PASSWORD]
protocol=zoneedit1
server=www.zoneedit.com
wildcard=YES
[YOUR DOMAIN(S), COMMA DELIMITED LIST]

FOR DYNDNS

# Basic configuration file for ddclient
#
# /etc/ddclient.conf
daemon=600
pid=/var/run/ddclient.pid
use=web, web=checkip.dyndns.com/, web-skip='IP Address'
login=[USERNAME
password=[PASSWORD]protocol=dyndns2
server=members.dyndns.org
wildcard=YES
[YOUR DOMAIN(S), COMMA DELIMITED LIST]
custom=yes, example.com

More info on configuration

Continuing on:

cp sample-etc_rc.d_init.d_ddclient /etc/rc.d/init.d/ddclient
/sbin/chkconfig --add ddclient

This makes it startup automatically next time you reboot

/etc/rc.d/init.d/ddclient start
1. This is to start it the first time by hand
2. DON'T RUN THIS until your server is fully setup if you are moving servers because people will start hitting this server after you run this

That's it, not too difficult and gives you a real domain name to point at your EC2 instance.

Transparent Persistence is Here

2008-01-27T14:11:00.001-08:00

The idea of not having to deal with storing your application's data has long been a pipe dream, but it seems that the folks behind the db4o object database have made it a reality. I've written a couple of posts in the past about db4o and how easy it is to use and it appears to be getting easier and easier with Transparent Activation and Transparent Update.

Transparent activation is basically just lazy loading objects in the object graph on an as needed basis. Transparent Update is the opposite; storing objects when they get dirty without you having to explicitly say so.

You can check out the press release on Transparent Update here.

Getting Started with JAX-RS and Jersey - Simple REST Web Services for Java

2008-01-10T18:34:00.001-08:00

I came across the need once again to build a public facing web services API and we all know that to be in the cool gang these days, you have to do it REST style. Now I've been a big fan of REST over SOAP (or any other type) for a long time for one simple reason: it's simpler. And simple is king in my books. Anyways, I decided to see if anyone had come up with a good, simple REST framework for java since there hasn't really been anything that I've found over the years.

With a little digging, I found what I had been looking for and it goes by the name JAX-RS (jsr311). Now this is pretty bleeding edge, the buzz has just started in the past few months and there's not much around to help you out. But as you should see in the samples below, it's well worth checking out.

Now lets get into the nitty gritty:

1. Download Jersey and add required jars to classpath

Jersey is the reference implementation, get it here.

Now you only need three jars from the Jersey download:

asm-3.X.jar
jersey.jar
jsr311-api.jar

That's all you need. Compare that to Axis and this alone should make you want to switch.

2. Add the Jersey Servlet to web.xml

You can use a lightweight http server that comes with the JDK or Jersey, but who's gonna do that in the real world? So here is how to get it running in Tomcat.

<servlet>
        <servlet-name>JerseyWebApplication</servlet-name>
        <servlet-class>com.sun.ws.rest.spi.container.servlet.ServletContainer</servlet-class>
        <load-on-startup>1</load-on-startup>
    </servlet>
    <servlet-mapping>
        <servlet-name>JerseyWebApplication</servlet-name>
        <url-pattern>/rest/*</url-pattern>
    </servlet-mapping>

3. Make a service bean

Now all you have to do is make service beans via JAX-RS annotations. This is remarkably easy. We'll start with the simplest Hello World example:

@Path("/helloworld") // sets the path for this service
public class HelloWorldBean {
   @GET // This method will process GET 
               // requests to the @Path value set above
   @ProduceMime("text/html") // content type to output
   public String getClichedMessage() {
       return "Hello World";
   }
}

4. Run Tomcat and browse to /helloworld

Startup tomcat with your /WEB-INF/web.xml, the 3 jars in /WEB-INF/lib, and your compiled HelloWorldBean in /WEB-INF/classes. Now browse to http://localhost:8080/yourContext/rest/helloworld

You should see in your browser: Hello World.

That should give you all the basics you'll need to get started. Next time I'll post how to easily send XML responses directly from your objects.

Monitor Your Servers in Facebook

2007-11-01T15:36:00.001-07:00

If you have been thinking of an excuse to tell your boss why you need to use Facebook at work, here's your answer.

Monitor your web servers in Facebook! Get a notification when your site goes down and when it comes back up. Show your server status on your profile too so your friends can see if you're a network admin rockstar or not.
Features

Notifications via Facebook feed, email and SMS
Checks every ten minutes
10 second setup
FREE!

Check it out here.

New Scaling Out Blog at www.scalingout.com

2007-10-16T22:20:00.001-07:00

I started a new blog focused on Amazon Web Services and cloud computing over at www.scalingout.com. It will cover the latest news about the industry and showcase how people are using EC2, S3, etc. Check it out, subscribe, let me know your thoughts.

How to Add OpenID Support to your Java Application

2007-10-02T14:46:00.001-07:00

Here is a quick and easy step by step to add OpenID support to your application. We are using joid because it's the lightest weight implementation with the least number of dependencies (2 jars).

Download joid from http://code.google.com/p/joid/
Copy joid.jar, log4j-*.jar, and tsik.jar to your lib directory (so they end up in WEB-INF/lib).
Add OpenIdFilter to your web.xml (see below for how to add it)
Add OpenId login form (see below for a sample jsp page)

After a user logs in, you can access the username that they're signed in as with:

String loggedInAs = OpenIdFilter.getCurrentUser(session);

Simple huh?

OpenIdFilter for web.xml

<filter>
<filter-name>OpenIdFilter</filter-name>
<filter-class>org.verisign.joid.consumer.OpenIdFilter</filter-class>
<init-param>
<description>Optional. Will store the identity url in a cookie under "openid.identity" if set to true.</description>
<param-name>saveInCookie</param-name>
<param-value>true</param-value>
</init-param>

</filter>
<filter-mapping>
<filter-name>OpenIdFilter</filter-name>
<url-pattern>/*</url-pattern>
</filter-mapping>

OpenID Login Form

<%@ page import="org.verisign.joid.consumer.OpenIdFilter" %>
<%@ page import="org.verisign.joid.util.UrlUtils" %>
<%@ page contentType="text/html;charset=UTF-8" language="java" %>
<%
String returnTo = UrlUtils.getBaseUrl(request);

if (request.getParameter("signin") != null) {
try {
String id = request.getParameter("openid_url");
if (!id.startsWith("http:")) {
id = "http://" + id;
}
String trustRoot = returnTo;

String s = OpenIdFilter.joid().getAuthUrl(id, returnTo, trustRoot);
response.sendRedirect(s);
} catch (Throwable e) {
e.printStackTrace();
%>
An error occurred! Please press back and try again.
<%
}
return;
}
%>
<html>
<head><title>A Page I Want to Login To</title></head>
<body>
<h1>Login</h1>
<p>
This is a sample login page where a user enters their OpenID url to login.
</p>

<%
String loggedInAs = OpenIdFilter.getCurrentUser(session);
if(loggedInAs != null){
%>
<p align="center">
<span style=" background- padding:5px;">You are logged in as: <%=OpenIdFilter.getCurrentUser(session)%></span> - <a href="logout.jsp">Logout</a>
</p>
<%
}
%>

<div style='margin: 1em 0 1em 2em; border-left: 2px solid black; padding-left: 1em;'>
<form action="index.jsp" method="post">
<input type="hidden" name="signin" value="true"/>
<b>Login with your OpenID URL:</b> <input type="text" size="30" value=""
name="openid_url"/>
<input type="submit" value="Login"/><br/>
<i>For example: <tt>someone.bloghost.com</tt></i>
</form>
</div>

<p>
<strong>Don't have an OpenID?</strong> <a href="https://pip.verisignlabs.com/" target="_blank">Go</a>
<a href="http://www.myopenid.com/" target="_blank">get</a>
<a href="https://myvidoop.com/" target="_blank">one</a>.
</p>

</body>
</html>

Downloading JDK to your Linux Server via SSH

2007-08-10T18:21:00.001-07:00

This is a pretty common problem that I'm sure almost every java developer has had to deal with. Due to Sun's licensing restrictions, you cannot simply use up2date, apt-get or yast to get the latest JDK. Even if you add the extra repositories for them to look at, it's usually not the latest JDK. On Windows, it's easy since you generally always have a UI via Remote Desktop. Linux is usually accessed via SSH so all you have is your shell command line which means you can't surf on over to Sun's Java site, click "Ok, I accept all the terms and conditions", and download it. So you end up downloading it on your local machine, then uploading it to your server via SCP. But this takes forever because you download 60 MB on your slow local connection, then upload it even slower (assuming your upload speed is slower than you download speed). So an hour later you finally get your jdk on the server.

So is there a better, quicker way to do it?

I'm glad you asked. Yes there most definitely is:

Go to jdk download site: http://java.sun.com/javase/downloads/
Click Download button on the distro you want (usually the basic JDK)
Accept the license agreement
Right click the download link (usually the RPM in self extracting bin) and in the right click context menu choose Copy Link Location if using Firefox or Copy Shortcut in Internet Explorer.
Now you have the location of the download so go to your SSH shell where you are logged into your Linux server and use the following command:
wget PASTE_FILE_LOCATION_HERE -O jdk6-rpm.bin

You need the -O otherwise the filename length is too long

Now install it:

chmod a+x jdk6-rpm.bin
./jdk6-rpm.bin

And finally, add JAVA_HOME to your environment variables:

create or edit .bash_profile file in /root and add "export JAVA_HOME=/usr/java/latest"

That's about it. Much much faster.

Third App Migrated to Amazon EC2

2007-06-15T10:27:00.001-07:00

A couple of days ago, I moved another application to Amazon Web Services. This is my third application I've setup on their infrastructure and I have to say, I think I'm addicted to the throwaway nature of Amazon Elastic Compute Cloud. I've got the whole long term, persistent storage thing nailed and can do quick recoveries so I am no longer worried about servers crashing... and boy that's a good feeling.

On our previous dedicated servers, there's always a worry about having a server meltdown because there is no quick way to get a new one up and running. It could take a day or more. Now I just launch new servers for fun. ;)

The big lessons learned:

Automate absolutely everything. So when something goes wrong, you run a script or two to get a new instance running. And practice, practice, practice doing emergency recoveries (I've had to recover from crashed instances several times now so I learnt this lesson the hard way).
Scale early. As soon as you notice some performance or memory limitations, start adding more instances to spread out the load. Need performance, put your database on a separate server from app. Need redundancy, replicate your database on two servers. Need memory, fire up some more instances just for caching (memcached, ehcache, jbosscache).
Keep your database as small as possible. Store as much data as possible on Amazon Simple Storage System (S3), NOT in your database and just store the S3 key to the data in your database. Consider putting any blob or large text fields in S3. This will make it much easier and faster to manage database backups, plus your database will perform better.

Are you an EC2 user? Let me know your experiences.

Amazon EC2 Ephemeral Storage (/mnt) and MySQL

2007-06-04T10:52:00.000-07:00

Amazon Elastic Compute Cloud (EC2) includes 160 GB of local storage. This storage is split into two partitions:

~10 GB for your VM/AMI image instance which includes the OS and software.
~147 GB for your "ephemeral storage" which is where you should put your data or anything that is going to need a lot of space. This storage is in /mnt.

So in order to let your MySQL database grow, you'll need to put the data files in /mnt and the easiest way to do this is to make a symbolic link from the default MySQL data storage location (usually /var/lib/mysql) to a directory on /mnt. Be sure to do this immediately after installing MySQL so you won't run into any issues.

Here is a step by step how to do it:

mkdir /mnt/mysql

Creates your new data directory

chown mysql:mysql /mnt/mysql

Gives mysql ownership to the directory

/etc/init.d/mysql stop

Stop MySQL before moving data files

mv /var/lib/mysql/* /mnt/mysql

Move your data files

rmdir /var/lib/mysql

Delete the old directory

ln -s /mnt/mysql mysql

Create a symlink to your new directory on /mnt

/etc/init.d/mysql start

Restart MySQL

That's all she wrote. One last very important thing to remember: This storage is NOT reliable. If your instance is terminated or crashes, your data is lost forever! So always be sure to do regular backups to S3.

How To Fix LinkageError when using JAXB with JDK 1.6

2007-05-11T01:05:00.000-07:00

If you run across an error like this when trying to use JAXB:

java.lang.LinkageError: JAXB 2.0 API is being loaded from the bootstrap classloader, but this RI
(from jar:file:/somedirectory/jaxb-impl.jar!/com/sun/xml/bind/v2/model/impl/ModelBuilder.class) needs 2.1 API. Use the endorsed directory mechanism to place jaxb-api.jar in the bootstrap classloader. (See http://java.sun.com/j2se/1.5.0/docs/guide/standards/)

It's actually a very simple fix, but painful enough to warrant a post. Put the jaxb-api.jar that you're trying to use into JDK_HOME/jre/lib/endorsed. If the endorsed directory doesn't exist, make it. This is apparently only a problem with JDK 1.6, not with JDK 1.5.

Liveness Rule #2: Asynchronize Everything for Writes

2007-04-27T01:35:00.000-07:00

And by everything, I mean at least as much as you possibly can. Writes are slow for all the same reasons brought up in the caching post, and then some. With writes you can add more potential performance speed bumps to "database hit list", such as table locking or row locking depending on how the database handles it and disk write speeds.

Caching works wonders on the Read side and because caching allows you to take the Read load off the database, it actually does help a lot to increase Write performance. What else can we do to increase liveness for operations that might take more than a fraction of a second? The answer: Go asynchronous.

Good examples of things that should be asynced:

Sending mail notifications. This usually takes a significant amount of time, so you should always async this
Database updates that the user doesn't need to see a response for. For instance, logging, counters (how many times has the user viewed item X), etc?

How Do I Implement Asynchronous Ops?

If you're using Java, the java.util.concurrent package is your new best friend. This has everything you need to asynchronize things in a simple and elegant way. The classes you will be interested in are Executor, ExecutorService, Executors (static factory methods). An Executor is essentially just a queue of tasks with a thread pool to dedicated to executing those tasks.

Here's a quick getting started guide to asynchronicity heaven:

Create the ExecutorService:
ExecutorService executor = Executors.newFixedThreadPool(THREAD_POOL_SIZE);
Keep this available as a global variable, you can use this same Executor throughout your entire application if you want.
Now pull out any code you want to execute asynchronously into small tasks that implement Runnable. For example, an emailer task might look like this:
public class EmailTask implements Runnable {
private Email email; // the email we want to send.
public EmailTask (Email email) {
this.email = email;
}
public void run() {
// SEND MAIL HERE
}
}
Now when executing the event action, create a new instance of your Task class and pass it to the ExecutorService's execute command:
EmailTask emailTask = new EmailTask(email);
executorService.execute(feedRetrieverTask);
Done! That task will execute at some point in the future and you can respond to your user immediately without making him/her wait.

Another huge benefit of Asynchronizing things is that application automatically becomes more scalable and doesn't degrade as traffic goes up. Lets say you did the above example synchronously and it took 2 seconds on average to send an email. That may be fine when you have a small number of users (although 2 seconds is way too long to make someone wait), but what happens if you have have 1000 users? That's a total of 33.3 minutes (2000 seconds) of waiting time just for 1000 users. And lets say you have a maximum of 200 processing threads for your app server (Tomcat default). It won't take much to run out of processing threads if your traffic goes up, so now you have people waiting to get the application to respond at all. Pretty soon your app is on it's knees, dying a slow painful death. You can avoid all of these problems by taking a few extra seconds to implement it like I've shown above.

Learn it, live it, love it. Once you start getting into the habit of doing things like this, there's no turning back and you can rest easy knowing that you've future proofed your application.

Liveness Rule #1: Cache Everything for Reads

2007-04-27T01:27:00.000-07:00

Caching is really the holy grail of liveness. Consider the difference between hitting your database for every request vs pulling that data directly out of memory. Here is a small comparison of performance related properties with pros in green and cons in red.

Memory:

Near instantaneous access to data
Concurrent contention is virtually nil
Limited amount of memory

Database Hit:

Disk seek speed
Concurrent contention as threads wait for disk access
IO speed
The database query speed
Connection setup/teardown
Marshalling and unmarshalling
Large amount of disk space available

These are all pretty obvious points, but it is just to highlight how much more stuff is done when you don't cache. These things are really taxing on the entire system, much more so than pulling stuff from memory which also means your application is far more scalable.

Our experience:
Traffic was increasing in large amounts and the response times were getting worse and worse. Not to the point where it was the end of the world, but bad enough that we had to do something about it before it got even worse (over a second is bad). Now that we cache almost everything, response times are down to a few milliseconds, right up there with Google search responses.

Caching is actually pretty simple to implement. No matter what solution you use, the interface is always something like java.util.Map where you put, get, and remove elements.

Liveness In Web Applications

2007-04-26T23:53:00.000-07:00

Liveness: A concurrent application's ability to execute in a timely manner is
known as its liveness.

In other words, liveness is making your application respond extremely fast to your users requests. Having had to deal with this over the past few months on rel8r, I thought I'd share my experiences.

Renewing SSL Certificates on Tomcat

2007-01-02T17:59:00.000-08:00

So I had to renew a couple of SSL certificates that are used by sites running standalone Tomcat, here is what I had to do:

Generate a new CSR request. This is easier than when first starting out since you've already created your keypair from the last time you bought your certificate:
keytool -certreq -keyalg RSA -alias tomcat -file certreq.csr -keystore .keystore
Now open certreq.csr in a text editor and copy and paste the contents into your SSL issuers website form to finish the process of getting your new certificate.
Now you should get an email to verify that you are the one who submitted the request and a link to Approve it. This email will go to the email address on your whois record for your domain name to verify that you own the domain.
After you Approve the SSL request, you can download your new certificate along with the issuers intermediary certificate.
You must first install the issuers Intermediate Certificate:
keytool -import -alias intermed -keystore .keystore -trustcacerts -file sf_issuing.crt
Then import your fresh new certificate:
keytool -import -alias tomcat -keystore .keystore -trustcacerts -file ariel1.spaceprogram.com.crt
And finally, restart Tomcat and double check by surfing to your page and checking out your certificate by clicking on the padlock icon on your web browser. Make sure the expiry date is correct.

That's about it. Nice and easy.

Poor Man's Oracle Client

2006-09-18T14:39:00.000-07:00

I generally despise having to install Oracle software because it's so big and bloated, it's scary. Most downloads won't fit on a typical CD, even their Oracle Express edition is bigger than Microsoft Windows! I don't know what Express means to them, but "huge" isn't the description I would think of.

But there is light at the end of the tunnel, Oracle now has Instant Client which is only 35MB! Wow! And all you have to do is unzip it. It includes OCI, OCCI, and JDBC drivers, but if you want to make it useful out of the box, you should get the SQL Plus package for Instant Client (722KB!) and unzip it into the same directory. Put your tnsnames.ora into the directory as well if required. Then run sqlplus.exe as you normally would and voila, you're done.

Download Oracle Instant Client
Download SQL*Plus for Instant Client (from same download page)
Unzip Instant Client into a directory

eg: c:\oracle_ic

Unzip SQL*Plus into the same directory
Put tnsnames.ora into the same directory (OPTIONAL)
Run: sqlplus username/password@SID

Get SQL Explorer for a free JDBC based client if you want to connect with a visual client.

Db4oUtil (aka. The Easiest Way to Get Started With Db4o)

2006-07-04T15:35:00.000-07:00

Want the quickest and easiest way to get started with db4o? Use the following Db4oUtil class. This class is very similar to the HibernateUtil class that you may be familiar with if you're a Hibernate user. It uses a thread local object so it's thread safe, makes multi-tier usage a no brainer and it's just plain simple to use.

The simple step-by-step for this tutorial:

Copy the Db4oUtil.java code below into a new class
Use the Db4oUtil methods in your code, samples below

That is it... seriously. You can optionally tweak the fields in Db4oUtil to your liking, like you'll want to change the YAPFILENAME field for each application you will use this with.

Db4oUtil.java

public class Db4oUtil {
    // EDIT THESE SETTINGS
    private static final String YAPFILENAME = "test.db4o.yap";
    private static final int PORT = 0;
    /*
     If you want the server to be networked, change the port number above and uncomment the USER, PASSWORD lines below/
     Then in getObjectServer, uncomment the objectServer.grantAccess line
     */
    //private static final String USER = "username";
    //private static final String PASSWORD = "password";
    
    private static ObjectServer objectServer;

    private static final ThreadLocal dbThreadLocal = new ThreadLocal();

    public static ObjectContainer getObjectContainer() {
        ObjectContainer oc = (ObjectContainer) dbThreadLocal.get();
        if (oc == null || oc.ext().isClosed()) {
            oc = getObjectServer().openClient();
            dbThreadLocal.set(oc);
        }
        return oc;
    }

    public static void closeObjectContainer() {
        ObjectContainer oc = (ObjectContainer) dbThreadLocal.get();
        dbThreadLocal.set(null);
        if (oc != null) oc.close();
    }

    public synchronized static ObjectServer getObjectServer() {
        if (objectServer == null) {
            objectServer = getObjectServerForFilename(YAPFILENAME, PORT);
            // and give access
            //objectServer.grantAccess(USER, PASSWORD);
        }
        return objectServer;
    }


    public static void shutdown() {
        if (objectServer != null) {
            objectServer.close();
        }
    }

    public static ObjectServer getObjectServerForFilename(String yapfilename, int port) {
        File parentDir = getDbDirectory();
        File dbfile = new File(parentDir, yapfilename);

        // for replication //////////////////////////
        Db4o.configure().generateUUIDs(Integer.MAX_VALUE);
        Db4o.configure().generateVersionNumbers(Integer.MAX_VALUE);

        // other options
        Db4o.configure().exceptionsOnNotStorable(true);
        Db4o.configure().objectClass("java.math.BigDecimal").translate(new com.db4o.config.TSerializable());

        // now open server
        ObjectServer objectServer = Db4o.openServer(dbfile.getPath(), port);

        return objectServer;
    }

    private static File getDbDirectory() {
        // will store data files in {user.home}/db4o/data directory
        String dbfile = System.getProperty("user.home") + "/db4o/data";
        File f = new File(dbfile);
        if (!f.exists()) {
            f.mkdirs();
        }
        return f;
    }
}

You can edit the fields at the top to modify your database filename, etc.

Using Db4oUtil

This is a simple code sample of usage:

// Get ObjectContainer - you can repetitively call this function in different methods without the need to pass the ObjectContainer around
ObjectContainer oc = Db4oUtil.getObjectContainer();

// YOUR CODE HERE - use the ObjectContainer and do all your stuff here

// Close ObjectContainer when done 
Db4oUtil.closeDb();

// Close object server when completely exiting application
Db4oUtil.shutdown();

That's all there is to it!

Say Goodbye to your Relational Database

2006-05-18T13:53:00.000-07:00

db40 Has Entered the Building

I have been using db4o a LOT lately, and I don't expect I shall return. I can say without question that it has increased my productivity by at least... 1 million times. Well probably not that much, but a heck of a lot. It definitely takes a bit of getting use to if you come from a relational database world, but consider never having to map your objects to a relational database. And how about never having to make the database at all! You just worry about your object model and db4o will take care of the persistence (long term storage).

Check out these examples to see just how easy it is.

Starting the Database Server

// openServer() takes the filename where you want your database stored and a port
ObjectServer server = Db4o.openServer("somefilename.yap", 1234); // Now you can give access to particular logins server.grantAccess("username","password");

There are other ways of doing this: you can run it in non-server mode, but I don't recommend that since you'll probably just end up changing it down the road if your application is multi-threaded (and what application isn't these days?); you can also use an embedded server by using port zero [

ObjectServer server = Db4o.openServer("somefilename.yap", 0);] and getting connections from the ObjectServer directly [ObjectContainer db = server.openClient();] which isn't too bad because it'll be faster, but it can't be networked.


When your program is ready to shutdown, you'll want to call server.close() to shut it down.

Getting a Connection

ObjectContainer db = Db4o.openClient("localhost",PORT,USER,PASSWORD);
try{
     // with the client, we can do all the fun stuff in the examples below 
    // ALL EXAMPLES ARE ASSUMED TO BE PLUGGED IN HERE
} finally {
     db.close();
}
Saving and Updating an Object
Doesn't get much easier than this.
db.set(myObject);
 Query
This is equivalent to your SQL Select.  For this example, lets say we saved a couple of objects that looked like this:
Contact contact = new Contact();
contact.setName("Travis");
db.set(contact);
contact = new Contact();
contact.setName("Jimbo");
db.set(contact);
Now to query for the contact named Travis, you do this:
Query q = db.query();
q.constrain(Contact.class);
q.descend("name").constrain("Travis");
ObjectSet result = q.execute();
while (result.hasNext()) {
     Contact c = (Contact) result.next();
     System.out.println("Contact: " + c.getName());
}
The example above is a SODA   query which as you can see, uses strings and can be built dynamically. db4o does have another method of querying called Native Queries. db4objects recommends Native Queries and it's for good reason. Native queries don't reference fields by strings, they're compile time checked, are object oriented and, most importantly, they are refactorable. 

List <Contact> contacts = db.query(new Predicate<Contact>() {
     public boolean match(Contact contact) {
         return contact.getName().equals("Travis");
      }
});
for (Contact contact: contacts) {
     System.out.println("Contact: " + c.getName());}
 
Deleting an Object
db.delete(myObject);
Gotchas
Now not everything is greener on the other side, there are still many little gotchas that may bite you, for instance:
No ID support like Hibernate / EJB so you can't let db4o rebind objects based on an id field in your objects
This is extremely apparent in a web based application where you have to be sure to reload your objects in every request, so they can be in your new db4o session (assuming session per request pattern).  I have figured out ways to work around this though to make it automated and I'll try to share these with you in a later post.
Weak management GUI
If you're used to working with nice tools like the tools provided by major database vendors, the db4o Object Manager doesn't really compare. But hopefully with time and with more adopters, the Object Manager will get better and we'll see third party tools. 
No joins
This isn't bad if your object model is not complex and you can have all your objects connected to each other in your object graph, but you have to be careful when your data set gets large and your object model is a complex graph with bidirectional relationships because you may end up loading your entire database! Or at least a good chunk of data that you may not always need. And this is a *huge* performance killer.
Now the obvious answer would be to break apart your object model which is easy to do in a relational database because you have id's and foreign keys. In db4o, if you break apart your model, there is no built in way to reference your disconnected objects. 
It is possible to workaround this though, by using keys and ID's as fields in your objects and querying for the related objects when you need them.

Conclusion
In any case, Carl Rosenberger and the db4objects   folks have done a great job with this open source object database and they are definitely the leaders in the space.  The simplicity, performance, and productivity gains outweight and disadvantages by a long shot. I urge you to try it.  

Also, I plan on sharing more of my experiences with db4o in the near future, along with code such as how to easily use it in a web application, how to partition, and how to scale.



Tags: db4o | object databases | tutorial



Installing Java 5 JDK and Tomcat on Ubuntu (using VMWare)
2006-05-18T09:31:00.000-07:00
This document will walk you through installing Apache Tomcat on Ubuntu Linux using VMware for virtualization. Most of the steps apply even if you're not using VMware.
Download and install VMWare on your server
Download and install VMWare client on your workstation (if it is different than your server)
Option 1: Prepackaged Ubuntu VM (Please note, this will NOT work on a headless server)
Download Ubuntu VM from www.vmware.com
Open Ubuntu VM (user/pass is ubuntu/ubuntu) in VMWare client
Option 2: New fresh install of Ubuntu (you have to do this if you have a headless server)
Download Ubuntu ISO from www.ubuntu.com
Create new virtual machine in vmware, choose Linux -> Ubuntu
Mount the cd drive for the virtual machine to your downloaded iso in the virtual machine settings
Start virtual machine, this will ask if you want to install, so perform the full install (this is just a regular Ubuntu install)
Now you should be in Ubuntu linux
modify /etc/apt/sources.list (ex: run: sudo nano -w /etc/apt/sources.list)
- Change the first section lines deb http://archive.ubuntu.com/ubuntu breezy main restricted
to deb http://archive.ubuntu.com/ubuntu breezy main restricted universe multiverse
You can also add universe multiverse to deb-src and do the same to the breezy-update lines too.
(breezy will be dapper in 6.X versions)
run: sudo apt-get update
Install JDK
Option 1:
run: sudo apt-get install sun-java5-jdk
Option 2: (if option 1 doesn't work)
download jdk 1.5 from sun, the Self extracting linux version, .bin extension (NOT rpm)
run: sudo apt-get java-package
run: fakeroot jdk***.bin
run: sudo update-alternatives --config java
- select the j2sdk1.5-sun option
run: java -version
- just to make sure it's the new version
add:
export JAVA_HOME=/usr/lib/jvm/java-1.5.0-sun-1.5.0.06
if you used Option 1 for installing JDK above or:
export JAVA_HOME=/usr/lib/j2sdk1.5-sun
if option 2 was used
to /home/ubuntu/.bashrc
Open a new console window to continue with the rest of the steps
Download tomcat 5.5 from tomcat.apache.org
extract tomcat (to a directory under your HOME directory is a good idea - /home/ubuntu/java/apache-tomcatXXX)
go to tomcat directory/bin
run: ./startup.sh
surf to http://yournewserversip:8080/
You should see the Tomcat welcome page
If you want to run Tomcat as a service: http://tomcat.apache.org/tomcat-5.5-doc/setup.html

Other tips:
- If you want to use 7-zip which is nice for all compression/decompression tasks, run: "sudo apt-get install p7zip", then to extract anything, you just run "7za x myfile" (.zip or .tar or .tar.gz or .7z, etc)
- If you want an ssh server (sshd) on your new ubuntu box, run: "sudo apt-get install openssh-server". Apparently Ubuntu does not come with an ssh server installed out of the box.
- if you use nano, while you're editing .bashrc above, it's a good idea to add an alias for nano with the -w option like: alias nano='nano -w'
- VMware kicks a**, so be sure to try it (note: I have no affiliation with VMware, I just like good products)

If you have any issues with the steps above or want to add anything, please post your comment below.

Good night and good luck...


Normalize Schnormalize (aka Real-Time Data Warehousing)
2006-05-05T12:32:00.000-07:00
Some applications need real-time reporting, ecommStats Web Analytics is one such app.  ecommStats customers want to see what is happening on their website and they want to see it now.

A little background might be in order: In general, with enterprise software, you move and transform data from your Transactional Database (the one that is involved in the day to day work and storing every little piece of data) to your Data Warehouse so that you can generate reports faster and easier, which basically boils down to Business Intelligence.  And this is generally a heavy process so it will run once per day, once per week, or whatever fits into the business needs and the gap must be longer than it takes to actually run the process (if the process takes 24 hours to run, it's probably not a good idea to run it every day).

But why wait until the night time (the night time is the right time) or the weekend to move all that data into a separate database when you can do simple preemptive operations on your database to give your users real-time data?

 Normalization is great and all, but a hybrid approach can work wonders.  The data is still normalized, but we have extra tables and extra columns that act as our data warehouse.

Lets take the Search Phrase report for instance, it is a time based report which shows you how many people have searched on a particular search phrase or keyword.  In a normalized system, you would have your search phrase in one table and the request that came in in another table (this is seriously simplified to make a point):


SearchPhrase
id
phrase


SearchRequest
id
phraseId
date
user



Now you could sum up all the requests that used the phrase everytime someone wants the Search Phrase report, but do you want to make your users wait for this report?  There could be thousands of words and thousands of requests for each word which quickly puts you into the millions of rows!   The user might as well go have a coffee while they're waiting.  And do you want your poor server to have to work so hard for this little report?

But by adding a single column, we can make the report almost instantaneous:

SearchPhrase
id
phrase
requestCount
lastRequestDate



Now when we're showing that report, we don't even have to look at the SearchRequest table, so what used have to go through potentially millions of rows, now is just a few thousands rows (or however many search phrases you are looking at).  Obviously this example won't give you the time based reporting, but using an extra time bucket table, with similar techniques can get the results you want.

And lets say you also wanted to show when the last time some searched for a particular phrase. You could look in the SearchRequest table, sort by date descending and then grab the first one, but this is time, disk and processor intense if the request table is large. So try simply adding a date column to the SearchPhrase table like above and update it when the request is made. Don't be afraid to duplicate data when the performance benefits can be substantial.

Now how much harder is this to implement in your code?  It's actually very simple in most cases, usually just requiring an extra update.  In the example above, when saveSearchRequest(SearchPhrase phrase) is called in the code, it will save the SearchRequest, and do a simple update on the requestCount column:
UPDATE SearchPhrase SET requestCount = requestCount + 1, lastRequestDate = sysdate where id = PHRASE_ID;
 

This one time update is insignificant when compared to the many times that queries will be run against your tables.

In conclusion, using these hybrid techniques for database design can reduce wait time significantly and you can report in near real-time, rather than making your users wait days for their data. The new columns and/or tables may take up more space, but space is cheap and peformance is not.  It truly is a small price to pay when compared to the huge price of doing this repeatedly in a purely normalized database or a non-real-time huge bulk process to move the data into a data warehouse.

Tags: normalizing | data warehouse | analytics | real-time | databases