The processor on the Raspberry Pi isn't all that fast, so why not offload it? I have a HP ML350 that is mostly idling, except when I need to use PostgreSQL, so I thought to set it up as a desktop server for my Pi.
I set up a RDP server since it works well with Linux and also Windows. First thing's first, install xrdp and all other dependencies:
apt-get install build-essential libpam0g-dev libssl-dev x11vnc vncviewer fakeroot checkinstall x11vnc xrdp rdesktop
Then, install your desktop of choice. I chose XFCE since it's pretty lightweight
apt-get install xfce4
I've had problems getting it to start X remotely, so I fixed it with the following. Do this in the user's account (/home/pi for example)
touch .Xauthority
chmod 600 .Xauthority
And that's it!. Start up your Raspberry Pi, install rdesktop, and connect to your server:
rdesktop -g 1024x768 -P server_address