If you are not getting a package on Ubuntu, please check to see if your /etc/apt/sources.list file is updated. This is common with most VPS providers

On VPS, especially with challenging small memory limit (e.g. 64MB or 128MB) lighttpd is the best option instead of Apache. In this article I will show you how to do that. On a fresh box (which has no apache installed) type this to get lighttpd webserver installed

apt-get install lighttpd

Now install PHP5 CGI package and PHP5 command line package

apt-get install php5-cgi php5-cli

Edit the following file

nano /etc/php5/cgi/php.ini

and uncomment this line

cgi.fix_pathinfo = 1

Now enable fastcgi module

lighty-enable-mod fastcgi

Restart Lighttpd

/etc/init.d/lighttpd restart

You get the welcome page

Lets install MySQL 5.1 server and client

apt-get install mysql-server mysql-client

and optionally phpMyAdmin

apt-get install phpmyadmin

Now create a test page to see if phpinfo is all Ok

nano /var/www/info.php

Type in the following

<?php
phpinfo();