Deploying a Grails app on EC2 from scratch.

Hi All

Ok so i have found some time write up how to deploy a Grails app onto EC2. This is a step by step guide to setting up Apache, Tomcat, Mysql, Java on an ubuntu Ec2 Box

For those who do not know. EC2 is Amazon's (that's right the one famous for books) hosting service of its webservice suite. It allows you to run virtual server images on it amazingly scaleable infrastructure. Plus you only pay for what you use. If your server is up for 4 hours you pay for 4 hours. $0.10 per hour (about). You can build your own server as i will show you in the steps below and manange it on their infrastructure.


These are the steps i have taken and they work for me. I am sure there are hundreds of ways to skin a cat (so the cat skinners say) but this is my way. Take it or leave it :)

Pre-Requisites

1) Get Firefox i am sure you have it already if you don't get it you will need it. As of writing this i am using 3.0.11 download here

2) Install the EC2 Firefox plugin (Elasticfox) download here

3) Install the S3 firefox plugin download here


4) Open an amazon account. This is a five min job requiring a valid credit card. Signup here
Once you have an account you then need to open a EC2 account and an S3 account.


5) Then next step is for you get familiar with the tools. You can do this by following the getting started guides in this ebook.

Gettings started


IMPORTANT !!!

You need to read the getting started guide and get familiar with things before you read the rest of this blog post. Otherwise it will not make sense. I am assuming from here onwards you have read an tried out the service.


Setting up a server to run a Grails app.

By now you should have terminated any test instances you had running on EC2 and be ready start a fresh.


1) Start with a baseline Linux image

Ok i have decided to host my application on Linux basically because there was no descion to be made. I would only ever host a serious website on Linux and never windows.

The base linux version i used is Ubuntu 8.10 (intrepid) i am also using a 32bit and not 64 bit. You can find the AMI codes here

https://help.ubuntu.com/community/EC2StartersGuide

I have chosen to go with AMI ami-5059be39 because it is based in the US and is 32Bit. If you want to run your servers in Europe then go with this ami-80c0e8f4

Ok now you should have a running instance of the ubuntu basic server. Connect to the server using an SSH terminal

Create a directory called downloads in side your home directory /home/ubuntu and cd into it.

2) Next you want to install Java

You should be in the downloads directory you created above

Download java 1.6. Right that now happens to be JDK 6 update 14. To do this i goto the java download website .

Choose JDK 6 update 14
Choose linux
Then copy the download link for the NON rpm version the second link

Now on your EC2 machine run the following command

wget <THE JDK DOWNLOAD URL>

This will now download the JDK install file onto the box

NOTE you might want to rename the file:

mv <NAME OF THE FILE>  jdk-6u14-linux-i586.bin

Next copy the file to /usr/local

sudo cp jdk-6u14-linux-i586.bin /usr/local

Change the permissions

sudo chmod +x jdk-6u14-linux-i586.bin

Then run the install file

sudo ./jdk-6u14-linux-i586.bin

This should unpack the java file to this directory /usr/local/jdk1.6.0_14

Next create a symbolic link to the directory called java

sudo ln -s /usr/local/jdk1.6.0_14 /usr/local/java

That is java installed.

3) Next install MYSQL

Download mysql via the same way you downloaded java above using wget you get the download link here

Choose the Linux (x86) download

Copy the mysql tar file into /usr/local

sudo cp mysql-5.1.35-linux-i686-glibc23.tar.gz /usr/local

Delete any existing mysql directories. (this maybe optional)

sudo rm -rf /etc/mysql

Next follow these commands to install it. IMPORTANT: Don't follow any of the step that are listed by the mysql_install_db script they will be covered later.

sudo groupadd mysql
sudo useradd -g mysql mysql
sudo cp mysql-5.1.35-linux-i686-glibc23.tar.gz /usr/local
sudo cd /usr/local
sudo gunzip mysql-5.1.35-linux-i686-glibc23.tar.gz
sudo tar -xvf mysql-5.1.35-linux-i686-glibc23.tar
sudo mv /usr/local/mysql-5.1.35-linux-i686-glibc23 /usr/local/mysql 
sudo cd mysql 
sudo mkdir logs
sudo chown -R mysql . 
sudo chgrp -R mysql . 
sudo scripts/mysql_install_db --user=mysql 
sudo chown -R root . 
sudo chown -R mysql data 
sudo bin/mysqld_safe --user=mysql &  

Right now you should have running mysql instance

Copy the standard startup script into the linux startup area

sudo cp /usr/local/mysql/support_files/mysql.server /etc/init.d/mysql

Setup mysql to run at all levels

sudo update-rc.d mysql defaults

Set a root password

sudo /usr/local/mysql/bin/mysqladmin -u root password 'new-password'
sudo /usr/local/mysql/bin/mysqladmin -u root -h 'localhost' password 'new-password'
 

Ok now you have a running mysql instance

4) Next you want to move the Mysql installation on to an amazon EBS.

You should already know about an EBS from reading the EC2 getting start guide above.

Create a new EBS using Elastic fox and attach it to your EC2 instance. You will be asked to enter a device when you are attaching the EBS to your running EC2 instance. Enter the following:

/dev/sdh
  

Once you have done that you want to mount and format that diskspace. I am simply following the instructions found here but have modified it bit as my paths are different

Install XFS. This is a secure file system.

sudo apt-get install -y xfsprogs
   

Setup the mount and fomat the disk

sudo modprobe xfs
sudo mkfs.xfs /dev/sdh

echo "/dev/sdh /vol xfs noatime 0 0" | sudo tee -a /etc/fstab
sudo mkdir /vol
sudo mount /vol
    

You should now have a new filesystem under /vol

Stop mysql if running

sudo /etc/init.d/mysql stop
    

Move all the mysql files

sudo mkdir -p /vol/usr/local/mysql

sudo mv /usr/local/mysql /vol/usr/local
     

Remount the mysql directory to the volume

sudo mkdir /usr/local/mysql

echo "/vol/usr/local/mysql /usr/local/mysql     none bind" | sudo tee -a /etc/fstab
sudo mount /usr/local/mysql

      

Restart mysql

Stop mysql if running

sudo /etc/init.d/mysql start
     

You now have mysql running from an EBS


Setup mysql database user and backup script

/usr/local/mysql/bin/mysql -p -u root 

create database

CREATE DATABASE demodb CHARACTER SET utf8 COLLATE utf8_general_ci; 

create app user and backup user and give access:

CREATE USER 'demouser'@'localhost' IDENTIFIED BY 'demouser123';
GRANT ALL PRIVILEGES ON demodb.* TO 'demouser'@'localhost' WITH GRANT OPTION;

CREATE USER 'backup'@'localhost' IDENTIFIED BY 'backup123';
GRANT LOCK TABLES, SELECT ON demodb.* TO 'backup'@'localhost';

Setup db backup:

Download the mysql backup script:

http://sourceforge.net/projects/automysqlbackup/

For putting files in s3 use this

http://s3tools.org/s3cmd

apt-get install s3cmd

6) Install Tomcat

This is easy to install
Download tomcat 6 into your /home/ubuntu/downloads using wget
Copy the jar file it to /usr/local

Use the java jar command to unpack tomcat zip

sudo /usr/local/java/bin/jar -xvf /usr/local/tomcatXXX.zip

Create a symbolic link to Tomcat

sudo ln -s /usr/local/tomcatXXXX /usr/local/tomcat

Next create tomcat user to run tomcat

sudo groupadd tomcat
sudo useradd -m -s /bin/bash -g tomcat tomcat

Change to tomcat user and edit the .profile to add java in the PATH

sudo su - tomcat
vi /home/tomcat/.profile

Edit the file using vi to contain

export JAVA_HOME=/usr/local/java
PATH=$JAVA_HOME/bin:$PATH

configure the server.xml to use an AJP connector to work with apache.

save your old /usr/local/tomcat/conf/server.xml to server.xml.orig

cp /usr/local/tomcat/conf/server.xml /usr/local/tomcat/conf/server.xml.orig

edit a new server.xml

vi /usr/local/tomcat/conf/server.xml 

copy this config in there:

<?xml version='1.0' encoding='utf-8'?>
<Server port="8005" shutdown="SHUTDOWN">

  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
  <Listener className="org.apache.catalina.core.JasperListener" />
  <Listener className="org.apache.catalina.mbeans.ServerLifecycleListener" />
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />

  <Service name="Catalina">

        <!--
    <Connector port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" />
        -->

        <Connector      port="8009"
                                protocol="AJP/1.3"
                                redirectPort="8443"
                                connectionTimeout="300000"
                                maxThreads="200"
                                backlog="50"
                                enableLookups="true"
                                emptySessionPath="true"

        />

        <Engine name="Catalina" defaultHost="localhost">

      <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true"
            xmlValidation="false" xmlNamespaceAware="false">

      </Host>
    </Engine>
  </Service>
</Server>

That is tomcat setup. You can run it if you like to see it working.

Now just run tomcat

sudo su - tomcat
/usr/local/tomcat/bin/startup.sh
      

6) Install Apache

This is easy

sudo apt-get install apache2

Enable AJP mod

sudo a2enmod proxy_ajp

Create a new site to configure AJP with tomcat. You can change the to whatever you want.

sudo vi /etc/apache2/sites-available/webapp

add the following config to link apache to tomcat

<ifmodule>
ProxyRequests On
ProxyVia On

<location>
Order allow,deny
Allow from all
ProxyPass ajp://localhost:8009/
ProxyPassReverse ajp://localhost:8009/
</location>

</ifmodule>

Enable your site

sudo a2ensite webapp
  

disable the default apache config

sudo a2dissite default
  

start or restart apache

sudo /etc/init.d/apache2 restart
   

That is basic apache installed

To secure apache with basic authenication:

sudo mkdir /etc/apache2/passwd
cd /etc/apache2/passwd
sudo htpasswd -c site-access 

add these lines to the webapp config to secure:

<Location />
Order allow,deny
Allow from all

AuthType Basic
AuthName "Restricted area"
AuthUserFile /etc/apache2/passwd/site-access
Require valid-user

ProxyPass ajp://localhost:8009/
ProxyPassReverse ajp://localhost:8009/
</Location>

To enable SSL on the apache instance:

Enable ssl mod

sudo a2enmod ssl

generate your .key and .crt files. The company who you bought your SSL certs from should tell you how to do this.

make a new director to put the .key and .crt file in:

sudo mkdir /etc/apache2/ssl

copy the .key and .crt files into that directory. eg:

/etc/apache2/ssl/www.mydomain.com.crt
/etc/apache2/ssl/www.mydomain.com.key

Create a new site to configure ssl.

sudo vi /etc/apache2/sites-available/ssl

copy this config

<VirtualHost *:443>
DocumentRoot /var/www/

<IfModule mod_proxy_ajp.c>

ProxyRequests On
ProxyVia On

<Location />
Order allow,deny
Allow from all
#AuthType Basic
#AuthName "Restricted area"
#AuthUserFile /etc/apache2/passwd/site-access
#Require valid-user
ProxyPass ajp://localhost:8009/
ProxyPassReverse ajp://localhost:8009/
</Location>


</IfModule>

<IfModule mod_ssl.c>

SSLEngine On
SSLCertificateFile /etc/apache2/ssl/www.mydomain.com.crt
SSLCertificateKeyFile /etc/apache2/ssl/www.mydomain.com.key

</IfModule>
</VirtualHost>

Just change the paths at the bottom to point to your .crt and .key files.

Ok few that was long ish for me to write...

By now you should have Tomcat running with Apache and Mysql.



UPDATE:

Extra steps i take for Grails:

Going live with grails

Hope this helps someone....

Paypal Pro Plugin documentation done

Hi

I have release the Paypal Pro plugin. Basically this is NOT a replacement for the existing Grails paypal plugin. It allows you to directly interface with a Paypal Webpayment Pro account. This is a service provided by Paypal that you have to pay for each month but it allows you to take payments on your website via forms on your website without the customer ever getting redirected to Paypal.

The plugin is all working and i have been using it on www.ebookstamper.com for the last few weeks.

Check it out here:

http://grails.org/plugin/paypal-pro


Peace

Grails Super File Upload plugin complete and uploaded (mind the pun!)

Ho ho ho

I have uploaded and released my latest Grails plugin

Super File Upload

Finally you can easily style your file upload control and have a fully working progress bar with it.

If you want to see it action then head to my new website www.ebookstamper.com

Enjoy :)

Going live with your Grails app part 1 of X

So I thought I would right up some problem/solutions I had with deploying a Grails app into production.

First step is reducing grails WAR file size.

I have depolyed my latest app http://www.ebookstamper.com on Amazon EC2 (i will do a later step by step post for EC2 deployment). After a few round trips i realised that i was getting pi**ed off with the time it was taking to upload my 20-30meg war file to my server. So after a bit of digging around on the web and some trial and error I devised a way to reduce my WAR size. Here are the steps below:

1) Use Tomcat 6.X
I won't bother giving arguments for using Tomcat 6. Let me just say if you are developing a new app then take it from me this is what you want to be using for your app server. I worked on a large project for the last 12 month getting this website live http://www.carphonewarehouse.com/ and we really squeezed out the performace of Tomcat and the sun JDK1.6 with great results.

2) Set up a shared library in tomcat

Make a directory called shared/lib under tomcat home directory

mkdir -p $TOMCAT_HOME/shared/lib

3) Tell tomcat about this lib

Find the file $TOMCAT_HOME/conf/catalina.properties

Change this line

shared.loader=

to

shared.loader=${catalina.base}/shared/lib,${catalina.base}/shared/lib/*.jar

4) Build your Grails war file

Ok first off you should build your grails war file as normal to get all the jars.

grails war

Or if you going to deploy the app as root context "/" on tomcat do this

grails war ROOT.war

The main point of this exercise is to have grails build a complete WAR file including all the jar librarys you need. These jars come from all over. Grails home/dist, Grails home/lib, plugins and your app lib. So instead of you manually going to find all these jar files just have grails do it for you.

So once you have your WAR file you will have all the jars you need for your app inside the war file in WEB-INF/lib.

5) Copy all the jar files to $TOMCAT_HOME/shared/lib

So unpack the war file and copy all the jar files to the $TOMCAT_HOME/shared/lib directory you created earlier. My simple technique for this was.

a) Upload the ROOT.war file to your server and put it in the $TOMCAT_HOME/webapps directory. This way it is one (a big one) file to upload.

b) Start tomcat and have it deploy your application. This will get tomcat to unpack the war file for you.

c) Stop tomat then go into the unpacked war lib directory $TOMCAT_HOME/webapps/ROOT/WEB-INF/lib and move all the jar files in there to $TOMCAT_HOME/shared/lib. You can then delete the contents of webapps.

6) Next build yourself an empty war file

Now this is the magic bit that i found somewhere out on the web. The --nojars arguement.

grails war --nojars ROOT.war

The --nojars argument will build a grails war file without any jars in the WEB-INF/lib directory in the war file. You should see that this will reduce the war file size by rather pleaseable amount.

7) Redeploy your new lightweight ROOT.war on tomcat
You can then redeploy your new war on tomcat and have a new working app

Next time around you only need to build the war without any jar file.

Now this should be obvious but i will point it out anyway. If you add a new plugin, upgrade grails or add a new lib jar to your app then you should repeat all of this from the start to avoid class not found problems.

Enjoy

Love Grails? Give something back to the community

Hi All

Been thinking today how important it to the success of the Grails eco system that people give back as much as they take. Grails is simply a revolutionary full stack web development framework. It personally saves me so much development time that it is simply my only choice for web development now.

Anyway I have been using Grails on and off for about 2-3 years. I think is started dabbling with it from version 0.3 but have seen it grow from strength to strength upto its current release of 1.1.1.

The projects I built in grails were most beta ideas and most never seen light of day. They are in my great ideas hardrive graveyard. Anyway one of them has finally seen the light of day: www.ebookstamper.com

Throughout the development of this site I came accross a few missing features in grails that I would only have found from building a commercial website. So I decide to build these features myself in the form of a plugin. I have to tell you it one the best decicions i made.

Grails makes it so easy to create a new plugin and release it that it seem criminal not to. Anyone can apply for a grails plugin developer account and Graeme will probably grant it in 5 mins from his iphone. Then you can check in your work and hey presto you can call yourself a grails plugin developer.

So what can a plugin be. Well anything really. Have you written some code in your Grails app that you think can be made more general. Well do it and refactor it out it into a plugin.

Here are some good ideas for plugins:

Wrap an existing java library as a grails plugin.
There are so many great java libs out there that do specific cool things. Make them available in grails as a plugin. Some good examples of this already are the amazon s3 plugin or the awesome searchable plugin or even the quartz plugin.

Wrap an existing javascript library
This the same as the java lib example above. There are some many javascript utils out there. If you use one think about making it into a plugin for grails. I did this with the super file upload plugin i wrote. I also see it with the grails ui plugin.

Wrap up a handler for new protocol
Have you had to write your own protocol handler for your grails app release it as a plugin. New xml protocols come out all the time.

Wrap up some cool algorithim you have written
So you have written some code that does something cool. If you think it can be useful to someone else then release it as a plugin.

Create a taglib library
Have you written some cool tag libs that you use again and again. Why not package them up as a plugin. I have done this with my paypal pro plugin. You can use the tag libs to create credit card fields such as country select box, credit card type select box, and expriy date etc


Finally rememeber, this all about Karma. If you give good things wil always happen in some shape or form.

So start today: See what code you can give back to your Grails Communtity :)

Peace

My new Grails site finally goes live www.ebookstamper.com

Hi all

So i finally put my new grails built website live this week.

http://www.ebookstamper.com

The elevator pitch is:

On demand stamping of PDF ebooks with your customers name and email

The long description is:

A web based service providing on demand stamping of PDF ebooks with your customers name and email. Helps protect your intellectual property by providing a deterrent against file sharing.

People who write ebooks and self publish will know what I am talking about.

I have been working on it in my spare time (is there such thing as spare time) for the last few months. Recently I took some quality time out and worked on it full time to get it out the door. The good new is that I have developed 4 grails plugins that have come out of building this website.

I love grails and it has help me so much in getting things out the door as soon as possible so I feel it is only right to give back to the community. So I have released these four plugins:

These are:

Javascript validator plugin
Grails template engine plugin

Super file upload plugin
Paypal Pro plugin

Although the code for the last 2 is checked in i have not found time to write up the docs. But will get to that asap. If you want to see the Super file upload plugin (SFU) in action go to my new website above and click on the demo button. The styled red browse button you see there is using the SFU plugin. When you upload a PDF you get the upload progress bar showing too.

Also I am hosting the whole thing on Amazon EC2 so will write up some posts on running grails apps on EC2. Some ticky bits worth documenting.

Peace...