Tuesday, June 30, 2009

Deploying a Grails app on EC2 from scratch.

Hi All

Ok so i have found some time write up how to deploy a Grails app onto EC2. This is a step by step guide to setting up Apache, Tomcat, Mysql, Java on an ubuntu Ec2 Box

For those who do not know. EC2 is Amazon's (that's right the one famous for books) hosting service of its webservice suite. It allows you to run virtual server images on it amazingly scaleable infrastructure. Plus you only pay for what you use. If your server is up for 4 hours you pay for 4 hours. $0.10 per hour (about). You can build your own server as i will show you in the steps below and manange it on their infrastructure.


These are the steps i have taken and they work for me. I am sure there are hundreds of ways to skin a cat (so the cat skinners say) but this is my way. Take it or leave it :)

Pre-Requisites

1) Get Firefox i am sure you have it already if you don't get it you will need it. As of writing this i am using 3.0.11 download here

2) Install the EC2 Firefox plugin (Elasticfox) download here

3) Install the S3 firefox plugin download here


4) Open an amazon account. This is a five min job requiring a valid credit card. Signup here
Once you have an account you then need to open a EC2 account and an S3 account.


5) Then next step is for you get familiar with the tools. You can do this by following the getting started guides in this ebook.

Gettings started


IMPORTANT !!!

You need to read the getting started guide and get familiar with things before you read the rest of this blog post. Otherwise it will not make sense. I am assuming from here onwards you have read an tried out the service.


Setting up a server to run a Grails app.

By now you should have terminated any test instances you had running on EC2 and be ready start a fresh.


1) Start with a baseline Linux image

Ok i have decided to host my application on Linux basically because there was no descion to be made. I would only ever host a serious website on Linux and never windows.

The base linux version i used is Ubuntu 8.10 (intrepid) i am also using a 32bit and not 64 bit. You can find the AMI codes here

https://help.ubuntu.com/community/EC2StartersGuide

I have chosen to go with AMI ami-5059be39 because it is based in the US and is 32Bit. If you want to run your servers in Europe then go with this ami-80c0e8f4

Ok now you should have a running instance of the ubuntu basic server. Connect to the server using an SSH terminal

Create a directory called downloads in side your home directory /home/ubuntu and cd into it.

2) Next you want to install Java

You should be in the downloads directory you created above

Download java 1.6. Right that now happens to be JDK 6 update 14. To do this i goto the java download website .

Choose JDK 6 update 14
Choose linux
Then copy the download link for the NON rpm version the second link

Now on your EC2 machine run the following command

wget <THE JDK DOWNLOAD URL>


This will now download the JDK install file onto the box

NOTE you might want to rename the file:

mv <NAME OF THE FILE>  jdk-6u14-linux-i586.bin


Next copy the file to /usr/local

sudo cp jdk-6u14-linux-i586.bin /usr/local



Change the permissions


sudo chmod +x jdk-6u14-linux-i586.bin



Then run the install file


sudo ./jdk-6u14-linux-i586.bin



This should unpack the java file to this directory /usr/local/jdk1.6.0_14

Next create a symbolic link to the directory called java


sudo ln -s /usr/local/jdk1.6.0_14 /usr/local/java


That is java installed.

3) Next install MYSQL

Download mysql via the same way you downloaded java above using wget you get the download link here

Choose the Linux (x86) download

Copy the mysql tar file into /usr/local


sudo cp mysql-5.1.35-linux-i686-glibc23.tar.gz /usr/local


Delete any existing mysql directories. (this maybe optional)


sudo rm -rf /etc/mysql



Next follow these commands to install it. IMPORTANT: Don't follow any of the step that are listed by the mysql_install_db script they will be covered later.


sudo groupadd mysql
sudo useradd -g mysql mysql
sudo cp mysql-5.1.35-linux-i686-glibc23.tar.gz /usr/local
sudo cd /usr/local
sudo gunzip mysql-5.1.35-linux-i686-glibc23.tar.gz
sudo tar -xvf mysql-5.1.35-linux-i686-glibc23.tar
sudo mv /usr/local/mysql-5.1.35-linux-i686-glibc23 /usr/local/mysql
sudo cd mysql
sudo mkdir logs
sudo chown -R mysql .
sudo chgrp -R mysql .
sudo scripts/mysql_install_db --user=mysql
sudo chown -R root .
sudo chown -R mysql data
sudo bin/mysqld_safe --user=mysql &


Right now you should have running mysql instance

Copy the standard startup script into the linux startup area


sudo cp /usr/local/mysql/support_files/mysql.server /etc/init.d/mysql


Setup mysql to run at all levels


sudo update-rc.d mysql defaults


Set a root password


sudo /usr/local/mysql/bin/mysqladmin -u root password 'new-password'
sudo /usr/local/mysql/bin/mysqladmin -u root -h 'localhost' password 'new-password'


Ok now you have a running mysql instance

4) Next you want to move the Mysql installation on to an amazon EBS.

You should already know about an EBS from reading the EC2 getting start guide above.

Create a new EBS using Elastic fox and attach it to your EC2 instance. You will be asked to enter a device when you are attaching the EBS to your running EC2 instance. Enter the following:


/dev/sdh


Once you have done that you want to mount and format that diskspace. I am simply following the instructions found here but have modified it bit as my paths are different

Install XFS. This is a secure file system.


sudo apt-get install -y xfsprogs


Setup the mount and fomat the disk


sudo modprobe xfs
sudo mkfs.xfs /dev/sdh

echo "/dev/sdh /vol xfs noatime 0 0" | sudo tee -a /etc/fstab
sudo mkdir /vol
sudo mount /vol


You should now have a new filesystem under /vol

Stop mysql if running


sudo /etc/init.d/mysql stop


Move all the mysql files


sudo mkdir -p /vol/usr/local/mysql

sudo mv /usr/local/mysql /vol/usr/local


Remount the mysql directory to the volume


sudo mkdir /usr/local/mysql

echo "/vol/usr/local/mysql /usr/local/mysql none bind" | sudo tee -a /etc/fstab
sudo mount /usr/local/mysql



Restart mysql

Stop mysql if running


sudo /etc/init.d/mysql start


You now have mysql running from an EBS


Setup mysql database user and backup script


/usr/local/mysql/bin/mysql -p -u root


create database


CREATE DATABASE demodb CHARACTER SET utf8 COLLATE utf8_general_ci;


create app user and backup user and give access:


CREATE USER 'demouser'@'localhost' IDENTIFIED BY 'demouser123';
GRANT ALL PRIVILEGES ON demodb.* TO 'demouser'@'localhost' WITH GRANT OPTION;

CREATE USER 'backup'@'localhost' IDENTIFIED BY 'backup123';
GRANT LOCK TABLES, SELECT ON demodb.* TO 'backup'@'localhost';


Setup db backup:

Download the mysql backup script:


http://sourceforge.net/projects/automysqlbackup/


For putting files in s3 use this


http://s3tools.org/s3cmd

apt-get install s3cmd






6) Install Tomcat

This is easy to install
Download tomcat 6 into your /home/ubuntu/downloads using wget
Copy the jar file it to /usr/local

Use the java jar command to unpack tomcat zip


sudo /usr/local/java/bin/jar -xvf /usr/local/tomcatXXX.zip



Create a symbolic link to Tomcat


sudo ln -s /usr/local/tomcatXXXX /usr/local/tomcat



Next create tomcat user to run tomcat


sudo groupadd tomcat
sudo useradd -m -s /bin/bash -g tomcat tomcat



Change to tomcat user and edit the .profile to add java in the PATH


sudo su - tomcat
vi /home/tomcat/.profile


Edit the file using vi to contain


export JAVA_HOME=/usr/local/java
PATH=$JAVA_HOME/bin:$PATH


configure the server.xml to use an AJP connector to work with apache.

save your old /usr/local/tomcat/conf/server.xml to server.xml.orig


cp /usr/local/tomcat/conf/server.xml /usr/local/tomcat/conf/server.xml.orig


edit a new server.xml


vi /usr/local/tomcat/conf/server.xml


copy this config in there:

<?xml version='1.0' encoding='utf-8'?>
<Server port="8005" shutdown="SHUTDOWN">

<Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
<Listener className="org.apache.catalina.core.JasperListener" />
<Listener className="org.apache.catalina.mbeans.ServerLifecycleListener" />
<Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />

<Service name="Catalina">

<!--
<Connector port="8080" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8443" />
-->

<Connector port="8009"
protocol="AJP/1.3"
redirectPort="8443"
connectionTimeout="300000"
maxThreads="200"
backlog="50"
enableLookups="true"
emptySessionPath="true"

/>

<Engine name="Catalina" defaultHost="localhost">

<Host name="localhost" appBase="webapps"
unpackWARs="true" autoDeploy="true"
xmlValidation="false" xmlNamespaceAware="false">

</Host>
</Engine>
</Service>
</Server>



That is tomcat setup. You can run it if you like to see it working.

Now just run tomcat


sudo su - tomcat
/usr/local/tomcat/bin/startup.sh



6) Install Apache

This is easy


sudo apt-get install apache2


Enable AJP mod


sudo a2enmod proxy_ajp


Create a new site to configure AJP with tomcat. You can change the to whatever you want.



sudo vi /etc/apache2/sites-available/webapp


add the following config to link apache to tomcat


<ifmodule>
ProxyRequests On
ProxyVia On

<location>
Order allow,deny
Allow from all
ProxyPass ajp://localhost:8009/
ProxyPassReverse ajp://localhost:8009/
</location>

</ifmodule>

Enable your site


sudo a2ensite webapp


disable the default apache config


sudo a2dissite default


start or restart apache


sudo /etc/init.d/apache2 restart



That is basic apache installed

To secure apache with basic authenication:



sudo mkdir /etc/apache2/passwd
cd /etc/apache2/passwd
sudo htpasswd -c site-access



add these lines to the webapp config to secure:

<Location />
Order allow,deny
Allow from all

AuthType Basic
AuthName "Restricted area"
AuthUserFile /etc/apache2/passwd/site-access
Require valid-user

ProxyPass ajp://localhost:8009/
ProxyPassReverse ajp://localhost:8009/
</Location>





To enable SSL on the apache instance:

Enable ssl mod


sudo a2enmod ssl


generate your .key and .crt files. The company who you bought your SSL certs from should tell you how to do this.

make a new director to put the .key and .crt file in:


sudo mkdir /etc/apache2/ssl


copy the .key and .crt files into that directory. eg:


/etc/apache2/ssl/www.mydomain.com.crt
/etc/apache2/ssl/www.mydomain.com.key



Create a new site to configure ssl.


sudo vi /etc/apache2/sites-available/ssl


copy this config

<VirtualHost *:443>
DocumentRoot /var/www/

<IfModule mod_proxy_ajp.c>

ProxyRequests On
ProxyVia On

<Location />
Order allow,deny
Allow from all
#AuthType Basic
#AuthName "Restricted area"
#AuthUserFile /etc/apache2/passwd/site-access
#Require valid-user
ProxyPass ajp://localhost:8009/
ProxyPassReverse ajp://localhost:8009/
</Location>


</IfModule>

<IfModule mod_ssl.c>

SSLEngine On
SSLCertificateFile /etc/apache2/ssl/www.mydomain.com.crt
SSLCertificateKeyFile /etc/apache2/ssl/www.mydomain.com.key

</IfModule>
</VirtualHost>


Just change the paths at the bottom to point to your .crt and .key files.

Ok few that was long ish for me to write...

By now you should have Tomcat running with Apache and Mysql.



UPDATE:

Extra steps i take for Grails:

Going live with grails

Hope this helps someone....

16 comments:

Roshan Shrestha said...

How did you solve the automatic attachment of EBS volumes when you shutdown and restart your instance? EBS volumes can only be attached to a running instance, and therefore when you restart your instance, MySQL won't run as the volume isn't mounted.

I have found some scripts that do it, but they all require your private credentials be stored in the AMI. The solution for me has been to start the instance, and once it is running, "reboot" it.

Peter Delahunty said...

Hi

I followed the steps in this document:

http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1663

It seems that when the instance start is is already mounted to the ebs. Possibly because one of the commands in the doc. But mysql did not start up. Had to do that manually.

Need to look into it all a bit more.

Anonymous said...

Hi,
Thanks for the article. Definitely helps me. Is there an up front one time cost to set up EC2? I wanted to get an EC2 instance but was told by a friend that there is one time fee of $300. Would appreciate if you could let me know.

blog said...

@Anonymous - there's no upfront fee for basic on-demand use of EC2. If you expect to be using the service heavily, you can save some money in the long term by reserving instances. This requires paying some money upfront in return for getting a substantial discount on the hourly rates. See the EC2 pricing page for details.

--Matt

blog said...

Hi Peter,

Have you considered using Cloud Tools? I haven't yet gotten around to trying it myself, but it appears to provide the same (or very similar) software stack as you describe in this post.

--Matt

Peter Delahunty said...

Hi Matt

Thanks for answering the last question. Always (Fear, Uncertainty and Doubt)FUD with new tech.

I did look at Cloud Tools. However they don't seem to support Tomcat 6 or latest version of Mysql. I guess I also wanted to setup my own server so that i knew exactly what was on it. There are many ways to setup things and everyone does things differently. ha ha guess that is why i love grails :)

blog said...

No problem. I was excited to see the original blog post announcing reserved instances, mostly because it makes the cost of always-up servers more reasonable. I'm sure not everyone caught all the details.

That's true about Cloud Tools. I'm surprised they're still on Tomcat 5.5 (pretty old at this point). I'll ask Chris Richardson (the creator) if there are plans to update Tomcat and MySQL in the AMI(s). By the way, I understand the idea of wanting to keep control over what's on your box, but at the same time, too much control can lead to too much maintenance.

--Matt

Chris Richardson said...

I saw the comments about CloudTools.org and just wanted to let you know that we plan to release new features and updated AMIs in the very near future for both www.CloudTools.org and for www.CloudFoundry.com. Stay tuned....

blog said...

Thanks Chris. Good! That was what I was expecting. I'm glad you found this discussion even though I failed to send you a note. :)

--Matt

David H. Young said...

Regarding...
<ifmodule>
ProxyRequests On
ProxyVia On
....

I had to use
<ifmodule mod_jserv.c>
since ifmodule requires a condition, like a typical if statement...

Thank you for this excellent resource. This was my only stumble. You cut my chore of setting up my EC2 environment to about an hour.

Francis said...

Thank you so much for this explanation. The only stumble block I encountering is with the mysql installation. At the very last step, I get access denied for 'root'@localhost'. I'm quite puzzled by this ... Any help someone?

sheldon said...

I am deploying JBoss apps on an Apache+JBoss+MySQL instance (I outlined the steps here. I am having a problem with the ajp connection, each time I restart JBoss, I have to restart Apache. Otherwise I got "service temporarily unavailable'. I didn't have to do that before using Apache 2.0.x and mod_jk. Any thoughts?

FitzChivalry said...

I would LOVE to see this rewritten for using an EBS-backed AMI. I'm trying to set up a Grails app on one of the new micro instances, and am only lightly Linux-aware, and therefore can make some educated guesses w/r/t changes that need to be made to these steps, but I was totally flummoxed when I got to the part about formatting and mounting the EBS... the EBS was already mounted, and I couldn't figure out how to modify your steps to accommodate the difference.

Steve Robbins said...

Thanks I followed your guide but I was a bit confused at the Setup db backup part. I don't know if I did it right because I can't seem to find the website backups!

sadowsky said...
This comment has been removed by the author.
Anonymous said...

Looks great! But, is the detail of this how-to still current? It's currently 2.5 years old, and the technologies surely will have changed.