With EC2 the way one configures a server is by creating a machine image (AMI). But there has to be a better way! The first time I created an AMI almost a year ago it was a lot of fun. Hitting "launch" and seeing a new instance pop up was really cool. But a few weeks later reality started to set in, as I was coming up with little tweaks almost on a daily basis - anything from changing an SSHD option to installing additional software required reimaging, and that began to be painful. I pretty quickly settled into the following procedure:
- Make the change on a current running instance to make sure it works.
- Launch the original image so I have a clean instance to modify.
- Have a cup of coffee while the instance starts up.
- Make the change again, test again.
- Bundle the instance, have lunch waiting for the bundling to complete.
- Launch the new image.
- Have another cup of coffee waiting for the instance to launch.
- Test again that everything is good.
- Launch additional instances for all the servers that use the modified image.
- Move data from the old instances to the new ones and switch service over.
Well, in truth, most of the time after step 7 I discovered some problem forcing me to circle back to step 2. Usually, by the time I was done, the better part of a day had gone by and I didn't exactly feel productive. Pretty quickly I stopped revving the images and instead kept a set of notes of the fixes I would apply manually to a fresh instance before it would be ready, and I kept pushing off the moment of rolling those fixes into a fresh image. You can imagine that this only meant that the config of my servers quickly started to drift apart as I applied various fixes to different parts of the fleet. Not a good way to run a reliable service!
Another problem I faced is that a machine image is static and I wanted a more dynamic instance configuration system. Images are static and in the end each image really represents one single server. Yet a lot of the power of EC2 lies in multiplying servers, either to handle load (scaling up) or to provide flexibility for testing, development, new projects, and additional developers. In all these cases I want a more modular system where I have a base image with software that I need pretty much everywhere and that doesn't need frequent updates. On top of that I want to layer software modules - think RPM package plus customization, like Apache configured as reverse proxy for Rails just to take one example. Finally, I want to tie the server into a single- or multi-server deployment so it can talk to the right database server, have the right hostname, download the right data dump from S3, etc.
For the past months we've been developing a server configuration system called RightScripts that supports our own and our customers' needs. We need to be able to define servers out of building blocks and then gang them together into multi-server deployments. When we have a multi-server config setup we need to be able to easily clone it, change some deployment-specific environment variables, and launch a second deployment. Or instead of having just one app server, we want to be able to define an array of app servers and have RightScale launch more as the load goes up (and reap some as the load reduces to save money). All this requires thinking about servers as objects that we multiply and dynamically configure, not as something we freeze and thaw.
Enough theory, how do RightScripts look in practice? We start with a base image, which typically is the CentOS 5 RightImage we made available a few months ago. It contains the stuff that we expect to find on all servers: the various EC2 command-line tools, Perl, Ruby, Java, gcc, crypto stuff, syslog-ng, a specific sshd config, and so on. On top of this base we defined a number of software modules, each one really being a shell script that installs and configures a piece of software for a specific purpose. The script usually needs one or several packages installed and it may need some file attachments. Let's take an example: Our Apache base install starts by installing the CentOS httpd, httpd-devel, and mod_ssl packages (
yum install httpd httpd-devel mod_ssl) and then consists of the following bash script:
service httpd stop if [ -d /mnt/www ]; then echo "Apache Base Files Exist" else ## Move Apache mv /var/www /mnt ln -nsf /mnt/www /var/www fi ## Move Apache Logs rm -fr /var/log/httpd mkdir -p /mnt/log/httpd ln -s /mnt/log/httpd /var/log/httpd ## Set Admin Email perl -p -i -e "s/root@localhost/$ADMIN_EMAIL/" /etc/httpd/conf/httpd.conf ##Change to threaded worked perl -p -i -e 's/#HTTPD=\/usr\/sbin\/httpd.worker/HTTPD=\/usr\/sbin\/httpd.worker/' /etc/sysconfig/httpd ##modify Server Signature perl -p -i -e 's/ServerSignature On/ServerSignature Off/' /etc/httpd/conf/httpd.conf perl -p -i -e 's/#ExtendedStatus On/ExtendedStatus On/' /etc/httpd/conf/httpd.conf ##disable php for now mv /etc/httpd/conf.d/php.conf /etc/httpd/conf.d/php.disabled
As you can see, this is really simple, and it should be! With the above definition entered as a boot script into RightScale we can associate it with any server where it's needed and RightScale will take care of installing the packages and executing the script at boot time. The nice thing is that the script doesn't bake the Apache package version into an image; we get the latest one, or we could select a specific one in the script. Also, it's easy to see what is changed from the distribution's installation and tweak it incrementally to different needs, so when we want a somewhat different setup we're not diffing installations, we're just looking at the script for what was changed. Finally, since the time we configured a number of different servers using these scripts, we've had to update the base CentOS image, and it was a breeze to restart all the servers with the new base image plus the specific set of scripts needed for each server. Before, we would have had to apply the same base CentOS change to each image of each server.
I'll leave two additional levels of functionality for future posts. These RightScripts don't have to be executed at boot time, they can also be executed later as "pre-configured" actions that can be run when something changes. For example, to add a new app server instance to a load balancer instance config. Also, when servers are tied into multi-server deployments, the scripts can refer to variables that provide environment-specific values, such as the IP address of the database server to connect to, or the number of Apache processes to run.
If you're interested in learning more about RightScripts, please contact us at firstname.lastname@example.org. RightScripts are available with RightScale's free accounts, but more advanced features are reserved for the premium accounts.