A few weeks ago, this brilliant idea crossed my mind. Why don't I build a private cloud at home, in my garage, so that I can get the experience of setting up the technology?
Vijay told me I was crazy, so to prove just how easy it would be I did it anyway, and as a result I've had a private cloud running in my garage since early March.
First, Some Background
Before I get started there are a few housekeeping items I'd like to quickly address.
TL;DR - Too Long, Didn't Read
I think it's only fair to tell you that this post is a bit lengthy since I wanted to tell the full story. I also wanted to cover as much of the details of the process as possible. Even so, I wasn't able to deep dive into everything so if you have questions please fire them off in the comments and I'll be happy to keep the conversation going there. With that said, grab a cup of coffee and enjoy the story!
If you're looking for the tl;dr version, the diagram under So Meta... should tell you everything you need to know.
And if you're looking for an in-depth guide to best practices for building private clouds that include reference architectures, check out Brian Adler's white paper on Designing Private and Hybrid Clouds.
The Road Ahead
It may also be helpful to have a quick outline of the topics I'm going to discuss. First, I'll walk you through some of the decision making process. How did I arrive at the decision to build a private cloud? How did I choose the technologies and hardware? Then I'll tell you a bit about the mechanics. How did I put the pieces together? What does my infrastructure look like? And lastly I'll show you some of the fruits of my labor (my favorite part). :)
I had been contemplating an upgrade to my local home file server for quite a while. My research had lead me through all of the NAS devices and other consumer options but at the end of the day I really wanted to build my own system.
That realization led me to looking at some rack mountable servers with hot swappable SATA on eBay for hours on end. When I finally found the right server, and it was enroute to me via the logistics loving courier, I realized that I wasn't far off from being able to set up a small private cloud with just a little more hardware.
In order to proceed with some confidence I needed to make a few decisions about what technologies I would use. I wanted to be able to attach additional storage to my compute VMs. I also wanted to have the option for my cloud enabling technology to handle network security.
Another requirement was technologies that could be acquired and run for free, but had enterprise support available. Of course, whatever I chose would need to be compatible with RightScale so that I could easily provision and automate anything I might choose to run on my new garage cloud.
For the cloud enabling technology I settled on Citrix CloudStack 2.2.14, because it met my above criteria and has been proven to power some of the largest public and private clouds in the world reliably. Of course I could also have chose Eucalyptus or OpenStack which are technologies supported by RightScale. In fact, I'm seriously considering doing it all over with each of them for the experience. ;)
From there, the decision to use the Citrix XenServer 5.6 hypervisor was a clear choice since I knew it would work seamlessly with CloudStack, plus it supported security groups.
I realized pretty quickly that the loss of my CloudStack Management Server could be pretty catastrophic if I intended to run anything of consequence on my garage cloud. In order to have some resiliency I would have to do continuous backups of the management database at a minimum. Having a runbook or automation scripts to easily reconfigure the management server from my backup would be even better.
If you're thinking that sounds a lot like a RightScale ServerTemplate,
I'm glad you're enjoying the koolaid you're right on the money. In fact the Database Manager for MySQL 5.1 handles safely and continuously backing up your database content, as well as providing tools for restoring that data in case of a disaster.
Given that I needed most of the functionality of the database manager for my management server, I decided I would simply make a clone of that template and add the CloudStack specific installation to that new template. The result is my CloudStack Management Server Single-Node ServerTemplate which you can use to set up your own small scale private cloud without having to worry about the CloudStack installation and disaster recovery planning.
The AWS Node's Connected to the Garage Cloud
By now you may be asking yourself how I was able to use a RightScale ServerTemplate to setup my CloudStack Management Server on my bare metal hardware inside my network. If you took a closer look at the diagram above you've probably already figured it out, I'm not. Instead I'm running my CloudStack Management Server on an AWS instance and connecting it to my network via an OpenVPN IPSec tunnel.
Let's just get this out of the way, it's a little crazy to run the cloud management server for your private cloud; on a public cloud. I did it so that I could take full advantage of the configuration and orchestration tools built into the RightScale Database Manager ServerTemplate. Also that way I didn't have to "waste" a physical system that I could be utilizing for more important tasks, like running VMs!
This approach has it's own issues of course. Such as, How will my management server communicate with my hypervisor hosts? Will the images be stored on the management node, requiring my hypervisors to essentially download their OS from the internet each time?
As it turns out, setting up connectivity became the most complex part of my implementation, and I can't say that I would recommend it to anyone. I wound up using OpenVPN, setting up my All-in-One CloudStack Management server as the VPN server, and connecting my home network to it with an OpenVPN client.
*NOTE: I did do a fair amount of hacking to get OpenVPN to work in my environment, so that my management server could talk to all the servers on my local network. That's built into the ServerTemplate linked above, but chances are you'll need to do some tweaking to get your network working smoothly. Your mileage may vary, caveat emptor, etc.
Needless to say, networking is hard, when you get stuck it's probably a routing problem, and I'll stick with software engineering. :)
On the topic of having to traverse the VPN for images and disk I/O, since my file server was acting as the primary and secondary storage for CloudStack, my hypervisor hosts would be using a resource local to my network for it's root drive as well as any additional volumes I would attach. So, no weird situation where VMs are traversing the internet or my VPN.
Cloudifying the Bare Metal
With my storage, CloudStack Management Server, hypervisors, and networking figured out it was time to bolt everything together which meant spending some quality time with the CloudStack Install and Administrator guides.
Since I didn't have a very large cloud, nor did I have any serious networking gear to setup VLANs etc. I chose to use the basic networking mode of CloudStack. I also chose not to create a separate storage network to handle the burden of disk I/O traffic.
With such a simple configuration and flat networking my configuration came together pretty quickly. The one thing which was fairly confusing is that once I'd added all of the appropriate components (a Zone, Cluster, Hosts, IP ranges etc) I expected to be presented with a big red "GO" button to initialize my environment. Instead, CloudStack was periodically checking the configuration of my environment and when it determined that it had everything it needed the initialization started automatically. That struck me as not terribly intuitive, but it became clear if I was watching the log file.
Orchestrating the Garage Cloud
Once CloudStack was happy with my environment and configuration I could take a step back and start interacting with my new cloud the way that I've become familiar - using RightScale.
Cloud Registration Two-Step
The steps were easy here. I went to Settings -> Administered Clouds in the RightScale dashboard and added a new CloudStack based cloud. I was then prompted to provide the name of my cloud (I know, I get no style points for 'rjghome', it's boring) and a set of CloudStack API credentials with Admin permissions.
Now, this is where the process gets a tiny bit confusing. Once I've completed those steps, I'm immediately asked for another set of credentials. This time I'm not asked for detail like the API endpoint URL or the cloud name, but just a set of API credentials with User permissions. What's happening here is that the first step is simply registering my private cloud with the RightScale dashboard. That first step requires Administrator permissions because it has to be done by someone who has the authority to allow the cloud to be consumed by RightScale users.
Imagine that I were doing this as an IT project for a corporation, and I had built out a much larger infrastructure in my corporate datacenter. The purpose of that private cloud would be to allow people in the organization to use RightScale to deploy their own workloads on that private cloud without having to depend upon IT. However, IT would have to "register" that cloud with RightScale, making it available to other consumers within the organization. That step of initial registration requires Administrator permissions, then the IT department can distribute API credentials with User permissions to each group that will be consuming. You can read more about registering a CloudStack cloud in the RightScale support docs.
So, it's a producer/consumer model. Make sense? Cool!
RightImages for CloudStack
Ok, just one more step before I can start launching servers through RightScale on my Garage Cloud! Each of the ServerTemplates that RightScale publishes now includes a placeholder for a RightImage for each of the supported Hypervisors on CloudStack (KVM, XenServer, ESXi/VMWare) so that when you import them from the MultiCloud MarketPlace they will "just work."
However, in order for them to "just work" you need to upload the appropriate RightImage into your CloudStack environment, specifying the MD5 fingerprint of the image. RightScale will then match up the image you've uploaded to your CloudStack environment with the placeholder that's in the published ServerTemplates allowing you to launch them on your cloud.
After completing all of those steps I was finally able to launch a simple Base ServerTemplate for Linux which brings us full circle to the beginning of this post. It was super exciting to see a server become operational (something I've witnessed thousands of times) on my own private cloud, knowing that it was running on hardware I setup and configured myself.
So, really, why did I do it? Really the short answer is "because I could", but it was also an excellent learning experience. This exercise gave me the opportunity to really see the moving parts of a private cloud implementation, and let me get my hands dirty with the actual technologies.
I also now have a great test bed for building ServerTemplates without running up any public cloud usage fees. I guess that means I'll have more motivation to make my Wordpress All-in-One ServerTemplate multi-cloud, eh?
If you'd like some guidance on building a private cloud from me or one of my RightScale colleagues, just request a free consultation.