RightScale Blog

Cloud Management Blog
Cloud Management Blog

Build a Private Cloud in Your Garage

A few weeks ago, this brilliant idea crossed my mind. Why don't I build a private cloud at home, in my garage, so that I can get the experience of setting up the technology?

Vijay told me I was crazy, so to prove just how easy it would be I did it anyway, and as a result I've had a private cloud running in my garage since early March.

First, Some Background

Before I get started there are a few housekeeping items I'd like to quickly address.

TL;DR - Too Long, Didn't Read

I think it's only fair to tell you that this post is a bit lengthy since I wanted to tell the full story. I also wanted to cover as much of the details of the process as possible. Even so, I wasn't able to deep dive into everything so if you have questions please fire them off in the comments and I'll be happy to keep the conversation going there. With that said, grab a cup of coffee and enjoy the story!

If you're looking for the tl;dr version, the diagram under So Meta... should tell you everything you need to know.

And if you're looking for an in-depth guide to best practices for building private clouds that include reference architectures, check out Brian Adler's white paper on Designing Private and Hybrid Clouds.

The Road Ahead

It may also be helpful to have a quick outline of the topics I'm going to discuss. First, I'll walk you through some of the decision making process. How did I arrive at the decision to build a private cloud? How did I choose the technologies and hardware? Then I'll tell you a bit about the mechanics. How did I put the pieces together? What does my infrastructure look like? And lastly I'll show you some of the fruits of my labor (my favorite part). :)

Cloud-in-My-Garage Epoch

I had been contemplating an upgrade to my local home file server for quite a while. My research had lead me through all of the NAS devices and other consumer options but at the end of the day I really wanted to build my own system.

That realization led me to looking at some rack mountable servers with hot swappable SATA on eBay for hours on end. When I finally found the right server, and it was enroute to me via the logistics loving courier, I realized that I wasn't far off from being able to set up a small private cloud with just a little more hardware.

Technology Shuffle

In order to proceed with some confidence I needed to make a few decisions about what technologies I would use. I wanted to be able to attach additional storage to my compute VMs. I also wanted to have the option for my cloud enabling technology to handle network security.

Another requirement was technologies that could be acquired and run for free, but had enterprise support available. Of course, whatever I chose would need to be compatible with RightScale so that I could easily provision and automate anything I might choose to run on my new garage cloud.

For the cloud enabling technology I settled on Citrix CloudStack 2.2.14, because it met my above criteria and has been proven to power some of the largest public and private clouds in the world reliably. Of course I could also have chose Eucalyptus or OpenStack which are technologies supported by RightScale. In fact, I'm seriously considering doing it all over with each of them for the experience. ;)

From there, the decision to use the Citrix XenServer 5.6 hypervisor was a clear choice since I knew it would work seamlessly with CloudStack, plus it supported security groups.

So Meta..

I realized pretty quickly that the loss of my CloudStack Management Server could be pretty catastrophic if I intended to run anything of consequence on my garage cloud. In order to have some resiliency I would have to do continuous backups of the management database at a minimum. Having a runbook or automation scripts to easily reconfigure the management server from my backup would be even better.

If you're thinking that sounds a lot like a RightScale ServerTemplate, I'm glad you're enjoying the koolaid you're right on the money. In fact the Database Manager for MySQL 5.1 handles safely and continuously backing up your database content, as well as providing tools for restoring that data in case of a disaster.

Given that I needed most of the functionality of the database manager for my management server, I decided I would simply make a clone of that template and add the CloudStack specific installation to that new template. The result is my CloudStack Management Server Single-Node ServerTemplate which you can use to set up your own small scale private cloud without having to worry about the CloudStack installation and disaster recovery planning.

The AWS Node's Connected to the Garage Cloud

By now you may be asking yourself how I was able to use a RightScale ServerTemplate to setup my CloudStack Management Server on my bare metal hardware inside my network. If you took a closer look at the diagram above you've probably already figured it out, I'm not. Instead I'm running my CloudStack Management Server on an AWS instance and connecting it to my network via an OpenVPN IPSec tunnel.

Let's just get this out of the way, it's a little crazy to run the cloud management server for your private cloud; on a public cloud. I did it so that I could take full advantage of the configuration and orchestration tools built into the RightScale Database Manager ServerTemplate. Also that way I didn't have to "waste" a physical system that I could be utilizing for more important tasks, like running VMs!

This approach has it's own issues of course. Such as, How will my management server communicate with my hypervisor hosts? Will the images be stored on the management node, requiring my hypervisors to essentially download their OS from the internet each time?

As it turns out, setting up connectivity became the most complex part of my implementation, and I can't say that I would recommend it to anyone. I wound up using OpenVPN, setting up my All-in-One CloudStack Management server as the VPN server, and connecting my home network to it with an OpenVPN client.

*NOTE: I did do a fair amount of hacking to get OpenVPN to work in my environment, so that my management server could talk to all the servers on my local network. That's built into the ServerTemplate linked above, but chances are you'll need to do some tweaking to get your network working smoothly. Your mileage may vary, caveat emptor, etc.

Needless to say, networking is hard, when you get stuck it's probably a routing problem, and I'll stick with software engineering. :)

On the topic of having to traverse the VPN for images and disk I/O, since my file server was acting as the primary and secondary storage for CloudStack, my hypervisor hosts would be using a resource local to my network for it's root drive as well as any additional volumes I would attach. So, no weird situation where VMs are traversing the internet or my VPN.

Cloudifying the Bare Metal

With my storage, CloudStack Management Server, hypervisors, and networking figured out it was time to bolt everything together which meant spending some quality time with the CloudStack Install and Administrator guides.

Since I didn't have a very large cloud, nor did I have any serious networking gear to setup VLANs etc. I chose to use the basic networking mode of CloudStack. I also chose not to create a separate storage network to handle the burden of disk I/O traffic.

With such a simple configuration and flat networking my configuration came together pretty quickly. The one thing which was fairly confusing is that once I'd added all of the appropriate components (a Zone, Cluster, Hosts, IP ranges etc) I expected to be presented with a big red "GO" button to initialize my environment. Instead, CloudStack was periodically checking the configuration of my environment and when it determined that it had everything it needed the initialization started automatically. That struck me as not terribly intuitive, but it became clear if I was watching the log file.

Orchestrating the Garage Cloud

Once CloudStack was happy with my environment and configuration I could take a step back and start interacting with my new cloud the way that I've become familiar - using RightScale.

 

RightScale Free Trial for OpenStack Cloud Management

 

Cloud Registration Two-Step

The steps were easy here. I went to Settings -> Administered Clouds in the RightScale dashboard and added a new CloudStack based cloud. I was then prompted to provide the name of my cloud (I know, I get no style points for 'rjghome', it's boring) and a set of CloudStack API credentials with Admin permissions.

Now, this is where the process gets a tiny bit confusing. Once I've completed those steps, I'm immediately asked for another set of credentials. This time I'm not asked for detail like the API endpoint URL or the cloud name, but just a set of API credentials with User permissions. What's happening here is that the first step is simply registering my private cloud with the RightScale dashboard. That first step requires Administrator permissions because it has to be done by someone who has the authority to allow the cloud to be consumed by RightScale users.

Imagine that I were doing this as an IT project for a corporation, and I had built out a much larger infrastructure in my corporate datacenter. The purpose of that private cloud would be to allow people in the organization to use RightScale to deploy their own workloads on that private cloud without having to depend upon IT. However, IT would have to "register" that cloud with RightScale, making it available to other consumers within the organization. That step of initial registration requires Administrator permissions, then the IT department can distribute API credentials with User permissions to each group that will be consuming. You can read more about registering a CloudStack cloud in the support docs.

So, it's a producer/consumer model. Make sense? Cool!

RightImages for CloudStack

Okay, just one more step before I can start launching servers through RightScale on my Garage Cloud! Each of the ServerTemplates that RightScale publishes now includes a placeholder for a RightImage for each of the supported Hypervisors on CloudStack (KVM, XenServer, ESXi/VMWare) so that when you import them from the MultiCloud MarketPlace they will "just work."

However, in order for them to "just work" you need to upload the appropriate RightImage into your CloudStack environment, specifying the MD5 fingerprint of the image. RightScale will then match up the image you've uploaded to your CloudStack environment with the placeholder that's in the published ServerTemplates allowing you to launch them on your cloud.

So...Why again?

After completing all of those steps I was finally able to launch a simple Base ServerTemplate for Linux which brings us full circle to the beginning of this post. It was super exciting to see a server become operational (something I've witnessed thousands of times) on my own private cloud, knowing that it was running on hardware I setup and configured myself.

So, really, why did I do it? Really the short answer is "because I could", but it was also an excellent learning experience. This exercise gave me the opportunity to really see the moving parts of a private cloud implementation, and let me get my hands dirty with the actual technologies.

Plus, how cool is it to be running a full 3-Tier PHP stack in about 30 minutes on your own private cloud?

I also now have a great test bed for building ServerTemplates without running up any public cloud usage fees. I guess that means I'll have more motivation to make my Wordpress All-in-One ServerTemplate multi-cloud, eh?

If you'd like some guidance on building a private cloud from me or one of my RightScale colleagues, just request a free consultation.

 

 

Comments

Mike, when I got it all put together I snapped a couple cellphone photos, intending to grab something better later. Here's a link to those lower quality snaps. :) That dresser it's sitting on contains all my carwash/detailing goodies! The plan is to put it in a small rack at some point. http://www.flickr.com/photos/rgeyer/sets/72157629331312068/
Posted by Ryan Geyer   Ι   March 29, 2012   Ι   10:37 AM
hmm this looks awfully familiar... mine comes in my own rack. :) http://www.bradderstine.com/wp-content/uploads/2012/03/DSC00264.jpg
Posted by bderstine (not verified)   Ι   March 29, 2012   Ι   08:49 PM
Response by Ryan Geyer   Ι   March 30, 2012   Ι   09:28 AM
Brad, About half way through this process I discovered your blog post on the same topic. I tried reaching out to you via twitter and your comments, but was unable to reach you. I'd love to compare notes! And yes, I am VERY jealous of your proper rack!
what was the total cost for the project?
Posted by AAndersen (not verified)   Ι   March 30, 2012   Ι   04:22 AM
Response by Ryan Geyer   Ι   March 30, 2012   Ι   09:27 AM
All in it cost around $1100. That excludes a few items I already had on hand like cabling, an LCD (not necessary really), the DD-WRT router/VPN, and the 8 port switch. The whole thing could be reproduced pretty readily under $1500 I imagine.
Hello Ryan, Could you post a detailed list of the hardware used ? I want to buy some servers for a similar project and i`m looking for cheap servers your setup sounds interesting. Best Regards, Hristofor Georgiev
Hristofor, here's the bits that make up my private cloud. 1) 1/ea - Supermicro Super Server cse-822t-550l 6-Bay rack mount (NFS/NAS Server) 2) 3/ea - SuperMicro 6015B Server Dual (2x) Intel Quad Core Xeon 1.86Ghz, 16GB, 160Gb (Hypervisors) 3) 1/ea - Linksys WRT54GS Router. Basically, it's one 2U rackmount box with a SATA backplane for file storage, and three 1U boxes for hypervisors. If I were to do it again *right now* using stuff on ebay, I'd pick these.. File Server - http://bit.ly/I271Zq Hypervisors - http://bit.ly/I27fj8
Posted by Ryan Geyer   Ι   April 21, 2012   Ι   04:41 PM
That's great and awesome. With this we can build cloud according to our choice.
Hey Ryan, I'm a Computer Networks student (2nd year), with a little experience in Java and C programming languages and a bit of network acquaintance... I want to start off in cloud computing and eventually get to be able to create my own private cloud, however I am not sure where to start, i.e. background reading, required package/software knowledge... what advices would you give me ?
Robiro, the first step I'd take is to sign up for http://aws.amazon.com. For 1 year, you can get a free micro instance so you can explore their options and see what's available. I haven't found a good book to read, but I'm one of those people who dives in and likes to see how things work. If you have a good system at home with lots of memory, you can install a private cloud like VMware ESXi or any of the cloudstack/openstack builds.
Hi Ryan, I need to build a priavte database cloud and move multiple on-premise databases to the private cloud. Is it possible for you to outline some steps for achieving this? Don't we also need some sort of mechanism for users to request and destroy database instances on-demand? Thanks RG
Posted by Rohit (not verified)   Ι   September 03, 2012   Ι   02:22 AM
Hi Ryan, I need to build a private database cloud for an health care system(just a small simulation for my final year project) and move multiple on-premise databases to the private cloud. Is it possible for you to outline some steps for achieving this? concerning the server thing i was told i could use linux fedora OS.....i do i design this Thanks patrick
Posted by patrick (not verified)   Ι   October 25, 2012   Ι   07:59 AM
Good summary, very cool, and you could also build the private cloud in less space and lower cost with a CriKit http://www.crikit.info

Post a comment