RightScale Blog

Cloud Management Blog
RightScale 2017 State of the Cloud Report
Cloud Management Blog

Migrating to Docker: Barely Controlled Chaos

Migrating to Docker: Barely Controlled Chaos

The RightScale engineering team is moving the entire RightScale cloud management platform comprising 52 services and 1028 cloud instances to Docker. This blog is the third in a series chronicling “Project Sherpa.”

Week 1 of Project Sherpa at RightScale is now complete. I’ll sum it up as intense, exciting, and (yes) fun. The sheer amount of questions, answers, and discussions in our Project Sherpa channel on Slack was pretty amazing and even inspiring.

The biggest thing that happened last week is that standards and conventions evolved ​really​ quickly once we got a lot of bright minds playing with the problem space. We had a heck of a time keeping everyone in sync, ensuring symmetrical knowledge exchange, and making sure that we didn't have any contradictory "standards."

Before the project started, we had codified our Interface Contracts between Dev and Ops in our Docker Bible (our how-to documentation). Here is an example specifically around Docker ENTRYPOINTs:

  • Your Docker image must have an ENTRYPOINT that runs a script; the script needs to respond to both types of bootstrap request: the one-word (just bootstrap and exit) request, and the two-word (bootstrap and enter service) request.
  • Your application must also do a sanity check on startup to ensure that it has been bootstrapped — that its database schema exists and all migrations have been run — and the app should crash if it has not been bootstrapped.
  • Your app must expose the bootstrap command word, even if it does nothing other than exit 0.
  • If your app performs database migrations, it should expose the migrate command word which runs migrations and then exits 0.
  • If your app runs an HTTP server of some sort, then it should expose the web command word.
  • Your app may expose other command words if you coordinate with Ops; for instance, web-master and web-shard are common for apps that behave differently in the global cluster vs. shards.
  • Other than bootstrap, there is no hard-and-fast rule about command words that every image must support.

During Week 1 our standards evolved as we got more people using Docker. One change was a new, simpler way to deal with Docker entrypoints and commands. Because many of the services we're Dockerizing predate the era of microservices, they do more than one thing; a given SCM repository will contain not only a REST API, but also daemons that run in the background to do batch processing. Therefore our images have more than one "personality," and when we run containers from them, we need to specify which personality we want.

We had been copy/pasting some boilerplate shell scripts between repositories to encapsulate all the complexity associated with running the right personality/command and ensuring that the inputs to the app are consistent regardless of the command. One of our engineers developed a simpler approach that bakes a reusable script into our Ops team's base image and allows the individual apps to have a tiny config file that describes the available commands.

We think that the first wave of standards  mostly regarding inputs and deployment details  has sunk in and the rate of change will go down in Week 2. We expect another wave of chaos once we get around to monitoring and alerting, where we haven't documented as much.

Our developers are really enjoying tackling the Docker migration as a team. Here is one of the quotes:

"It is kinda fun having everyone working on basically the same thing. Solutions to issues are coming in from all directions that are being used across multiple containerization projects on multiple teams."

And, on top of that we’re already making good progress. We got three services containerized already and expect the pace to pick up as we go. We’re also seeing some side benefits as well:

“Because it's containerized, it runs 3x as fast and uses half the memory, and also you get a pony if you log in.”

Onward and upward to Week 2!

To learn more about how we are using Docker, watch our on-demand webinar, Overcoming 5 Common Docker Challenges: How We Did It at RightScale, which covers:

  • Tips for securely producing high-quality, easy-to-manage Docker images
  • Options for multi-host networking between laptops and the cloud
  • Orchestrating Docker in production: service discovery, configuration, and auditability
  • Increasing host density with Docker
  • Dynamic monitoring and alerting for Docker containers

Watch the on-demand webinar.

This article was originally published on FierceDevOps.


RightScale 2017 State of the Cloud Report

Read More Articles on Migrating to Docker:
Migrating to Docker: Getting the Flywheel Turning
Migrating to Docker: Why We’re Going All in at RightScale
Migrating to Docker: Halfway There
Migrating to Docker: The Second (Harder) Half
Migrating to Docker: The Final Results