What's with the check boxes you ask? It's a partial visual representation of what it looks like to manage 83 users, 15 RightScale accounts, and 11 permissions for each user in each account. That's 13,695 check boxes I'd potentially need to click. So I decided that I needed to either hunker down with a carafe of coffee for a few days or find a better solution!
Now, before I go too much further, you should know what it was that I was trying to accomplish. I was tasked with getting user access to all of our RightScale demo accounts sorted out and standardized.
At RightScale we have a mix of sales staff, product management, marketing, and even some engineers who need some level of access to the demo accounts we use when showing prospective customers how RightScale works. Of course, not all users are created equal. A Cloud Solutions Engineer like me, for instance, would have very different permissions than a Sales Associate.
What I needed was to be able to group users into specific roles, assign permissions to those roles, and then apply those changes to the RightScale user access control. As I mentioned at the outset, that meant managing nearly 14,000 individual permissions (actually a bit more since there were more than 83 users to start with before I removed some).
To put the task ahead of me into perspective, consider that the image at the top of this blog post contains exactly 640 checkboxes. To see what 14,000 check boxes looks like, you can check out this page.
Needless to say, there was no way I was going to manually go through the process of changing all of these permissions. It would take me hours, possibly days, and what would happen if I actually needed to change some of the user assignments at a later date? I didn't even want to consider the ongoing overhead.
User Management at Scale
Now, as daunting as changing so many permissions seemed at the time, our environment with 83 users and 15 accounts is actually fairly small by comparison to the IT footprint of a large enterprise. Imagine this same scenario with 100 accounts and several hundred users, all with very specialized access needs.
Now imagine that each of those users needed to access not just AWS, or Rackspace, or Google Compute Engine, but all three, as well as a dozen or so private cloud environments.
Next imagine that you were responsible for managing access control for all of those users across all of those resource pools and services. You'd have an exponentially larger number of check boxes, and you'd also have to interact with dozens of different API endpoints with varying degrees of user access control.
As you read on, remember that all of the user-access control that is being manipulated automatically applies to every resource pool that you manage with your RightScale account(s). Users are given one username, one password, and one URL to remember. They log in to RightScale and have consistent access (managed by and visible to the enterprise) to all of their cloud resource pools.
If you don't currently use RightScale and want to see how user-access control works (among many other things), you can try our Free Edition.
I'm going to go on to tell the (rather lengthy) story of how I arrived at a tool that allows me to easily update permissions across multiple users and RightScale accounts simultaneously. If you're looking for the short version...
I wrote a Ruby Gem named rs_user_policy that consists of a library and a command line utility that ingests JSON files describing users and their associated permissions and then applies them to RightScale accounts via the API, using the right_api_client Ruby Gem.
While the command line tool uses JSON files as it's input source, the library was designed to be extensible enough to use other sources for role definitions and user assignments. For instance LDAP or AD are logical sources, which I'll elaborate on later in this post.
You can skip the rest of the story and go straight to the conclusions now if you like.
Fortunately everything we do here at RightScale revolves around automation. This becomes especially important when you start to operate at scale with many users, resource pools, and applications. Managing that scale simply can't be done through a UI. Instead a well-documented REST API becomes mandatory to reduce burden and eliminate human error.
With that in mind, everything that I was trying to accomplish to audit the user access to our sales demo accounts would be accomplished via the RightScale API, and I set out to create a simple tool to take care of the heavy lifting.
Organizing and Categorizing
My first task was to identify the types of users who needed access to the demo accounts and what they needed to do.
For most users, all they needed was to be able to log in to the account and view the deployments that were there. For them, only the "observer" and "billing" permissions were appropriate.
For other users, like the Cloud Solutions Engineer team and select folks from product management and engineering, a more expansive set of permissions was appropriate.
I set out to start documenting these "roles" and chose to express it in a JSON document. My first draft of this description looked something like the following:
This basically states that users assigned to the "Engineer" role get the "observer," "actor," "admin," "designer," "billing," "server_login," and "library" permissions on every sales demo account. For those assigned the role "SalesAssociate," only the "observer" and "billing" permissions are granted. Lastly, for the role "SalesAssociate," these people get slightly elevated privileges for the account with the ID 12345, namely gaining the "actor" permission.
* Note: If you're wondering about the meaning of these RightScale permissions, take a look at User Role Privileges
With a basic mechanism for describing roles decided on, I then needed to assign the users who were already members of the sales demo accounts to the defined roles. This is where I started to actually produce some lines of Ruby code to call the RightScale API.
I wrote a few lines that would use the RightScale API to find all the child accounts for our sales demo enterprise account, iterate over them, and produce a list of all the existing users. Because I was going to assign those users to roles, I figured I'd assign a default role to newly discovered users. I chose "immutable" to describe existing users and to indicate that their role and permission set should not be changed. The output looked something like this:
So then I could start to assign the roles to users who were found in the RightScale accounts by looking through the generated list and changing "immutable" to the role that is appropriate for the user. By way of example, I could assign "firstname.lastname@example.org" to the "Engineer" role, and "email@example.com" to the "SalesAssociate" role to indicate the permissions that those users should be granted.
Lastly, if I found a user who I no longer wanted to have any access to any accounts, I could assign the role "delete," and all their permissions would be revoked.
RightScale API calls used:
With user roles defined in one JSON file, and role assignments for each user in the accounts in another, it was time to actually compare existing users and their permissions to the policy these files defined.
I'll spare you the nitty gritty detail (you can dive into it here if you're interested), but this process consisted of finding the permissions the user currently had, comparing that against the desired permissions for the users role, and removing or adding the appropriate permissions to enforce the policy.
RightScale API calls used:
Plot Twist #1
Satisfied with my accomplishment and self-assured that I'd solved the problem completely, I went ahead with assigning everyone whose role was obvious and then ran the tool.
All of the assigned users now had exactly the defined permissions across all of the demo accounts.
However, there were still plenty of users who legitimately required access to the demo accounts, but whose roles and associated permissions weren't readily apparent. I started to ask the users what they used the demo accounts for, and very quickly realized that I would likely have to create a different role for each person since they each had very specific requirements.
To accommodate this, each user needed to be able to be a member of more than one "role." Furthermore, roles could be defined a bit more abstractly since they could be combined. For instance, I could create roles that have elevated permissions for only a specific RightScale account and combine that with a general "SalesAssociate" role for the same user, effectively giving a user basic permissions on all accounts and elevated permissions for an account that is appropriate for that user.
Difficult as it is to see in the resized images above (click on them to see the images full size), what's going on here is that I've added two new roles named "GCEPowerUser" and "AzurePowerUser," which give elevated permissions to specific RightScale accounts. In my user assignment file, I now have a property for each user named "roles," which is an array of one or more roles the user belongs to.
This allows me to assign two new users, a Google engineer and an Azure engineer, appropriate permissions for their roles.
Plot Twist #2
After interviewing all of the users with special role requirements and accommodating them, I once again ran my tool to align all of the permissions to the policy.
All of the appropriate permissions were set, and everyone had the access they needed. I decided that I would set this up to run on a regular basis (once a day) to make sure that the permissions remain in compliance with the policy.
Then it dawned on me...
The tool only acts upon users who are already added to one of the RightScale accounts it acts upon. As we hire new employees, they would have to be manually invited to one of the accounts. That would cause them to be discovered by the tool and assigned the "immutable" role, and their permissions would never be maintained by the tool.
So, I needed to allow the tool to add new users by specifying them in the user assignments file.
Above is a snippet of a user assignment file that contains only a single net-new user. This could just as easily be mixed in with several other existing users. The "roles," "company," "first_name," "last_name," and "phone" fields are all required. You can also supply any other field defined by the User#create call in the API, perhaps the most interesting of which is "password," because if no authentication information (password, or identity_provider_href and principal_uid) is supplied, a strong password will be automatically generated for the user and written back to the user assignments file.
With that, the user assignment file can be the authority for all users who should exist for the set of accounts that are maintained by the tool.
RightScale API calls used:
LDAP or AD Integration
With those use cases covered, I was satisfied that the tool could do the job I'd created it for. Namely to run on a regular basis and make sure that the existence and permissions of users was enforced across all of our sales demo accounts according to the content of a couple JSON files.
However, the tool was designed to be extensible regarding the authoritative source for users and permissions. There are effectively two types of information: the roles and their permissions and user information (role assignments, first and last name, phone, etc.).
In the Ruby Gem, there is an interface defined for the policy (the roles and permissions for those roles) and the user assignments. You need only implement that interface to pull from the data source of your choosing to leverage the rest of the functionality.
Keep an eye out for a future blog post from our product team, who will explore this idea some more.
Audit History and Compliance
In addition to automatically managing user permissions, the tool also produces an audit log after reach run. This audit log is also in JSON and records all of the actions it took. You can see an example of the output below:
When this tool runs against our demo accounts, I grab this output file and toss it in an S3 bucket so I can go back and see the changes the tool has made over time. A very valuable tool for audit and compliance.
When I was first asked to clean up the user access to our demo accounts, I didn't think much of it. It wasn't until later that I realized the scale of the problem and knew that I needed to approach it differently.
Managing user-access control on a large scale with the RightScale API demonstrates how powerful it is. When I was faced with a seemingly insurmountable problem, I was able to leverage the API to not only perform the task just once, but to also implement a tool that would help me to enforce the changes I'd made on an ongoing basis.
As I mentioned before, my task of managing ~100 users and 15 accounts is peanuts compared to the challenge a large enterprise or managed service provider might be facing. When you reach that scale, having a unified user-access policy for all of your cloud resource pools and being able to automatically manage it through an API becomes an absolute business requirement, not just an academic exercise.
If you're facing the problem of managing a large-scale user base in RightScale, by all means, check out the rs_user_policy gem I created.
If you're facing a different problem that seems daunting, consider taking a second look to see if you might be able to solve the problem more elegantly and easily using the RightScale API. To get started, you can get a RightScale free trial.