RightScale Blog

Cloud Management Blog
Cloud Management Blog

RightLink Agent Security Features & Upgrading from V4 RightImages

Posted by Phil Cox   Ι   January 24, 2012

A fundamental problem in cloud management is "how do I get the remote instance to do what I want it to?" Taking this task on for a few systems is doable with a number of techniques, making it scale for many thousands is not quite as simple. At RightScale, we have been on the bleeding edge of this issue since the early days of cloud computing, and we have learned a lot along the way. One of those lessons has led to our implementation of the RightLink agent and the RightNet protocol.

Introduction to RightLink

RightLink is the instance side agent that supports RightScale's RightNet protocol. The agent provides an improved and secure ability to leverage RightScale to manage large numbers of instances in the cloud. In the RightScale architecture, we leverage a light-weight RightLink agent on every instance to support our latest automation features. Prior to RightLink, which was released a bit over year ago, RightScale leveraged the command execution features of SSH to perform tasks on remote instances. With the introduction of RightNet and the RightLink agent, we are no longer reliant on SSH access for instance management.

The RightLink agent communicates with the core RightScale systems using the Advanced Message Queuing Protocol (AMQP). RightNet leverages AMQP's Simple Authentication and Security Layer (SASL) support to perform a basic session authentication to ensure that the RightLink agent is talking to a legitimate RightScale core component (a broker in our lingo). This session authentication uses a shared key to authenticate the ends. After the session is authenticated RightNet uses payload encryption (openssl with X509 certificates, PKCS7 envelopes and AES 256 CBC cipher for encryption) to protect that data while in transit, and to provide a much stronger authentication mechanism (public-private key versus only the shared key of the session). Both of these security features are to ensure that packets are properly segmented and protected in the highly multi-tenant aspect of the cloud.

All our version 5 (v5) RightImages and MultiCloud Images (MCI) include the RightLink agent by default. We started releasing v5 images over a year ago, and have seen a large, but not complete, adoption. For those of you still on v4 images, I am going to try to give you a couple more security motivations that may encourage you down the upgrade path.

  1. Ability to restrict SSH access on the instance: Because RightLink does not use SSH you can restrict access to the ssh service on Linux systems. With non RightLink enabled images (i.e., v4 and earlier by default), the RightScale platform ran scripts on the instance by ssh-ing into that instance directly, thus the need for ssh port to be accessible on the instance from the RightScale platform usually meant that it was accessible from any IP address. This created some exposure with potential brute force attacks. I will say that by default, RightImages configured SSHD to support public-key authentication only, so the risk of brute force password guessing was not an issue. What was an issue was that any vulnerability found in the SSHD server would then be potentially exploitable by anyone on the Internet.  With RightLink, this exposure can be mitigated.
  2. Managed SSH: In addition, v5 RightImages introduce a "Managed SSH Login" feature. This allows you to use a different SSH key for each user logging into a server. It can either use an SSH key uploaded by each user or the dashboard can generate a key for each user.  When using EC2 you may still select an EC2 SSH Key when launching the instance, however, it's only really necessary if you need to log-in before RightLink starts to troubleshoot something in the bootstrap process. Note that the SSH connection is from your desktop system (wherever you are running the dashboard UI from, not RightScale) to your instance, thus working seamlessly with any SSH access restrictions you put in place.

SPOILER ALERT: One of the items we are working on for RightLink v5.8 (next version coming out) is a Managed SSH Login that will bind each RightScale authentication principal to a distinct, non-root Unix user whenever they login via the dashboard. This is intended to improve the login auditing as well a enable each user to load a customized shell profile. We'd be very interested in your feedback as to the usefulness and desire of this specific feature.

Upgrade options

The cleanest and best way to move to v5 images is to find a v5 ServerTemplate, clone it and make the modifications needed to effectively duplicate the functionality you currently have. This will work like a charm if you if you did your scripts right and took a modular approach to deployment.

Next option is to change the RightImage MCI you're using to a v5 one and relaunch. The V5 execution of RightScripts is almost fully compatible with v4 so, in theory, that's all you need to do. The catch typically is that this brings updated versions of the OS and packages with it and may cause some incompatibilities. You will probably spend a bit more time troubleshooting this avenue.

Lastly, you can get RightNet support by RightLink-enabling your v4 instance, and many might be motivated to go that route. I would encourage you to move to v5. While you'll get the "not using ssh for command and control" benefit, you will miss many other benefits of the v5 image update.

Why Again?

Because there are some really cool features in v5:

  • Managed SSH
  • Bug fixes
  • Faster Execution of Operational Scripts
  • Added Chef Support in addition to RightScripts

It will take a bit of effort, but I guarantee the improvements you gain will be worth it! My advice to those RightScale customers with older versions: "If you're hanging on to v4 or earlier, you really should upgrade."

 

Comments

Blog says "thus the need for ssh port to be accessible on the instance from the RightScale platform usually meant that it was accessible from any IP address." Why can't we limit the ssh port access to only Rightscale trusted servers using firewall for the instance? Will the "Managed SSH" feature use SSH protocol for communication with the instance or will it use some other mechanism?
Posted by Daniel (not verified)   Ι   January 24, 2012   Ι   02:17 PM
Response by Phil Cox   Ι   January 24, 2012   Ι   02:28 PM
Daniel, you theoretically can limit to known RightScale instances, however you run into a potential issues with auto scaling: If you restrict by IP and we autoscale, then the new systems will not be in your allowed list and could cause a self-induced DoS. Same thing with failover. If a region fails and RightScale has to fail over to a different cloud/region, the ability to manage resources in the Cloud Provider would break until they updated the whitelist. As far as the “Managed SSH” feature, it does use SSH, but from YOUR system to the instance, not the RightScale core.
Response by Daniel (not verified)   Ι   January 25, 2012   Ι   04:35 PM
Phil, shouldn't it be easy to update the firewalls rules for the all the servers managed by RightScale automatically anytime RightScale adds or removes the servers? RightScale platform knows the IP addresses of the servers added or removed to the internal RightScale server pool and also has all the information to connect to all the managed servers to update the firewall rules. You guys already have the scale and infrastructure to run operational scripts for all servers under RightScale management. Why auto-scaling of Rightscale instances can’t be configured to automatically update the firewalls rules on all servers managed by RightScale? Other option is to limit access to the port to a predefined large enough range of Elastic IP pool for RightScale servers in the RightScale primary and DR site. Looks like we need to configure the firewall to allow transportation of RightAgent related packages. Without proper management of firewall rules from what you are describing it appear that for the RightAgent to work the communication port used by the RightAgent should be open for traffic in the firewall to be accessible from any IP address in the world!
[...] RightLink Agent Security Features and Upgrading from V4 RightImages (rightscale.com) [...]
[...] RightLink Agent Security Features and Upgrading from V4 RightImages (rightscale.com) [...]
[...] RightLink Agent Security Features and Upgrading from V4 RightImages (rightscale.com) [...]
Daniel, you can update firewall rules, but there are a number of reasons why I would not do it that way. Might be better to have a conversation on this, as it is very contextual. As for the RightLink agent, and other components of RightScale, sessions are initiated from the instance to RightScale servers. A typical client environment has firewall ingress rules that allow what ever services on the instance they want (i.e., not RightScale related) and egress rules that restrict certain ports (assuming statefull firewall), but typically not to a specific IP. As I stated earlier, it is possible to restrict egress IP as well, and we have a mechanism to do that, but operationally there are potential issues that usually outweigh any potential benefit. Again, more than happy to continue the discussion here, but might be more useful in a chat. If you'd like, send me email thesecurityguy at rightscale.com
Posted by Phil Cox   Ι   January 26, 2012   Ι   07:25 AM
[...] RightLink Agent Security Features and Upgrading from V4 RightImages (rightscale.com) [...]
Bryan, Let me know how it goes. Also, if there is anything I can do to assist, let me know as well.
Posted by Phil Cox   Ι   February 02, 2012   Ι   09:49 AM
I can't wait for the v5.8 templates and MCIs. Our company had to implement our own user separation by parsing the authorized_keys file in roots .ssh directory. I would be very interested in talking with anyone on your team in regards to this feature. Perhaps we could help identify extra use cases or configuration details. I had previously requested this feature. In my mind, this is a huge value add for RightScale. We had to implement it ourselves as we prepared for a PCI audit and everyone using root to access the box (separate keys or not) was not going to pass the audit.
Posted by Dave Morehouse (not verified)   Ι   January 24, 2012   Ι   08:34 AM
Dave, If you want to be a bit adventurous, there are a couple ways you can get it now :) Option 1: Get RightLink 5.8 - Fetch the RightLink git repository and follow the README.doc to install RightLink by hand Option 2: "bleeding edge" - Point your instances at Tony Spataro's personal tarball and boot in "tarball mode" (no guarantees that the tarball stays stable, but it works now) - To do "tarball mode" - choose a MultiCloudImage that uses a recent vintage of RightLink (e.g. 5.7). Ubuntu is strongly preferred - Tag your server with "rs_agent_dev:package=tony" - Launch the server If you're playing with it, try doing some sftp and scp commands and look at the audit history. you will be pleasantly surprised.
Posted by Phil Cox   Ι   January 24, 2012   Ι   10:11 AM
I too am very excited about the non-root user mapping. I am also in the midst of home-building an authentication cred distribution system to aviod a huge red flag in a SOC audit. I'll try out the 5.8 beta images.
Posted by Bryan McGowan (not verified)   Ι   February 01, 2012   Ι   08:53 AM

Post a comment