- Software development lifecycle (SDLC): A documented version of how you will develop software in a manner that ensures a certain level of quality in that code.
- Ticketing system for evidence collection: All compliance regimens require documentation and tracking, and a ticketing system provides a way to meet this requirement while automating the process.
- Source code control system: A system to provide appropriate tracking of changes as well as rollback functions.
- Mechanism to perform independent code review and testing: Automated or manual tools and processes that can be used to ensure a level of code quality that is consistent with the service you are delivering.
- Process for changes before deploying code to production: Formal approval process for all changes that will end up being deployed to the production product.
An Agile Process Under DevOps
Cloud provides developers with self-service access to infrastructure, creating a DevOps-oriented environment that often does not require a separate operations team. However, to ensure compliance, you still need to code review and test and get approval before pushing code to production. This is where it gets tricky. In such a shop, you could specify that all developers have authority to promote to production after an independent code review and test (you can’t get away without this). It would be a business and policy decision to allow any or all developers to promote to production, and one that you would need to justify to your auditors.
To meet almost any widely applicable compliance requirement (such as PCI, SSAE16, and so on), all code must be both reviewed and tested independently — that is, not by the developer who wrote the code. A single person can be both the reviewer and tester. So in a DevOps environment that uses Scrum, the process to meet this requirement would look like this:
- The sprint team creates a story about what to do in the agile tool (a ticketing system, for instance).
- A story/task is allocated to a team member (developer).
- The developer tags code changes with the ticket number related to the story.
- Once the changes are done, the developer puts the story/task into review/test state, and another developer picks it up, or, ideally, the ticketing system uses a workflow associated with it to route the task to the appropriate developer.
- Once the testing succeeds, the task is marked complete.
- The developer then promotes the story/task to production.
If you have implemented adequate automation and tracking tools, and all of your developers have the requisite skills to deploy production code, then you should be able to make a business case that they be able to do so. Don't overlook the requisite skills part of that last statement. In a true DevOps environment, you will need to ensure that the developers deploying to production are indeed qualified and authorized to approve the changes they are deploying. You may be able to justify having a junior developer promote to production if you have an adequate level of expertise in the review and test stage.
How We Do It at RightScale
It’s not trivial to create a bulletproof, robust, and consistent process that will pass an audit. Saying that you do it is easy, but evidencing it is the hard part. In particular, implementing the code review and testing process is the most difficult part of development and compliance in agile environments. At RightScale, we embrace the agile methodology through Scrum, but we have not taken agile all the way to a DevOps delivery model. Thus we have a bit more “tradition” in our development practices. But we use RightScale Cloud Management to deploy the automation systems that help perform our testing, and all of our development environment is effectively "in the cloud."
Software Development Lifecycle
Our SLDC is defined in our RightScale Agile Manifesto (RAM) — think Agile Manifesto++. The RAM covers the following:
- Anatomy of a release
- Commit and branching standards
- Sprint development capacity
- Sprint QA engineer concerns
- Scrum lifecycle
- Sprint metrics
- Integration, release candidate, and release branches
- Release sequencing
- Documenting story requirements
- Sprint end cleanup
- Feature freeze/branching date
- Engineering escalation process
- Development standards
Stories and Ticketing
For each story that is to result in a code change, a developer creates code review (CR) and QA subtasks in the agile tracking tool. The code cannot be merged into the master source code branch and ultimately deployed to production unless the CR and QA subtasks have been completed.
The development flow is as follows:
- When code is submitted to our version control system, the event is analyzed in real time. We use an automated task to ensure that each commit is associated with a legitimate story/task, and generate an alert if it is not.
- The commit message is verified to ensure that it contains the prerequisite ticket reference in the desired state. Our agile tracking tool creates a unique ID for each story.
- When a story is complete, we perform a merge into the master source tree.
- The merge is verified that it is complete and done by an individual who is authorized to perform the merge. In the event of an unauthorized merge, the engineering director or team lead determines the cause of the error and makes appropriate corrections.
- During a release, developers cut a release branch and perform final testing, and our operations team deploys the release into production.
- We have a scheduled audit that covers any abnormalities in the real-time process.
Note that the checks are not all devoted to compliance, but also to improving engineering visibility and ultimately efficiency. We can use them to see if someone had no time allocated to work they did, or worked on a deleted task. We also do more traditional sweeping validation because real time is subject to blips beyond our control.
By using our tools and implementing processes, we ensure that every change:
- Is documented via a story or task ticket
- Has undergone independent code review
- Has undergone independent testing
- Is approved for deployment to production
Furthermore, we have evidence of each of the steps for proof audits.
In some agile environments, managers may get pushback from developers who feel they are being constrained by the rigorous compliance processes, but these controls help improve the quality of code, and that is hard to argue against. Developers who think their code does not need review or test are fooling themselves, and will likely be the cause of a company’s major breach.
None of this effort to ensure compliance is free — it costs time and effort. We use tooling, along with our own RightScale Cloud Management, to help streamline the process. In the end, the benefits you gain more than outweigh the drawbacks. How much would a code review of the LinkedIn password storage have saved that company in the long run?
The bottom line is that you can have compliance in agile environments. Take the time to do it right and you’ll reap the benefits of both compliance and agile development.
To learn how to accelerate your software development lifecycle and maintain repeatable, compliant processes, download our white paper on using a self-service portal to access pre-configured development and test environments in the cloud.