Quicker, Easier, More Seductive: The Dark Side of DevOps

Author: David Pratt

DevOps can revolutionize the way we deliver software and be the “force” that harmonizes development with operations. However, it is not without its challenges. There are some common pitfalls, a dark side if you will, that may corrupt the process.

Moving to DevOps requires a shift in company culture and a change to traditional roles and responsibilities. In part 1, we looked at the benefits of DevOps. In this second article, we’ll discuss some challenges organizations face when implementing DevOps.

A Culture Shift

With traditional methods, there is a distinct separation between server operations and software development. There is often resistance to bringing these teams closer because the traditional infrastructure-developer relationship is “you build it, we’ll run it”. Giving up control and working together is never easy.

At first glance, DevOps may appear to be a way to further silo these teams, with automation replacing some of the necessary person-to-person interactions. Far from it! The real benefits of DevOps spring to life when both sides collaborate and grow the process, technology, and culture of success together.

As teams are exposed to the other’s responsibilities, developers can support deployment automation and thereby gain a greater understanding of the entire process. Infrastructure can also be involved in application architecture to collaborate on solutions and more quickly identify weaknesses. While the dark side may tempt you to keep functional areas separate to avoid conflict, a culture of sharing responsibility through collaboration is crucial for a successful DevOps culture shift.

Security Concerns with DevOps

Security can be a touchy subject within organizations. There are often strong opinions that govern who has access to what. The “what” often includes much more than usernames and passwords, even extending into operations. However, infrastructure and software systems are not things to be kept secret within modern LEAN or Agile organizations.

The notion that certain systems and processes should not be accessible or even visible within an organization has crippled many a fast-moving development team. There’s nothing worse than having a business-critical deployment stalled because a security process prevents senior developers from being able to diagnose errors in production.

Likewise, preventing developers from testing configuration changes in Development or QA environments has saddled many an application with performance burdens that have workable solutions! If you hear developers shooting down ideas because they require getting approval from a Darth Maul-esque infrastructure manager, you might have a security culture that’s working against you.

Tackling this issue requires senior leadership to trust their teams and carefully consider how to maintain security without obscurity. Developers should be able to see what is being deployed across all environments and have full access within development environments. They will learn about the infrastructure impact of application architectural decisions and produce higher quality software, while the infrastructure team will better understand the process that leads to infrastructure requests – often to improve application security!

Audit logs can provide accountability and help prevent repeated mistakes and enable root-cause analysis for fast resolutions. Trust and transparency are crucial to enabling self-service infrastructure.

The Scope of Automation

Automation is the obvious goal of many DevOps initiatives, but a common pitfall is failing to thoroughly document all inputs and outputs that a complete automation would handle. A common indicator that gaps exist in your deployment pipeline is when you hear statements such as “oh we can’t do that, so-and-so is on vacation,” or “we’ll need to schedule a resource to deploy that.”

Referring to the concepts of transparency, continuous improvement, and accountability, asking hard questions such as “is this as automated as it can be?” are extremely valuable. Recurring meetings where anyone can challenge the completeness of an automation can prove invaluable to an organization’s early DevOps efforts. Our experience shows frequent touchpoints early, followed by regular tie-offs with members of operations and development can smooth the path to success.

DevOps Source Control

Another common issue organizations encounter when moving towards fully automated Continuous Integration/Continuous Deployment (CI/CD) lies within the existing source control and code branching strategies. If you’re not already using Git, now is an excellent time to consider the switch. Many legacy source control systems cannot handle the complex branching and merging required to support automated delivery and concurrent development.

Consider revising “de facto” branching strategies, like developing against the primary branch. Features can and should be developed independent of one another. Leverage the magic of Git’s baked-in merge resolver and don’t fear the merge. After all, fear leads to anger, anger leads to hate, hate leads to your developers suffering through antiquated processes.

A great way to know if you’re doing it right is when a “two-pizza team” can build, test, and deploy a single feature or hotfix in one day or less. There are frameworks like GitOps which can assist with designing a supportive branching and merging strategy. Source control should never hold back a deployment or complicate a hotfix.

At the end of the day ask yourself, “Are our DevOps platform, tools, and methodologies making life better for the developers and engineers?” If teams are spending considerable time deploying code, butting heads with other teams, or feeling like work was better before DevOps, then perhaps you’ve accidentally strayed too close to the dark side.

There’s an easy path back to the light, though! During Scrum Master training, a coach loved to intone “Don’t do dumb stuff. Any of this advice is just a recommendation, and if it’s not working for your organization, remove it.” It’s better to fully automate smaller pain points than to try to do it all in one go.

As you build your DevOps platform, focus on bringing teams together, avoid the pitfalls, and contact us if you need additional help.

David Pratt is a Senior Architect at RevGen Partners, with years of experience tailoring digital solutions to our clients’ unique needs.