If you're considering how to build a DevOps team the best way possible, this one's for you.
This post sums up the advice I give people building DevOps teams: CTOs, VPs of R&D, Team Leaders, and more.
You'll get a step-by-step path from philosophy and mission to roadmap, team shape by stage, sizing, and hiring (or bringing in outside help), plus a short FAQ.
TL;DR
I'll give you the bottom line.
These are the steps for building your DevOps team:
- Polish your DevOps Philosophy
- Understand the DevOps Lead responsibilities: Product, Project, People, Service, Architecture
- Define your Team's Mission Statement
- Set useful DevOps Goals & Practices
- Set Guiding DevOps Principles
- Define the DevOps team's roadmap & strategy
- Set a definition of done
- Common DevOps teams pitfalls
- Pick a team shape that fits your stage
- Calculate your team's required capacity
- Hire the right DevOps Engineers
You can also apply these to existing DevOps teams and level-up your organization.
Let's get started!
Polish your DevOps Philosophy
You must first understand why organizations need DevOps, and how it can work in practice.
DevOps is an enabler role.
It's meant to enable the developers to build, improve, and take ownership over the system.
Let's break it down:
- Build = Create something new. Could be a new microservice, a new database, or a new monitoring dashboard.
- Improve = Introduce a change into something that exists. Could be fixing a bug, changing a database schema, or changing an alert threshold.
- Take ownership = Take charge of problems that arise with what you built and improved. This means when something needs improvement, the owner is the one who does it.
The above are what the developers should do.
Ownership includes on-call: the team that builds a service should be the one paged for it. DevOps owns the platform underneath it, plus the tooling that keeps on-call manageable (actionable alerts, dashboards, runbooks).
Now, DevOps should enable it.
Note: enabling is not giving full permissions and letting them have fun.
To enable is to give developers what they need to build, improve, and own the system.
But do it safely, and in a way that points the developers' energy in the right direction.
Understand the DevOps Lead Responsibilities
You have 5 hats as a DevOps Lead:
- Product (Platform)
Your clients are your company's developers.
Provide them with tools, knowledge, and automations.
Tools = Polished automations.
Automations = Automated knowledge.
Knowledge = Hard-earned information & insights.
Understand their requirements, and use Tools, Automations, and Knowledge to fulfill them.
Understand why they want what they want: Is it because of an underlying issue with the system? If yes, solve it before building anything.
If a tool or automation will save time, consider building or implementing it. - Service
Don't let your developers wait until you do something "the right way".
Sometimes they need immediate help to complete something.
Help them first, and invest time later in automation and tooling. - Project
Managing your DevOps team requires managing the work it does.
Turn the team's philosophy into a mission.
Turn the team's mission into goals.
Turn the team's goals into a roadmap.
Turn the team's roadmap into tasks.
Prioritize the tasks.
Set simple roles and responsibilities.
Hold your team members accountable to progression.
Make it easy to inform relevant team members on updates.
Make it easy to consult with a teammate on a subject of their expertise. - People
Each DevOps team member enjoys some types of work more than others.
One enjoys sharing knowledge, another enjoys being of service, and some enjoy building tools.
Different teammates are also interested in different subjects: Monitoring, Infrastructure, CI/CD, etc.
A team member that's working on what they enjoy is more fulfilled and productive.
Strive to overlap each teammate's goals with the company's goals. - Architecture
You have 2 architectures to worry about.
1) The company's product architecture (built by the developers).
2) The DevOps platform's architecture (built by the DevOps team).
Help the developers understand and control the application's effect on the infrastructure.
Enable the developers to make informed decisions by providing context.
Finally, build the platform to support the developers requirements.
At every step, only limit decisions that damage the company.
Examples: High cost, Stability impairment, Restricts observability, etc.
Define Your DevOps Team's Mission Statement
Here's one for you:
"Enable the developers to build, improve, and own the system".
It's pretty minimal, so it's going to help your team stay focused.
A healthy sign it catches on: When the team debates a decision regarding a task, they ask "Does it help us enable the developers to build, improve, or own the system?".
That's when you win.
Set Useful DevOps Goals & Practices
Your company's success will be determined by its speed and its product's quality.
To make it happen, remember that programming is science, not maths. I'll explain.
People used to think programming would be a mathematical discipline.
They thought programmers would write functions and prove them mathematically.
That's not what happened. Programming is a scientific discipline.
You write code, you test it, and you assume it's good until a test fails.
In essence, you're experimenting.
You might ask: "wtf? how's this related to setting useful goals?"
The answer is that the first thing you want to enable is running experiments easily.
So here are some useful goals:
- Developers can easily test their code in a consistent manner
- Production and testing environments are identical (Production will benefit from the quality of the tests)
- Developers can easily collaborate
- Developers can understand the state of the system and the impact of changes
Let's translate those goals into smaller goals or practices that will achieve the goals:
- Developers can create a testing/production environment with "One-Click"
- A continuous integration process lets developers collaborate by agreeing on the current, up-to-date version of the system
- Auto-create dashboards and alerts for new services
Set Guiding DevOps Principles
Some useful DevOps principles that help save your team time, improve the speed of delivery, and keep the system healthy:
- PoC before doing things "the right way"
- Make it work, then make it better
- The entire system should be fully recoverable from Git
- Use tools with a big community and well-documented interface
- Equip key-developers with DevOps knowledge to be the first point-of-contact for their team members (super users)
For more principles with examples, see The CTO DevOps Handbook.
Define the DevOps Team's Roadmap & Strategy
Roadmap = Goals * Strategy.
Once you set the goals (as mentioned above), you can start prioritizing.
#1 - The DevOps Categories
Enabling developers requires a DevOps Engineer to handle and enable the following:
- Provision infrastructure
- Deploy workloads
- Monitor the system
- Recover from issues
- Scale up and down
- Track & test changes (Codebase Management)
- Secure the system
- Store & retrieve data
- Configure the system
#2 - Examine each goal through each category
Every DevOps goal you set should be examined through the lens of each category.
Together, the categories cover every aspect of building, improving, and owning a software operation.
Let's do an example:
- Goal: Create a "One-Click Environment Automation"
- Categories to address:
- How should its infrastructure be provisioned?
- How should its workloads be deployed?
- How should its metrics and logs be sent, stored, and queried?
- ...
#3 - Strategic Principles
- Spend at least 50% of the team's capacity on DevOps goals as soon as you responsibly can:
Support the developers and teach them how to self-support to get there - Easy to modify > Perfect:
When time forces you to do something imperfectly, do it in a way that's easy to improve later - Prerequisites first:
Codebase Management -> Infrastructure -> Deployment -> Configuration -> Data Management. - At least moderate foundations quickly, reinforce later, but never weak foundations:
If something is a foundation (like infrastructure) and early on you lack boilerplates or experience with it, don't settle for weak foundations. But don't over-invest in strong ones if that takes too long, either.
Instead, moderately invest in the foundation, and revisit it later.
Set a Definition of Done
Think of it as the checklist a component must pass before you call it production-ready.
Ask the following questions for every single component in your system:
- Monitoring: Are there metrics, logs, traces, and alerts setup in an actionable way?
- Availability: Is there a mechanism to keep it alive during incidents?
- Resiliency: Can it recover from an error quickly?
- Recovery: Can it be fully restored to a previous state?
- Testability: Is it possible to test changes to it?
- Deliverability: Is there a process to release changes to it?
- Persistency: Will its data persist if the system is hindered?
- Integrable: Does it have a consistent and predictable interface allowing integration with it?
- Security: Is it accessible only by the parts of the system that absolutely need it?
- Dependencies: Are its dependencies fully tracked and managed?
Common DevOps Team Pitfalls
- Pitfall: DevOps work blocks developers work
Indicator: Developers need to wait for DevOps team changes to complete before continuing work
Cause: The DevOps team doesn't utilize its own practices (AKA shoemaker's son always goes barefoot) - Pitfall: Only support developers and maintain the system
Indicator: No progress on any DevOps goal or task
Cause: Either there are no clear DevOps goals or repeating developers requests haven't been automated - Pitfall: Adopting 'Best' practices instead of 'Suitable' practices
Indicator: Introducing methodologies and adhering to principles that go against the company's goals
Cause: Prioritizing methodology over company goals, usually because of a disconnect from the company goals or due to lack of DevOps experience
Pick a Team Shape That Fits Your Stage
There's no single right structure, but these rules of thumb fit most startups:
- Fewer than ~10 developers: usually no dedicated DevOps hire yet. A developer who enjoys infrastructure, backed by a consultancy or a fractional DevOps engineer for the foundations, is often enough.
- Around 10 to 20 developers: time for the first DevOps engineer. Hire someone senior enough to set direction alone.
- Beyond that: a small platform team that treats the internal platform as a product, with the developers as its clients.
As you grow, you'll choose between a central platform team and DevOps engineers embedded in product teams. Embedded engineers move fast but build one-off solutions; a platform team builds shared paths but can drift from developers' needs. A common middle ground: a platform team owns the shared platform, and each product team has a super user.
Outside help makes sense when you need foundations fast, can't justify a full-time hire yet, or are still hiring. Insist that everything lives in Git so your team can own it. Here's how to choose a DevOps consulting service.
Calculate Your DevOps Team's Required Capacity
Required DevOps Capacity = (Scale * Complexity) / Leverage.
- Leverage = Level of DevOps Engineers * Company Resources * Team Focus
- Scale = Number of instances of each component * Number of people in the organization
- Complexity = Number of components * Number of teams
Hire the Right DevOps Engineers
The types of DevOps Engineers:
- Barrels vs. Ammo
~ Ammo = People who can complete tasks but won't initiate them.
~ Barrel = People who understand what tasks are needed next but won't complete them.
Interviewer Tip: During the technical interview with a candidate ask about past DevOps accomplishments, and ask how the tasks were created and who did them. - Aspiration-Oriented vs. Prevention-Oriented
~ Aspiration-Oriented = Has goals, positive feedback encourages and focuses them while negative feedback discourages them and kicks them off track.
~ Prevention-Oriented = Avoids problems, positive feedback makes them lay back and reduce capacity while negative feedback focuses them and keeps them on track.
Interviewer tip: See if there's a common theme for projects the candidate did in the past. People focused on security tasks and deep attention to specific details are more likely to be prevention-oriented, while people who initiated many projects spanning multiple (DevOps) categories are more likely to be aspiration-oriented.
Working with service providers:
- Set clear desired results
And let the DevOps service provider assist in exploring the goal and provide perspective from other companies. - Expect transparency on progress
And judge it against the latest plan. - Expect clear planning
And make sure it has clear goals, takes into consideration the risks, and has a strategy that adheres to your DevOps principles.
Frequently asked questions
How many DevOps engineers do I need?
It depends on scale, complexity, and leverage. As a rule of thumb, one DevOps engineer can support roughly 10 to 20 developers when developers own their services and most work is automated. Less automation or more teams means more capacity; check it with the formula above.
What roles make up a DevOps team?
Early on, one or two generalist DevOps engineers, with someone wearing the five DevOps Lead hats. As the team grows, people specialize by category (infrastructure, CI/CD, observability, security), and some teams add a reliability (SRE) focus.
Should DevOps be a separate team?
Only as a platform team that enables developers to own their services. It shouldn't become a team developers hand work over to, because that rebuilds the wall DevOps was meant to remove. If developers wait on DevOps tickets to ship, you've hit the first pitfall above.
What should be in a DevOps team charter?
The mission statement, the goals and principles that follow from it, and a definition of done. Add clear ownership boundaries: what the DevOps team owns, what developers own, and who is on call for what.
Should I hire DevOps engineers or outsource?
Hire when there's steady, full-time DevOps work and you can attract someone senior enough to work without guidance. Use a consultancy or fractional engineer when you need foundations quickly, lack the expertise to judge candidates, or need to bridge the gap until a hire lands.
Summary
We covered a lot in this one-pager, and still left plenty out.
If you take away 1 thing from it, let it be this: A simple DevOps team mission statement is the most significant thing you need.
It sounds simplistic and abstract, but without it there's no guiding principle for what DevOps in your organization should look like.
Hope you enjoyed, and send me an email at michael@meteorops.com if there's anything else you'd like to see in here.
