Thoughts on DevOps and Continuous Delivery

When I think about DevOps and Continuous Delivery I am reminded of a quote by William Gibson:

“The future is here. It’s just not evenly distributed yet.”

There is no doubt in my mind that these two concepts will become mainstream similar to how Cloud Computing has become so within the software industry. It is just a matter of time before they get “evenly distributed”.

Connecting with DevOps and Continuous Delivery
These concepts fit well with some of my fundamental beliefs about software development:

• First, I believe the time between when we conceive of and create software and when a user can safely use it should be measured in days or a few weeks, not months or years. Ideally we write code today and you safely use the software tomorrow. Ideally. This time period between software creation and user benefit is known as cycle time. Anything that adds a delay should be actively worked on to improve or resolve. I admire companies like Etsy and Flickr in how they have already solved for this quite well.
• Second, I believe that it should be one united team that accomplishes the entire development/deployment software lifecycle. There should be no walls to throw software over to another team. Software should flow smoothly through one united team like cool clean water through a pipe until it arrives out of the tap for the user.
• Third, I believe you should employ feedback loops to ensure you build the right software. The definition of what is right has nothing to do with what is being asked for but rather what is actually used in the end. That is the feedback that matters to me. How many software companies can tell you exactly who is using their software, how often and which parts? The business side may define the right software as what sells and I recognize the importance of that (I appreciate the paycheck!) but although that is necessary, it is insufficient. Implementing effective validation, analytics and feedback loops is core to achieving software that people actually use. Stop reading this blog post right now and read The Lean Startup then come back and finish reading this.
• Fourth, I believe you should know what quality means to you and build the right software right. W. Edwards Deming stated that you should “build quality in.” You can argue about what exactly quality is (see Zen and the Art of Motorcycle Maintenance) or what it means within your particular environment, but however you define it, it should be there from the beginning and throughout. I do believe though that if you truly know your user you will then know what quality in your context should be.
• Finally, I believe any team worth its salt that produces and manages the operation of software should focus on relentless improvement (the true meaning of Kaizen). A team should be continually pushing to get past the OK Plateau where good enough has become acceptable. I do not want to be on a team where mediocre or “middle of the pack” is sufficient. Not moving forward to Continuous Delivery and Dev/Ops is a choice to accept an OK Plateau.

Achieving Continuous Delivery
Where I work we have a tough problem. We want to achieve Continuous Delivery… one might say we are obsessed (at least I know I am) to get us there. The challenge is how far we have yet to go. We deliver enterprise software both in the Cloud and on premise and it is no easy road to Continuous Delivery for us. We are not a startup where we can establish these practices from the beginning. Our release timelines are still measured in months, not days, or even weeks.

The good news is that we have already started to do automated build, integration, acceptance, regression, and deployment (to test environments) so we are well on our way. However, our automation is still in the early stage and this system is not robust nor complete enough to get us there yet. This great post by Ranjan Sakalley captures well how test automation is core to achieving this.

Because the challenge is so difficult, it will make the accomplishment that much sweeter. Reading the two books “Continuous Integration” and “Continuous Delivery” has definitely helped to understand how to approach the problem. This post by James Betteley also gives a good overview of the Continuous Delivery concept. Definitely these books and postings should not be taken as gospel and should be assessed from multiple perspectives but that does not diminish the value they represent. If you are wondering what tools out there can help, Paul Duvell (the author of “Continuous Integration“) has a great post on Continuous Delivery tools that can help teams on their journey.

When it comes to Continuous Delivery, I draw inspiration from Flickr and others who lead the way. Often I hear statements like “This is only for consumer facing sites like Facebook” or “our customers cannot adopt our software that quickly.” I totally disagree with those statements. I believe this is not just for consumer software products and our customers WOULD adopt our software if they could do so safely without major disruption to their business.

Understanding DevOps
DevOps is quickly gaining momentum in the software/IT industry and we have just started our DevOps team where I work. It is early days still for DevOps and this post by Dan Ackerson captures the adoption chasm that it still has to cross. I especially like the graphic on how to explain DevOps to the CEO. DevOps is compelling but it does have its critics (see here and here). With any new concepts like DevOps and Continuous Delivery it is important to take a balanced view and not get too caught up in the hype.

DevOps is usually positioned as the way to solve a poor relationship between Development and Operations. I believe it will help but DevOps should not be seen as a silver bullet. Luckily we have a good relationship between our development and operations teams so we see DevOps a bit differently. For us it is simply the team that defines success by achieving continuous delivery and all the good that it entails.

I believe that the road to DevOps and Continuous Delivery starts by defining what those two terms mean to you in your own environment. Do not take anything put out there as gospel and think critically for yourself. Then follow up this with the question “Why?” Why should you bother to invest the energy and time it takes to change and achieve this? To me it is all about getting valuable software to customers faster in a safe manner where they can confidently use it and we can get consistent valuable feedback on that use. It is about establishing the right culture and removing barriers between teams to get there. Only after you have established your “Why” should you move on to “What” and “How” to get it done.

Agile & Fly Fishing

This may be a bit of a frivolous post but this past summer I spent a day fly fishing with my son at a local lake and afterwards on the drive home I began to think of a few similarities between Agile development and fly fishing.

Similarity #1 - Preparing is important but events will require you to make adjustments

Preparing for a fly fishing trip is important. You can choose to just jump in the truck and head off to the lake but chances are you will have a better experience with some prep work. I like to review my gear and ensure my reel is in good working order, sharpen hooks, replace the leader line on my floating and sinking fly lines, do some online research on the lake and review recent fishing and weather reports. I usually arrive at the lake with a plan but I am ready to change on the fly as circumstances dictate. A good example of this is if there is a hatch (new bugs emerging) as that will change what type of fly I may use and the fishing method I employ.

The important point here is that planning is important but you have to be ready to change that plan on site and throughout the day. There are many factors to a day fishing on the lake and this is similar to how a software project can go. It is possible to over prepare and waste time planning your fishing trip or software project thinking of this possibility or that possibility. It is better to be well equipped and ready to change the plan as new information presents itself. Waterfall methodology certainly doesn’t work well for fly fishing (although the actual location where a waterfall spills into a lake can be a great fishing spot!).

Similarity #2 – The customer will change their mind

I may get some heat over comparing customers to fish but there are some parallels. Often a fly that I am using or a fishing method is working great and then all of a sudden (at least that is how it feels) it stops working and the fish stop biting. The change could be for any number of reasons such as the temperature of the water, a new hatch, wind or rain, or simply the time of day. The key point here is that the fish want something different and I need to adjust to stay on target. The enjoyable challenge of fly fishing is that I need to use the cues from the environment as I cannot simply pick up a phone and ask the fish what they now want. Customers of software also change their minds as their business environment changes. However, a nice advantage to software development is you can ask your customers, gather feedback on the market, and then adjust your plans (alter your backlog) accordingly.

Similarity #3 – You can only do so much at one time

Anyone who has ever caught two fish at the same time (a double header) knows that things can get a bit crazy. Imagine catching fish on two rods and having a boat and a young son to manage all at once. Things can go awry quickly! Agile projects can be like that when too much is taken on at one time (easy to do when there is a demanding customer at the end of a line!). A number of Agile methodologies advocate limiting the work in progress so that you can focus and build things right. Lately this has increasingly occupied my thoughts around Agile as we try to get more done in our projects by focusing on less.

There is one notable difference between fly fishing and Agile software development and that is while I am fishing I am definitely NOT thinking about work. Best of luck to everyone out there catching fish OR customers by being Agile.

Seeking Alpha from Agile

Why do we focus on becoming an agile organization?

We are not seeking incremental improvement. We are seeking to achieve astonishing results by adopting Agile. We are seeking knock-your-socks-off improvement.  We do this first because it will create a better place to work and second because it will enable us to significantly add value for our customers.

Our goals around building software are simple. We want to build the right stuff, build the right stuff right, build more of the right stuff right faster than our competitors and become a better team through it all. Getting there is very hard work, however. Our efforts to improve fall into four key areas:

• Precision:
  • Build the right stuff
• Quality:
  • Build the right stuff right
• Quantity:
  • Build more of the right stuff right faster than our competitors
• Team:
  • Become a better team

Basically we want to excel in these areas so that we can truly enjoy our jobs and kick our competition’s you-know-what. One way we get there is by relentlessly pursuing our goal to have all of our teams become high performance agile teams.

We are seeking an exponential Alpha from Agile.

In the world of financial investing, Alpha assesses the real return on investment (ROI) when the risks taken are factored in. It can be a bit technical (in fact it is a part of technical analysis) but the important thing to understand is that it is intended to get the true measure of performance for an investment against a market index.

When it comes to our effort in becoming truly Agile we are all seeking a major return on investment for the risk we are taking to change our organizations. We are seeking an exponential Alpha from Agile. Why else would we be so steadfastly struggling to change, to get better at changing, and adapt to this new way of working?

Wait, our competitors are adopting Agile too!

Are they really?  Perhaps they say they are adopting Agile but are they doing what is necessary to put in place the cultural changes needed to achieve the desired results? Are they willing to stick with it even after a few bumps along the way? Have they made the organizational changes necessary for every step from concept to customer delivery and beyond? Is Agile still just something in their R&D organization?

Maybe they are actually just following The Half-arsed Agile Manifesto instead.

Measuring High Performance Agile Teams

You might now understand what a high performing agile team is and what the key attributes of a team at this level are. You may even have some plans and strategies to get your team there. Now how do you measure whether your team has truly achieved a high performing level?

First, remember that most measurements are arbitrary. This means that people make them up and then other people agree to use them. For example, a metre (or meter) has been defined as the distance traveled by light in free space in 1/299,792,458 of a second. This was decided by people for people and widely adopted around the world. The point here is that the measurement you define for your teams need to be well understood and fully adopted by all.

Second, the metrics you choose need to be for the benefit of the team. Getting to high performance is not something you do to your teams and measure management’s impact on making it happen. Instead it is about equipping your teams with the information they need to make better decisions on their own journey of continuous improvement.

I feel that Agile should be about getting the right high quality product to market faster than your competitors while fostering a continually improving team. There are four dimensions to this worth measuring:

• Precision: “measure if you are building the right stuff”
• Quality: “measure if you are building the right stuff right”
• Quantity “measure if you are building more of the right stuff right than your competitors”
• Team: “continually improve your team”

The metrics you use should help your teams ask the right questions and make better and faster decisions in each of these four areas.

So what kinds of things can you measure? Here is a quick list of some that you may consider using:

• Velocity
• Acceleration
• Resource Utilization
• Backlog prioritization
• Burndown charts
• Task hour burndown
• Story point burnup
• Estimates vs actuals
• Feature validation with customers/users
• Usability tests (i.e., time on task)
• Defect breakout charts
• Independent security audit
• Adherence to doneness/acceptance criteria
• Adherence to checklists (checklist item burnup)
• Adherence to standards
• Peer code reviews
• Test coverage
• Running tested features
• Team member peer reviews
• Trust level within the team
• Capacity over time
• Anonymous surveys

The point of this post is not to tell you which metrics to use for your team. You may want to get educated on some or all of these and seek out other metrics. Decide for yourself which ones have the greatest value to your teams. Here is a link to a great presentation on Agile metrics including many of the above and more (with pros and cons). If a team is serious about getting to high performance they will learn about these metrics and others.

If there are other metrics that you have found valuable or links to resources that can help teams choose metrics please add a comment and share. Finally, remember that if the metrics are not creating opportunities for teams to ask interesting questions that lead to better decisions then gathering them might just be busy work.

R&D Open House

Recently my team hosted development partners at our office for our first ever R&D open house. The event followed a locally held independent partner conference known as the “Third Party Advantage Conference” where our partners get together to showcase new add-on products that extend the functionality of our core software. I have attended the conference twice now and it is a great experience where we are regularly astounded by what our partners have created.

Our open house event focused on sharing the information and tools our partners need to be successful with integrating to our product’s upcoming cloud/web enabled release. We are in the midst of a transformation of our product to new technology and educating and supporting our third party ecosystem is crucial to our success. Our joint customers will benefit if both our product and our partner’s products are in synch technologically and we know that a customer’s resistance to upgrade will be significantly reduced if their entire solution has been updated.

Our goal with hosting the R&D Open House is start a new journey of openness and collaboration with our development partners.

We are starting to tear down walls and remove barriers to communication so that we can progress as one extended R&D community around our product. The benefits are not just for our partners, however. We benefit greatly by learning how we can enhance our product and software development kit (SDK). We also get a motivational boost working closely with those who focus so much of their business on our product.

So what happens at a R&D open house? Ours had an opening welcome presentation from myself, a walkthrough of our partner support resources, instruction on how we automate acceptance, performance, scalability and regression testing, an overview of our usability guidelines for UI screens, a walkthrough of our SDK including UI Designer, and porting tools, and a presentation on how we work with our customers to evaluate the usability of our product. The partners also had 1:1 sessions with our developers working on the product to ask questions and work to resolve concerns in a few cases. We toured our office and explained our Agile organization and how we go from concept to completion.

At the end of the day we asked them to provide feedback focused on whether they would recommend the event to other partners.

With a rating from 1-10 with 10 being the highest we asked them to review us. So far the ratings put at us 9/10 with a number of ratings of 10/10 for the event. I am incredibly proud of my R&D team for putting this event together and making it a success. We’ll continue the spirit of openness now with frequent updates to our development partner website, a new ideas site for our SDK, a change to how we do support to capture knowledge better for other partners, and a regular webinar/Q&A session for partners to attend (especially those that could not physically travel to our offices).

So why blog about it? I know that other organizations could benefit from a similar idea. R&D teams are usually so busy we resist taking time away for something like this. That is a mistake. The motivational boost and knowledge sharing more than makes up for the time invested. Consider hosting your own R&D open house for your partners or customers and you may be pleasantly surprised.

Attributes of a High Performing Agile Team

In an earlier post I talked about what defines a high performance agile team. In this post I will attempt to outline some of the key attributes or characteristics that I’ve noticed high performance agile teams have. Some attributes depend on the context for a team. For example, what is needed for a team in a large enterprise to be high performing may be unnecessary for a team in a startup or smaller organization. A team of 3-4 people may not require the things that a 7-10 person team may need to become high performing. If your team is employing Scrum, Lean, or Kanban it could have an effect as well. After seeing many teams there are some common attributes that stand out:

The team has a vision of where they are going

The best teams have a clearly defined vision and shared purpose. It is difficult for team to act as “One Team” and pull together if they lack a common vision. Many teams create a team charter where they capture information around their goals as a team, how they work together and their expectations of each other. I believe it is highly valuable for a team to also establish an identity by naming themselves, having a team page or site and their own physical space, if possible.

Everyone on a high performing agile team trusts their teammates

One of the primary reasons it can take a while for an agile team to get to a high performing level is it takes time to build up trust. Speed and trust are directly connected. In a high trust environment there can be a free flow of ideas, constructive criticism, and open communication enabling continual improvement. In a low trust environment things slow down to a crawl (think about going through airport security, a notable low trust environment). Understanding the relationship between speed and trust is important and a great resource is Stephen Covey’s book.

Top performing agile teams have a culture of continuous improvement

It is a wonderful thing if team members like and trust each other and that is necessary but not sufficient to get to high performance. If your team is not focused on relentlessly improving (Kaizen) they will stagnate and never reach a high performance level (i.e., they get stuck on the OK Plateau). The best teams have an internally driven zest for improvement and look for any way to be more effective and efficient. They will map out an action plan to get there. I believe that in any project there are two key deliverables at the end, the product or service you are creating and ultimately a better team. Retrospectives, for example, can be an effective tool to help a team discuss and focus on continual improvement. Generally these are held at the end of an iteration but could actually be held at any time. I’ve seen through experience that teams that hold meaningful retrospectives, capture the areas they want to improve, prioritize them, and focus on the changes they have will to improve see the best results over time.

High performing agile teams are skilled

Talent matters. A team can have trust and a shared desire to improve but without skills, knowledge and experience it will take longer to get there. Every individual in a high performing agile team makes it a priority to improve their knowledge and skills for the sake of the team. If there are skill or knowledge gaps the team will identify those and map out an action plan to close those gaps. Experience will come in time, if it is lacking, but the best teams also reach out to others who have the experience for their guidance and wisdom.

Your teams will not get to a high performing level if you keep moving people around

Another critical attribute is having stable teams. People need to work together for extended periods of time to build up trust and if team members are coming and going the clock keeps getting reset. Often companies route people to the work (we need Dave to work on this feature…) rather than route work to the people (we have this high value work for the team(s) to pull in next). Keep your teams together and give them time to blossom.

High performing agile teams meet talk regularly

What, more meetings? How can that possibly help? The fact is that high performing teams do meet regularly, many of them every single day. The difference is that their meetings are highly focused, efficient, productive and relevant to all attendees. Actually these meetings may be better described as meaningful conversations. In our team we have daily standup meetings for 15 minutes to connect with our teammates. Whenever possible these are in person face to face. We are a distributed team though so some teams have people dialing in and using video conferencing technology.

Top performing agile teams validate their work with their customer(s) regularly

Sometimes I hear that becoming agile is all about speed to market but first it is about building the right software for the customer. It does not matter how fast you build it if it off target. The most successful teams will show their work regularly to their customers as it is coming together, not just at the end. In Scrum this can usually takes the form of a sprint review with customers present. In large enterprises the product owner may serve as a proxy to the customer, accepting the work as it completes. This may be fine for your team but high performing agile teams will take any chance they can have a real customer validate their team’s work.

High Performance Agile Teams

So you are on your Agile / DevOps / Continuous Delivery journey with either Scrum and/or Kanban, Lean, TDD, BDD, or another current software engineering approach. Do you feel like your team has achieved a high performing level now? If so, great and drop me a comment to share your secret sauce because we want to hear about it. If not, or if you are not sure, read on.

High performing Agile teams out-produce other teams by a wide margin.

This has been my direct experience. Be aware that helping an Agile team get to a top performing level is a journey which can take time and generally cannot be rushed (this depends on where the people in the team are starting from). There is no silver bullet solution to get there overnight, although there are techniques to speed up the cultural transformation. If you are new to Agile, or have adopted Agile and you are not fully satisfied with the results, then hopefully some of these techniques gained through our experience will help.

Some background: I have been leading Agile teams for around five years and managing complex enterprise software projects with distributed teams over a much longer period. Currently I am responsible for eleven Agile teams with team members concentrated in two locations (Vancouver & Chennai) with a few other team members in a number Canadian and US cities around North America. Right now I am focused on driving our transformation with Dev/Ops and Continuous Delivery and it is certainly exciting times.  However, I am just a small part of the team and not alone in this endeavor as our team includes five senior R&D managers and a talented Business Analysis team (aka Product Owners). Our challenges are the real deal and millions of dollars of engineering are at stake.

So what defines a high performing level for a software engineering team? I wish there was an a+b=c formula for this but unfortunately it is not so straightforward. That is not to say there are not measurable aspects of a high performing team (the key one being the quantity of quality software they produce, also known as the primary measure of progress). Though it is not easy to describe exactly what a team at a high performing level looks like, I do know it when I see it and I believe so will you. The following definition is provided with the understanding that there is no perfect definition:

“A high performance Agile team is a committed team that has the right people, has been effectively empowered, has established trust and works at a sustainable pace to deliver quality software of a quantity that reflects a consistent high velocity factoring in influences such as capacity and customer support.”

Whew, that’s a mouthful. Let’s break that sentence down to better understand it:

• Has the right people means that it all starts with hiring the right people if you are building a team from scratch or possibly making the tough changes needed within an existing team to ensure the right people with the right skills and attitude are in place.
• A committed team that has been effectively empowered means that every single team member is engaged and committed to the team’s success and accountable to their peers, and the team is self-organizing without command and control management getting in the way. Team members pull work in this model rather than having it pushed down on them.
• Has established trust is a critical element as trust is at the core of a high performing agile team. Without trust there can be no open communication and without open communication there can be no team improvement over time.
• Works at a sustainable pace to deliver quality software means that the team does not race to get things done leaving a wake of destruction in their path only to burn out, crash, rest and repeat. The team needs to do things right first (precision before speed) and work at a pace that every single team member can sustain. Many teams are now following a test driven development approach (TDD) to better ensure quality is built-in as the software is developed.
• Reflects a consistent high velocity is where things get a bit tricky. Whether you use velocity or track productivity with other metrics it is is important to be consistent and sustainable. However, one team’s high velocity may be an impossible level for another team to achieve. The key aspect here is that a team measures their output or velocity and over time it does not swing widely from high levels to low levels (i.e., low standard deviation). Velocity may start low as a team does the right stuff right and increase incrementally over time. A high performing team knows what they can commit to and their velocity will generally be consistently high.
• Factoring in influences like capacity and support means that there will always be things like vacation, illnesses, unexpected support work, major company events etc… that can and will impact a high performing team. The best teams learn how to deal with these situations and organize themselves to continue to produce software (albeit in a reduced capacity) despite the impact.

As an R&D Director I have learned that working with a high performing Agile team is like working with a highly experienced and competent professional. They know what they are doing and they’ll let you know what they need to be successful in any project. In many ways you’ll feel like you are being pulled by a team at this level rather than having to push them. Your biggest challenge is ensuring they are well equipped, fully supported and have a worthwhile challenge to focus their energies toward.

There are some characteristics of a high performing agile team that we have come to recognize. I have written a blog post entitled “Attributes of a High Performing Agile Team” to attempt to capture some of the key ones that have the greatest impact on a team becoming high performing.

So how do you get your teams there? A better question is how can you help your teams get there themselves and stay out of their way? I wrote a post entitled “Getting Agile Teams to a High Performing Level” that outlines some steps that could help others get there.

Finally, how do you know your team has attained the level of a high performing agile team? I believe metrics are important but I have found that continually reviewing the actual software being produced is better than any other proxy measurement. However, I wrote a post entitled “Measuring High Performance Agile Teams” to identify some aspects of an agile team that can be evaluated quantifiably.

Remember that we might struggle to define a high performing agile team but I am certain that when you get there you’ll know it.