A long time ago I was working at a software company that was undergoing rapid expansion. We were racking and stacking physical servers almost every day. The lead-time on most of the hardware was at least 2 weeks and our software engineers were always complaining that this was too slow. I remember thinking it being a little unfair to complain as I could usually turn around the setup of a server in under a day once everything was delivered.
Things continued like this for a while until one day we had the bright idea to buy a bunch of extremely large servers and then provision virtual machines for the developers as they requested them. This meant that we didn’t need to wait for hardware delivery and could spin up new servers with the click of a button. By working with development to provide a business justification we got approval to spend our entire yearly budget in one go. We setup VMware back when people had doubts that a ‘virtual’ machine could be trusted to act like a physical one.
As time moved on we made a conscious effort to help the software developers as much as we could. We were a software development company so this seemed like common sense. Formal requests for virtual machines turned into self-service portals that would allow developers and QA to provision environments without waiting on IT. On the operations side of the fence we moved behind a layer of tooling and this seemed to scale quite efficiently.
One of my fondest recollections of that time was wandering into a room full of QA and Developers and watching them draw up a grid of operating systems vs language to plan the test coverage for their next release. By the end of the meeting they had about 10 different operating system versions down the left hand side of the whiteboard (a mix of 32 and 64 bit variations of Windows on Desktop and Server) and around 15 different languages (most of the European countries and some of Asia). To test everything would require 150 virtual machines for a couple of days. It was a joy to see the group taking the initiative to plan this out. Shortly after the meeting the development manager sent out an email to his group telling them to hold off on creating any new VM’s while QA did their thing. It felt like we the company was making data driven decisions and we were now getting the most optimum value from our time. More importantly IT was no longer the bottleneck.
This new world wasn’t without issues of course. Over subscription of our physical hardware caused constant complaints about VM performance. We had to put a lot of effort into lifecycle management and policing what virtual machines were left on. We also had a bunch of issues with snapshots taking up too much disk space and VM’s dropping off the domain. But overall I remember this as a bit of a golden age, we had mitigated a lot of the constraints and were helping developers release their products more efficiently. We were also talking to the business while measuring and using that feedback to fix issues that were getting in the way of developers releasing code.
The company continued to grow. We were running thousands of virtual machines and most products started moving to online delivery which presented a whole new set of challenges. Suddenly we went from burning a gold master cd as the final step in the process to handing over to teams that had just as much to do after ‘dev complete’ as we had during development. We had that same bottleneck issue again, except this time instead of the bottleneck being at the beginning (our hardware orders) it was now happening at the end (our release times). Sometimes it would take months for completed software to become available online for our customers. Even once it did become available it might take several more iterations to become useful.
It was at this time we decided as a company to start seriously thinking about continuous delivery. I employed some people from ThoughtWorks to help me make a case for applying ‘DevOps’ to a few projects. We ended up with a large PowerPoint presentation and I got to travel around the world for a while evangelising it before I must have bored enough people that they gave me resources to build a team to give it a go.
Looking back now I realise that PowerPoint presentation contained all of the wrong things. We were justifying ‘DevOps and continuous delivery’ on the basis of:
- Reducing risk
- Faster time to market
- Increased agility
- Putting control back into the hands of the business
- Increased time efficiencies
Having spent the past few years releasing SaaS products I can see that those reasons, while on the surface look reasonable and often compelling, don’t really explain why you want to be ‘doing DevOps’. They appeal to a very narrow audience and are extremely hard to measure. Even if you could measure them there is no guarantee that improvement in those measurements means you have actually helped the business.
It seems that even recent articles expunge the same list of benefits that I got caught up with a few years ago. Here’s one example:
While these articles aren’t necessarily wrong, they don’t really do a good enough job at explaining the core reason of why you would adopt DevOps principles.
What is the goal of DevOps?
The goal of DevOps is very simple; it is to make a company more money. If I could go back in time I’d throw away my PowerPoint presentation and list the following reasons for adopting DevOps practices:
- Increases net profit
- Increases return on investment
- Increases cash flow
The goal for any company is to do those 3 things simultaneously. If you’re running a modern software business then you’re probably doing a combination of lean start-up, agile software development and DevOps. The reason why you’re doing that combination is to make the company more money. You do this by mitigating bottlenecks throughout the entire company which leads to increased throughput. Where throughput is defined as the rate at which the system generates money through sales.
How does DevOps help your company make more money?
You’ve probably noticed by now that DevOps alone won’t work. Your throughput is limited by the slowest link in the chain. If you have awesome product management, sales, marketing, development, QA, finance, [insert other parts of the chain here] etc. and your bottleneck is releases, then improving the rate at which you can release will show an increase in the money that your business makes. If releases aren’t your bottleneck then spending time improving that area is actually wasted time.
I’ve personally experienced this at a couple of companies. At one company we had a manual QA process that took 3 days per release (after Dev complete). It seems obvious now that implementing push button deployments and auto scaling wouldn’t improve the overall system or make the company any more money, as we weren’t the bottleneck. Conversely I’ve been in situations where the release process was so bad that we calculated it would take over a year to release 15 feature branches sitting in source control. Implementing continuous delivery in this particular case meant those features got released over a few months without any adverse effects on the service. The financial graphs showed 3 large jumps in earnings as new enterprise customers signed up to the platform as we unlocked the features they wanted.
DevOps increases throughput while at the same time reducing work in progress and operating expense.
Where throughput is software releases, work in progress is unreleased code and operating expenses are things like labour costs and hosting.
At the start of this blog we talked about reducing the hardware bottleneck. This shifted to there being a constraint at the production deployment step. DevOps practices manage these constraints in such a way that throughput increases. A software business operates like a production line and shipping features faster gives a higher return on investment. The sooner you can get value into the hands of a customer the sooner you can start charging money for it. You can outpace competition and grow market share.
By setting up automated deployment pipelines and reducing release cycle times you reduce the ‘work in progress’ inventory. If you have software sitting on the shelf not earning you any money then it is a depreciating asset.
How can you practically implement this stuff?
A lot of the cultural aspects of DevOps are supportive of the idea of finding and mitigating bottlenecks by working closely with other teams. The more automation an operations team puts in place the more slack you should build in your operations group. There is a tendency at this stage to continue building slack and spend the spare time on playing with cool technology. If you do this then you’re wasting time that could be better spent finding constraints in the entire system, often in other groups, and offering to help. Over the past couple of years I’ve spent time trying to spot bottlenecks and give up resources to other groups like QA, Finance, Marketing and even Development in order to increase overall company throughput. With the goal of the company making more money.
How can Outlyer help?
When we first started thinking about doing a start-up we looked at creating a product that would make deploying software easier. Having spent the last 12 years doing exactly that for various companies we had a good idea of what was required and how to do things better. However, after a while we came to the conclusion that even if we created the best SaaS ‘deployinator’ in the world it might not actually help software companies earn more money.
So we decided to start at phase 2 of DevOps. Once you can deploy, you need to measure. We decided to create the best monitoring software on the planet with a vision of helping software companies make more money. We built a solution that could be easily customised to collect data from any source and present it in a way that makes it extremely easy to answer questions. Our service can be used to monitor, alert and display key metrics to every group. Over time we hope that operations teams will start to measure all parts of the system and use our product to identify bottlenecks and effectively manage them, and in doing so increase overall throughput (sales).
Outlyer is in private beta while we work with some enterprise customers to ensure we stick to the vision. We will be launching a public beta in the near future. If you want to be notified when that happens visit outlyer.com and sign up with your email address.