Being able to rapidly deploy code through environments to production is a gamechanger in enabling business capability and delivering business value. There are lessons and concepts used in classic Lean manufacturing we can learn to help us get there.
By significantly increasing our deployment speed, we go beyond powerful efficiency gains to enable a radical change in business capability.
The primary environmental variable driving the difference between classic Fordist manufacturing and Lean manufacturing is one of demand shape. The environment that Toyota found itself in in post war Japan was very different from that of Ford. The demand was for low volume, high variety of production, with low availability of capital to sit doing nothing other than incur warehousing costs as inventory. Production had to match short term actual (rather than long term forecast) demand and deliver a quick return of value to customer and capital.
Building a rapidly responsive manufacturing capability necessitates moving towards ever smaller batches of product, and being able to support frequent switches of product. In doing so, you find that you deliver value faster from concept to cash, and your overall demand is effectively leveled, avoiding the boom and bust in workload requiring the pains of rushing to meet unrealistic quotas, regular hiring and firing and associated loss of the skills and knowledge you’ve invested in. You can build a steady workforce working sustainably, developing their skills, focusing on technical excellence, allocate capacity to improvement and so on.
So useful is the ability to deliver in leveled small batches of high variety that Toyota forces it by Heijunka — planning to build every product every day.
The primary constraint to being able to move product through the value stream in increasingly small batches — True North being a batch size of one — was changeover of dies on the large transfer-stamping machines. Changing these required (at a high level) stopping the line, swapping out the old die and disposing of it, preparing the new die, swapping in the new die with a high degree of precision verified by cycles of test stamps & adjustment, heating the die to operating temperature before restarting the line. This incurs such a transactional cost that the economic minimum lot size tends to be quite high; certainly well above a single vehicle.
The traditional management response to this would have to avoid incidence of the transactional cost through larger batch size and so dividing the cost between more products – in fact, if you ask pretty much anyone not introduced to Lean Thinking, they will tell you that larger batches are cheaper.
This becomes a recursive, self-amplifying problem; as batch sizes increase, the tooling becomes more massive and harder to change, increasing the batch size. At the same time, it becomes more expensive and capital intensive, increasing the cost that needs to be recovered and therefore also increasing the economic minimum lot size.
You can think of this as the classic arms folded, lean back
“We Can’t Because” school of thinking.
Luckily, this isn’t Toyota’s Way. Their style is the much more engaged
“We Can If…” Approach; understanding problems as obstacles on the True North Journey. True North is single piece flow, so anything blocking the journey to True North is something to be worked on to allow the journey to continue. So, the question posed to the improvement team is
“we can reduce the minimum economic batch size if…?” with the first order answer being
“…if we can reduce the line-stopped time taken to exchange the die” with a target of Single Minute Exchange of Die (or SMED), which would easily support an Economic Minimum Lot Size of less than one vehicle.
The outcome wasn’t the perfect process that would instantly enable SMED but an incremental approach that would start from current state and iteratively approach the target.
Obstacles to One Piece Flow In Enterprise Delivery
When I talk to technology delivery teams about single piece flow, and iterating towards it via increasingly small and more frequent releases, I regularly find an obstacle thrown up around the constraints of release – build and deployment.
For far too many teams, this is a slow, complex process, handling fragile artefacts. Releasing software into any new environment is a huge risk as it carries no assurance that it will actually work in the new environment the way it did in the old one. It takes days from completing the value-add work in the previous environment to being up and ready to do the work of the next one.
In a million pound project I ran last year, my highly experienced delivery exec insisted on having a red risk on build and release of the final system into UAT.“How do you know it will build?”she asked.
That we had been building it on every check in and successfully deploying it to UAT every fortnight for four months didn’t seem to sway her because in her experience things just don’t go that well.
Why not? Do we as an industry suck that much?
We stop the work in both environments — meaning non-value-adding time for both teams. We prepare the new environment by clearing it down. We deploy the code and configuration — hoping we don’t miss anything — and build for the new environment. Then we smoke test against gross errors, then regression test to see what we broke. Only then can the work of the new environment begin and the old environment cleared to start on the next thing. Of course, if that environment is not the first in the value stream, it needs to go through the same process.
And to repeat: this can take not the hours of an unimproved manufacturing line but days.
If you’re trying to deliver working software every few weeks, this means you’re spending a significant proportion of your time just deploying, which in itself delivers zero value.
Worse than that, we express very low levels of confidence in what we deploy, assuming it’s highly likely to fail (at best) or entirely break (at worst) the production infrastructure.
To accept this level of cost and risk naturally drives us towards
“We Can’t Because” thinking; towards the annual release and change freezes. It is to accept that we — as an industry — are fundamentally incapable of anything better. That our equivalents in manufacturing are fundamentally smarter than those of us in knowledge work industries.
We Can If…
As an industry, we’ve spent a long time emphasising how different we are from manufacturing. That Software is a Discovery or Product Development process, not a Production one. That Craftmanship is critical (sorry Sandro – I agree with the manifesto but think the term is misleading). Fundamentally that we’re better than those blue collar guys.
Well, I’m saying that they’re smarter than our prejudices allow.
In the value stream of delivering software — be it business projects, internal projects, changes or defects, and often all four in parallel — we’re still working in a value stream that has many characteristics that echo manufacturing. And the primary one is that value begins and ends with the customer. No value is delivered until that software is running on a production system and nothing that holds that up for a moment should be tolerated.
The True North of delivering value through software is to have it working on production the moment all the prerequisites have been completed; no waiting for any non-related functionality to be ready.
Therefore, we have to be able to support an Economic Minimum Lot size of a single change, a single feature, a single defect, and be able to reliably and rapidly deploy it through all development and test environments truly on-demand on its own timescale.
When we get this truly working, we can achieve all the following:
- Earlier realisation of business value to our stakeholders
- Better availability of environments
- More stable environments
- Better capability to cope with the variety of demand
- Higher productive availability of our team
- Better levelling of our team’s workload, with positive impacts on
- ability to plan capacity and allocate to improvement and quality
- people burnout
- Better ability to respond to business ideas
Of these, the last one is the real game-changer. The others simply deliver a more efficient delivery pipeline. Which is good, but this one’s radical.
In Lean, we talk about the uncertainty of improvement experiments and the criticality of defining them so that we extract maximum learning. Guess what? Business is in the same place. Being able to run rapid iterative experiments in the real world with actual customers and suppliers to validate their hypotheses is a capability that most businesses would kill for.
Which market segment do we address? What’s the ideal pricing? Does this product work? What’s the risk/reward balance point of security? Business would love to test these. And they’d love to test them when they’re running under highest volumes because that’s when the maximum opportunity for learning is.
To do that, they need to be able to atomically implement functionality on-demand, and back it out again. Also on-demand. And to try five variants with randomly chosen user groups, and back them out. And being told
“No, you can’t test in your peak season; there’s a change freeze” is just another way of disappointing the business; just another way to tell our customers that they can’t rely on us and we’re unresponsive and uncaring of their needs and priorities.
We can do better. So much better.
So let’s learn from Lean Manufacturing and make the deployment pipeline a greased helter skelter. Let’s look at all the things that makes our deployment fragile and slow and engineer them out. Let’s use the SMED methodology to simplify the process and reduce the outage time to the bare critical activities.
Let’s set a True North: Single Minute Enterprise Deployment of atomic functionality.
Let’s move from
We Can’t Because… to
We Can If….