Saturday, August 22, 2015

The Paradox of Agility

In Software Development "Agility" means that you try to improve your process continually, be aware of it, measure it, learn from your mistakes. Right?

To be able to do this you have to repeat your process again and again. You need to have a specific metric within a specific process you try to improve. The unspoken assumption is you want to have a "repeatable process", so you have something against which to measure the effect of changing some process-parameters.

Like say we try to improve our estimates about how much we can  accomplish in the next 2-week "sprint".  The fixed, repeating part is that we do two-week sprints.  It makes sense when you try to optimize the outcome to keep many parts of the process fixed and vary just some parameters in small increments to see how that affects the outcome.

That is the Paradox of Agility. You try to keep your process well-defined and repeatable in order to learn from it. But if you keep the process constant it means it can not change. And if you can not change it, how can you improve it?  Incrementally, yes. But small incremental change is not agility. Agility literally means "moving quickly and easily" (  
The practice of incremental improvement sounds like a recipe for finding a local maximum. By adjusting your process-parameters, your "bearings", you move upwards or downwards or sideways on the slope of a hill. You can always measure your altitude,  or your "burn-rate" and so try to maximize it.  Eventually you will end up on the top of hill having rejected process-parameters that took you downwards and keeping  ones that take you up.

Now you are on top of the hill, running your optimal Scrum process. Great. The only problem is it is a hill, but not The Hill.  You're on top of a small one.  Or at least you may be, you just don't know.

Take the agile SW development methodology "Scrum" as an example,  
Scrum is "iterative and incremental". I would add "highly structured". It has 2-week "sprints" which have phases and events like  "Sprint Planning", "Daily Scrum", "Sprint Review" and "Sprint Retrospective"

What I think the greatest  part of Scrum is the Sprint Retrospective. It
" ...   Identifies and agrees continuous process improvement actions".   

In other words you are trying to continually improve your project by repeating it over and over, to be able to learn the best ways to apply it.  This is the Paradox: You need to repeat it to improve it, but if you repeat it how can you change it? If you can't change it much, you can't improve it much. You can do that, but only incrementally.  And when you reach a local maximum, any further incremental change can only take you down.

One way to look at it is you are playing a game, and you can improve your game. Get a better score. But can you, should you, change the rules of the game? Why not, if other players agree.
So what can be done?  Not much. Ending up with local maxima is a well-known problem in operations research. It can be counteracted with techniques like "Simulated Annealing",  How to apply that to process-improvement might be an interesting research-project for a PhD student.

If there's not much we can do, there is one thing we can do. Think. Think about different process-models, Scrum or Kanban, or Home-Grown?  Understand that while a repeatable process sounds like a great goal, it can lead to stagnation and local optimums. In my next post I plan to present an alternative to Scrum, to emphasize that we can at least THINK of other process models, other hills than we're currently on.  That may be the biggest benefit of having a process model: They allow us to think about how they could be different.

 © 2015 Panu Viljamaa. All rights reserved