Is Agile Software Development a hype or does it deliver benefits that are destined to make this approach the main paradigm of development of the future?
Google for agile and you get thousands of results, many of which contain quite a bit of information demonstrating the success of agile behavior on the basis of team-oriented approaches. Even if this were all true, that wouldn’t mean that the new paradigm would be accepted. The insights of Thomas Kuhn [Kuhn, Thomas S. 1970. The Structure of Scientific Revolutions. Univ of Chicago Pr (T) (1970), Edition: 2nd, Paperback, 210 pages.] hold for our domain, too.
That difficulty apart, is it really that simple to deploy “methods” like “Scrum” or “XP” successfully?
Let me be quite up-front about what I think:
- there are a many Scrum methods as there are book authors
- there is no method, and there is no process, that can help a team to perform with the grace and effectivnes of agility, if the environment, in which the team is embedded, does not want such behaviour
- to behave in an agile manner I do not need so-called “agile methods”
- none of the so-called “agile methods” guarantees the the desired agile behaviour, not even at team level.
Well, I still think one of the key competences to deal with the fast increase of complexity in the systems we construct is to learn how to become more agile (see my post on “agile“). So I am an”agilist”, but with a difference, maybe.
The core of the “agile aproaches” is to take the (nearly ‘ancient’) insights of Barry Boehm and others at heart, and to give human factors the importance they always should have had in knowledge intensive tasks. Software is created by the “other soft-ware“, by human individuals themselves and on the basis of their social relationships. This is why the Agile Manifesto is so important, because it lays down philosophical foundations of how to get an organisation to learn agile behaviour from the roots up. These philosophical foundations do need re-intepretation in the respective context and for the specific team and task. Those who only see “less documentation” in the manifesto have not come anywhere near to the true meaning of the manifesto. This approach here is very similar to the concepts of the Toyota Production System.
I described what I consider agile behaviour in the other blog. What is important is that the (development) team, which is the centre of attention in the lime light, can on its own not deliver the performance that is expected. The team needs to be supported by an active network of all those, who together have all the specialist knowledge and prior experience to complete the task at hand. This is also an essential success condition with “traditional” development paradigms. I have never seen a development team that had all the knowledge and experience readily inside the team to develop a complex product, whatever the development paradigm.
Therefore we need to consider two different, but synergetic and interdependent approaches:
1. Team-Orientation: team-orientation can help avoid many of the “mudas” of the top-down control orientation of forward plan based resource allocation. A significant part of the problem stems from the fact that the development team cannot in itself have all the relevant knowledge and experience of the organisation for the task at hand.
2. External Team-Empowerment: the team must be empowered to access all the relevant knowledge and experience resources it requires and operate in a supportive environment.
The so-called “agile methods” focus mainly on supporting point 1. This has to deal with empowerment to, but concerns more directly the team itself. Experience with effective approaches for point 2 can, for instance, be found in community-based projects, such as in the open source software field. Prof. Heinecke, CEO of BMW Car-IT, spoke about this at the International SPICE Days 2010, and Mike Milinkovich, CEO of the Eclipse Foundation, spoke about building the required supportive environments at the Conference on Software-Competence for the Future in 2011. Trust, respect, and demonstrated competence facilitate the establishment of a meritocracy. This meritocracy serves as the internal reference framework and structures of the respective “communities” in the organisation.
It is this second perspective that will decide if team-based development paradigms can be deployed successfully in an organisation. How can you successfully import these effects, which have formed part of the evolutionary history of open source projects, how can you make them work effectively inside organisations? Well, Bosch has been looking at this for quite a while as part of its internal TOP programme. More about that some other time, hopefully. Thanks to my partner Bonifaz Maag I have quite recently come across an interesting study by IBM, which details such a transplantation of such constructs into a corporate setting and presents some results. [Patrick Howard, Dorit Nevo, and Pat Toole. Small Worlds: The Social Approach to Software Delivery. Executive Report. IBM Global Business Services. IBM Institute for Business Value, 2012.]
The “experiment” was at an IBM scale: three plus years and several thousand participants. The starting point was the need to achieve improvements and innovative capability “jumps” to at least keep pace with the ever increasing dynaxity (ever increasing complexity at ever decreasing intervals). So they asked themselves what the most import factor was, which decided about success or failure of development projects. The answer is not surprising based on the discussion so far:
“The unifying force in all of these projects was the professional, drawing on his or her expertise and relationship network to solve problems and identify solutions.”
Here it is again, the “soft” part of software development. IBM’s short form of the key:
“the better the social interactions, the better the results.”
The experiment started in 2008. Based on the structure of business processes IBM created the communities of experts and committers, which are so very typical for open source projects. This is what IBM calls “small worlds.” Appropriate infrastructures helped linking across organisational and geographical boundaries. Within a community the work of everybody was visible to everybody else. And IBM created a system, which facilitated establishing a trust-based meritocracy inside IBM.
“In a large-scale study of 112 of the 130 globally integrated communities, representing more than 5,000 software engineers delivering on 300-plus projects, we noted significant performance improvements in the complex activity of software engineering and delivery by these project teams. Over the past three years, the cycle time across all major programs, measured in terms of days to deliver a unit of software, was reduced by 30 percent. The quality of code, measured in terms of defects delivered to production for a unit of software, improved by 20 percent.”
This may not sound spectacular, but these are only the “marginal improvements” of existing practices. The real purpose was to tap a creative potential, which would allow higher innovation rates. Unfortunately the study des not give any measures about that – yet. I doubt the would have even referred to it, if this had not been a success. I suppose we’ll have to wait for the next report.
Back to my hypothesis that “external team empowerment” is necessary for development teams to develop their potential. IBM’s dry conclusion
“communities provide the social and knowledge resources needed for teams to perform better”
is certainly what one would have expected. More interesting are the other findings:
“Communities with higher levels of trust among members, and whose members are socially connected to one another, have higher performing teams,”
“communities that have a repository of relevant and well-documented reusable assets also have higher performing teams.”
So, here we return to Barry Boehm. Are we perplexed? [Boehm, Barry, and Richard Turner. Balancing Agility and Discipline: A Guide for the Perplexed. 2004. ed. Addison-Wesley Longman, Amsterdam, 2003.]
“Communities as a team’s virtual environment”
is the approach of IBM. IBM has shown how the experiences of Open Source, Toyota Production System, and team-based methods can be transferred successfully to commercial enterprises.
Are there any conclusions we can draw beyond this?