Why Agile Works

Why Agile Works

All agilists are familiar, of course, with The Manifesto for Agile Software Development, a statement of principles developed by a group of influential software developers and consultants, written and released in 2001. Now that agility has gone mainstream, however, we may have forgotten how controversial this document was at the time.  I had an interesting personal experience with the heat generated by the introduction of agile methods.  In 2000, a few months before the Manifesto was written, I interviewed Jim Highsmith for TechRepublic.com. We had what I thought was a pretty neutral conversation about what we were then calling “light methodologies”. The response was volcanic. Although TechRepublic has removed the comments from the archived version, the responses I got were basically of the “road to hell” variety.  The main arguments were either “This is an excuse for hacking” or “This blows up all the project management discipline we’ve been trying to teach the business for twenty years!” I was shocked at the vehemence, and, in some cases, the venom that a simple interview induced.

Agile is now conventional wisdom, but, like all revolutionary ideas, it took a long journey to get here. To really understand and apply agile, I think it’s important that agilists are familiar with some of the academic ideas that got us here. Agile development has strong theoretical underpinnings, based on research done at, for instance, Harvard and Berkeley, that examined the software development process. Let’s take a step back and look at some of the theory behind the agile movement.

Like many seemingly radical ideas, the Agile Manifesto was actually the culmination of an incremental realization, among both practitioners and theorists, that then-current methods of software development and project management had serious flaws.  These limitations were impeding the development of innovative software applications and causing important projects to take longer, cost more, and to be delivered without the key features that users required.   

The famous Standish Group CHAOS Studies, familiar to project managers worldwide, illustrate that the majority of projects fail to comply with schedule and cost projections, and often fail to deliver the benefits predicted when they were approved.  Additional research by entities such as the Department of Defense confirm these findings. At the 5th Annual Joint Aerospace Weapons Symposium in 1999, the results of a Department of Defense (DoD) software spending study were presented.  Of $35.7 billion spent by the DoD in 1995 for software, only 2 percent of the software was usable as delivered. The vast majority, 75 percent, of the software was either never used or was cancelled prior to delivery.

Academic research challenged common IT development and project management methods. In 1998, Harvard Business School academics Robert D. Austin and Richard L. Nolan studied large software projects. Their study questioned many of the fundamental ideas of IT development, and produced these key findings:

•”The first flawed assumption is that it is possible to plan large projects.

•The second flawed assumption is that it is possible to protect against late changes.

The third flawed assumption is that it even makes sense to lock in big projects early.

Watts Humphrey, a respected IBM researcher, followed this study with a paper outlining hisRequirements Uncertainty Principle, asserting:

“For a new software system, the requirements will not be completely known until after the users have used it.”

Hadar Ziv of the University of California followed soon afterwards with his Uncertainty Principle in Software Engineering, which states:

“Uncertainty is inherent and inevitable in software development processes and products.”

In an important and influential article that surveyed software development methods of innovative Internet companies, Alan MacCormack, assistant professor at the Harvard Business School,published a review of the history of software development methodologies.

MacCormack’s “Evolutionary Model of Software Development Methods” illustrates his history of IT systems development methods:

•Waterfall model – sequential process, maintains a document trail

• Rapid-Prototyping Model – disposable prototype, helps establish customer preference

• Spiral Model – series of prototypes, identifies major risks

• Incremental or Staged Delivery Model – system is delivered to customer in chunks

• Evolutionary Delivery Model – iterative approach in which customers test an actual version of the software

MacCormack also recommended a set of practices that could replace the traditional methods. Many old-school agilists consider this the first articulate explanation of the agile idea:

•Early release of evolving design and code

•Daily build of code and fast turnaround on changes

•Deeply skilled teams

Developments in industry, especially the lean manufacturing systems pioneered by Japanese firms like Toyota, validated many of the ideas brewing in the software and project management communities. The Standish Group finding, that around 60% of features built in to IT projects are rarely or never used, was also noteworthy.  Concepts like focusing on features the customer valued , and building in quality upfront rather than “testing it in” later,resonated with the software development community.

The development of agile methods accelerated in the 1990s, as Scrum was developed at Easel Corporation and Extreme Programming evolved at Chrysler.  Dynamic Systems Development Method, a light methodology, was introduced and quickly adopted in Europe.  Finally, in 2001, the Agile Manifesto was published, and agile development methods were on their way towards mainstream acceptance.

The connection between these academic ideas and studies and the underlying concepts of agile project management are clear. If users can’t foretell what they’ll want until they see it, if predicting and planning substantial IT projects is not possible, and if protecting projects against changes that arise during the development process is impractical, the ideas behind existing “waterfall” methods are clearly flawed, and an incremental, prototype-based methodology becomes the necessary remedy.  The revolutionary Manifesto was based on solid logic,research, and experience. That’s why agile works.

Agile: Emotional Maturity Required!

If you’re a project manager like me, you’re sick to death of the Standish Group’s Chaos Reports.  They come up in virtually every project management course I’ve ever taken or taught, and their basic premise is that, as project managers, we stink at our jobs. Since 2011, the success rate of the projects Standish evaluated landed between 27% and 31%, and the rate of projects either challenged or failed fluctuated between 69% and 73%. Not exactly a stellar record.

Source: Standish Group CHAOS report 2015

To be fair, the Chaos reports paint a much more nuanced picture than simply implying that project managers can’t manage projects. They measure elements such as the size of the project, the level of executive sponsorship, the level of user involvement, the skill set of the resources assigned, and much more.

Significantly, in this year’s report Standish Group has changed their definition of a successful project, from simply on time, on budget and within scope, to include the new language “with a satisfactory result”. My belief is that this change is a direct result of the agile movement, as the delivery of value is now recognized as an essential element of project success. As Jennifer Lynch from Standish told InfoQ in a recent interview, “We have seen many projects that have met the Triple Constraints and did not return value to the organization or the users, and executive sponsors were unsatisfied.” On their blog, Standish Group has taken a bold departure from traditional project management measurements to state, “The Standish Group believes that organizations should forget the triple constraints and focus on the value of their project portfolio, not individual projects.” Forget the triple constraint!? For those who’ve gained their PMP certification or toiled in the traditional project world, this is heresy. For agilists, the idea of focusing on value is obvious.

New in 2015, and of special interest to any agilist, is the inclusion of comparisons between agile and waterfall projects, and the insertion of a new metric, “Emotional Maturity” in the mix of success factors.

Let’s review these new elements individually. In a head-to-head comparison between agile and waterfall projects, illustrated below, the success of agile projects, regardless of project size, is significantly higher, with the number of challenged or failed projects correspondingly lower. In the InfoQ interview, Ms. Lynch states simply that agile can be helpful “…by failing earlier and restarting faster.” Agilists know there are many more factors involved in agile success, but “fail fast” is indeed a key element.  Regardless of Ms. Lynch’s simplification, the fact that Standish has rigorously reviewed over 50,000 projects of all types and sizes, and concluded that agile is significantly more likely to succeed, is a weighty outcome. Every agile advocate trying to convince leadership that agile works now has a very influential new arrow in their quiver.

Source: Standish Group CHAOS report 2015

As fascinated as I am by the agile statistics, the inclusion of “Emotional Maturity” as a success factor is especially significant to me. I’m a trainer of IT consultants, focusing on consulting and relationship skills rather than technology, and I’ve contended throughout my career that projects fail for human reasons much more frequently than for technical reasons.  It’s great to get this validation of that theory from Standish. As Ms. Lynch explains, “Emotional maturity skills are often called the soft skills. Emotional maturity skills include managing expectations, consensus building, and collaboration. We saw a major correlation between poor emotional maturity skills and lower success/value rates.”

The convergence of these two new elements, agility and emotional maturity, are especially potent.  Agile requires IT professionals to break down the walls between IT and the customer, and it requires project managers to look beyond the Triple Constraint to business value. Both of these transitions require the emotional maturity and consulting skills that Standish is now highlighting. What good is an elegant technical solution if we can’t facilitate a group to consensus?   What’s the benefit of software craftsmanship if we don’t have the ability to help clients articulate their stories? The old practices, of “project bureaucrats” hiding in their cubbies updating Gantt charts, and of an IT priesthood in a separate department, with supplicants begging for software or support, are dead. The concept of agility without emotional maturity and consulting skills is a non-starter, and it’s a giant boost to see the respected Standish Group validate that idea.

The U.S. Army’s Agile Narrative

The U.S. Army’s Agile Narrative

I was teaching an “Agile Project Management” class in Dallas. It was open enrollment, so there were participants from lots of different companies; Motorola, NEC, and a group from the U.S. Army.  We did an exercise in which we rated the agile ‘readiness’ of a fictional set of organizations, one of which was a government agency.  As we sorted and ranked the criteria, one participant made the comment that “obviously, the government agency will be ‘old wave’…they’ll have the most bureaucracy and be totally command-and-control…”.

That comment evidently got under the skin of one of the Army guys, because he, very politely, went into a rant about the agility of the modern Army.  He described the brainstorming techniques his squad had learned.  He enthused about the self-motivated nature of his team, and the acceptance by brass that the individual soldier can be creative on the field of action. He conceded that the Department of Defense still required a task-oriented project plan, a firm budget, and a committed schedule, but he was convinced even that was changing.

He was clearly right.  If you need confirmation that the core ideas of agile have gone mainstream, check out this new commercial from the U.S. Army. It might as well have been titled “Inspect and Adapt”.

“We train, adapt, and get smarter…”, says the narrator, “…not to keep up with change, but to drive it.” Over images of cross-functional teams of soldiers, doctors, nurses, teachers, and engineers, we learn that “no-one knows what challenges tomorrow will bring.”

This thirty-second commercial confirms that the fundamental ideas of agility have become conventional wisdom. “Challenges never stop”, as the commercial says, so continuous delivery,  continuous education, and continuous, conscious improvement is essential. Change never stops, and is not controlled or controllable; it is inherent in conflicts, in projects, and in life.  We can utilize iterative methods to try and keep up, but we must accept change as the driver. Cross-functional teams, with the ability to take ownership of a problem and marshall all the forces necessary to solve it, are as necessary in the military as they are in the agile IT department.

Even the U.S. Army, the very definition of command-and-control, has accepted these principles, because of the threat they face. Guerrilla warriors do not wear uniforms or stand in ranks. They sneak in, sneak out, and adapt to the circumstances of the moment.  As soon as we figure out their tactics, they change. On both sides of the front, adaptability means survival.

So too in business. Digital disruptors probe constantly for weakness, and pounce without warning or mercy. The pace of technological change increases,  business models morph, clients become increasingly sophisticated, and every human gets connected.  Every IT organization, and every enterprise, needs to acknowledge the truths that the U.S. Army has accepted; change will happen, self-motivated teams can deliver great things, and adaptability is the key to victory.

The Kansas City Royals: An Agile Team?

The Kansas City Royals: An Agile Team?

I’m a fan of Ned Yost. He’s assembled a team of highly-motivated, collaborative, and accountable players, set them an aggressive vision, given them the confidence of strong support, and set them free to practice their craft. They iterate through, inning by inning, learning and adapting as they go. The Royals are the perfect example of an agile team, and Yost looks to me like an agile manager.

Like Yost, agile project managers or scrummasters often have teams composed of supremely skilled and confident contributors. Like Yost, many agile project managers can struggle to get team members to subsume their personal ambitions and instead focus on team results. And, like Yostagile project managers must develop a leadership style that inspires and enables team members to achieve.

The coaching metaphor is an appropriate analogy to illustrate the type of project leadership that agile methods require. Great agile project leaders are coaches, with the critical understanding that, whether it’s fielding a pop fly or developing software, only the player can make the right decision under the pressure of the moment. Creating the environment that enables the experts to do what they do, and setting the strategy while allowing the players to create, are attributes of a winning coach and an agile project leader.

Notice that I use the phrase project leader rather than project manager; this is a key distinction. “Most projects are over-managed and under-led,” Jim Highsmith notes in his book Agile Project Management. While the tools of project management, such as plans, budgets, and schedules, are sometimes necessary to control the complexity of IT initiatives, no team member was ever inspired by a Gantt chart. When projects travel beyond the realm of the known and require innovative ideas and deliverables, legacy project techniques become an obstacle. Teams need the agility to adapt to the situation on the field, with a supportive leadership and culture so they can create.

Remember the first statement of the Agile Manifesto: “individuals and interactions over processes and tools.” Creating a collaborative environment that enables teams to achieve creative results together, and encouraging contributors to focus on group goals and agendas rather than the individual, are the agile leader’s most important challenges.

Many project managers have difficulty stepping away from the regimented techniques described above, such as scope, budget, and schedule planning.  It’s hard migrating into a world of changing requirements, evolving expectations, and self-directed teams. The evolution from traditional project management to agile techniques requires an evolution for the PM as well, from top-down, task-focused management to collaborative, influence-based leadership. The maturity to evolve from a world of tightly-bounded scope and schedules to a world of change,  iteration, and collaboration is the key distinguishing factor for successful agile project leaders. 

One of the foundation ideas of agile, taken straight from lean philosophies, is “light process, heavy skill”. The  highly-skilled, cross-disciplinary team,  unencumbered by excess process and motivated by a common vision, is the agile delivery unit, whatever the product.

Highsmith describes six elements to the creation of a self-directed agile team:

•Get the right people

•Articulate the project vision, boundaries, and roles

•Encourage interaction

•Facilitate participatory decisions

•Insist on accountability

•Steer, don’t control

One error I repeatedly see project managers make is recruiting team contributors solely based on technical capabilities. In agile teams, technical genius is not enough; behavioral aspects and maturity are, in my view, even more important. Every experienced project manager has dealt with the technical expert who is self-focused, needy, argumentative, immature, and disruptive. Agile teams don’t have time for this behavior. In the highly-charged environment of agile IT, one monkey can stop the show, when that monkey constantly requires special handling and disrupts the collaborative effort.

Agile project leaders understand that there’s a big difference between a spec sheet and a vision. Inspiring a team to create something unique and valuable requires more than a requirements definition; it requires leadership that paints a picture of what could be, if the team is creative and innovative. Agile teams internalize the idea that team intelligence is always smarter than even the smartest member and that participatory, interactive decision making results in direction that everyone buys into and drives toward.

Without accountability, all of these elements are meaningless. Accountability requires tough mindedness, since it implies that, when a team contributor turns out to be a poor fit for the project, the team will take action rather than sheltering and protecting an unproductive member.

In the debate about agile methods, the question often asked is “for what sort of projects is agile management appropriate?”  While the right project fit is an important question, we need to remember that an equally important question is “what sort of team is appropriate for agile methods?”  You can’t point at nine guys in the street and say “You’re the Kansas City Royals!”.

Building a team of mature, skilled, self-directed, and collaborative contributors is the central goal of any agile coach or leader. Project management process has its place in bringing order from the chaos of complex IT initiatives, but creating the right atmosphere and staffing it with the right contributors is the foundation of a high-achieving agile team. Just like the Royals.

My Agile Journey: Waterfall Zealot to Agile Evangelist

When I recall my career as a project manager, one day seems pivotal. I was a new graduate in“programming”, as we called coding back then, and was hired at a major NY bank as a specialist in the then-new PCs and network group. After managing a migration project that went long and over budget, I was invited to a session with a senior programmer on the bank’s team, for what I expected to be a scolding. Instead, Jerry took me to a blackboard and explained the simple fundamentals of managing an IT project. Jerry drew four boxes on the blackboard, and put a ‘D’ into each one. “What do you think the D’s are for?”, he asked. I just stared, stupefied. Jerry then condescendingly (this being NY), explained a simple 4D methodology for delivering IT programs.

Discover, Design, Develop, Deploy. Being a typical New York guy, Jerry didn’t explain this to me in terms of critical paths or network diagrams.

“Discover what the guy wants, and what you’re walkin’ into.” Jerry advised. “Design a solution to fix the guy’s problem. Develop that thing that you designed, then Deploy it for the guy.”

Over the next ten years of my career, I, like millions of IT project managers, amplified Jerry’s common sense project management philosophy by exploding each phase into tasks, assigning estimates to those tasks, and creating “gates” to ensure that no-one illicitly progressed without completing their activities. Those gates then became more codified, and eventually developed into the 17-binder proprietary project methodologies that were all the rage in the 1990’s.

I went to work for one of the Big 5 consulting firms, and was trained in their particular version of the multi-checklist, highly-enforced methodology. After a year, I became such a zealot of the regimented, gated approach that I joined the Project Committee. Then I got out in the field and quickly understood why it was unworkable in a consulting context. Simply put, no client being billed by the hour will abide dozens of hours of overhead, filling out forms and passing gates that don’t add business value. The amount of energy we spent trying to convince clients to pay for our project management time was prodigious.

I’d also noticed the beginnings of a “light methodologies” movement erupting, with guys like Rob Thomsett writing books like “Radical Project Management”. My quest became simple: find a method that maintained the rigor of a phase-gated approach with the low-overhead of these new ‘radical’ project ideas.

When the overhead of the big consulting giant became too much, I moved to a local system integrator. This business, dealing with smaller clients and less complex IT challenges, was nevertheless struggling with consistent delivery. We spent a long time experimenting with different versions of ‘project toolkits’,and eventually reached a light, adaptable 4D-style model that was less likely to scare away clients. With the simple application of a lean project management discipline, we grew the consulting practice significantly, and I became branch manager.

I left this gig to write my first book, “The IT Consultant”, and began to travel as an advisor to other IT shops. A pattern became clear, the more small IT shops I visited. Either they were strangling in bureaucratic project regimes that destroyed their flexibility, or they were making it up as they went along and disappointing clients.

I discovered a magic secret: put in place a few, simple project disciplines, apply some consulting skills, like collaboration and communication, and most IT shops will solve most of their problems.

On a writing assignment during this time, I interviewed Jim Highsmith, soon-to-be signatory of the yet-to-be-written Agile Manifesto, and Jim introduced me to ideas that were about to revolutionize the world of software development. The concepts of high-performing teams that managed and motivated themselves, of enforced client participation, of lean thinking throughout the process, fell so neatly in line with the on-the-ground experience I was having in the field, that I quickly became an adherent.

Yes, agile is revolutionary for those who came up in a gated, waterfall world. And yes, it’s going to become a lot more revolutionary in the next few years, as these agile ideas of collaboration, speed-to-value, adaptability, and iterative, incremental delivery move up the ladder from IT through marketing to the executive suite. But it’s based on real, pragmatic experience.

Those who lived through the shift from PERT-method, Gantt-chart, predictive management know that it taught us a lot, but it got strangled in its inconsistency. You can’t possible predict how projects will go, no matter how many gates you set up or papers you sign. The technology changes instantly. We can’t know what clients will want, since they rarely know themselves. We can’t accurately estimate huge programs, but we can time-box and cost-box projects and deliver valuable, useful increments quickly. We can’t manage talented knowledge workers into compliance, but we can lead them to glory. Once IT executives, their teams, and their customers absorb these ideas, agile will change everything.