Good, fast, cheap. Choose two. The project management “iron triangle” expands on the common law of business balance by contending that a project is constrained between scope, cost, and time. A change in one of these constraints necessitates adjustments in the others to compensate, or quality will suffer.

Fig. 1: Project management triangle

Waterfall development, the flawed ideal

For the first few decades of software development, the methodology most widely used to manage the scope, cost, and time constraints for software projects was the waterfall development model. In this model, these constraints are all fixed up front. This reduces cost because issues are resolved more effectively when addressed as early as possible.

Fig. 2: Royce waterfall model (from Wikipedia)

In a perfect world, where there’s no risk of change occurring over the course of a project, this is the ideal development model. In actuality, however, this model is deficient. As unexpected events occur over the life of a project, quality rapidly degrades unless there are concessions to the plan for constraints. Unexpected changes typically result in a lower quality deliverable that blows the budget, drags out the schedule, or is missing planned features. Even when features are implemented as designed, customers’ needs often change during the extended development cycle typically required by the waterfall approach, reducing or eliminating the value of planned features.

Iteration and the scientific method

Software development projects typically take place in the context of imperfection, discovery, and unexpected change. Design won’t capture every edge case. Market conditions change suddenly. Resources turn over. Bugs surface. And users provide feedback that necessitates modifications. Throughout the 1990s, the US Department of Defense heavily leveraged waterfall development (DoD STD 2167) but found that over 75% of software projects failed or were never used. How does a development team effectively manage the unexpected changes that occur in the real world?

The answer that has emerged to this quandary is agile software development (often referred to as “Agile”). Agile is not a self-contained and fixed development methodology or procedure like waterfall. It is a mindset that encourages flexibility and accountability in response to real-world scenarios. Agile can be implemented via numerous process frameworks, such as Scrum, that encourage adaptive solutions to complex problems by adopting the scientific method and empiricism, its fundamental principle.

Fig. 3: Scientific method (graphic by Efbrazil at Wikipedia, CC BY-SA 4.0 license)

Empiricism states that knowledge is best obtained through experience and evidence, and the scientific method requires all hypotheses to be tested against real-world observations. Likewise, the Agile mindset encourages accountability through measured processes that adapt based on acquired evidence. The Agile cycle of design, develop, test, deploy, and review (illustrated in the diagram below) embodies the scientific method.

Fig. 4: Agile development cycle

Managing risk through iterative development

Agile development process frameworks like Scrum aim to provide the benefit of waterfall development by shifting design left, but they split planning and development into smaller, iterative chunks to manage risk through shorter development timelines, rather than relying on comprehensive initial plans to carry development through an entire project. With Scrum, daily work is captured in conversation-driving user stories instead of fully-planned features. And a project’s overall vision is still captured (in a high-level story map), but scope, cost, and time constraints are not initially fully defined as they would be with a waterfall roadmap. Planning still takes place as well, but in more manageable “sprint” iterations that last from one to four weeks. Additionally, metrics are collected during sprints and reviewed to refine the high-level story map and improve the process in future sprints.

Ad hoc development, another extreme

Why do we need a development process such as waterfall or Scrum? Why not just drive development based on whatever is the highest priority today? It can be tempting to focus on implementation in an ad hoc way, avoiding design and planning altogether. After all, wouldn’t that be more productive?

Ad hoc development may be more productive, but only in the short term. Be careful to avoid jumping to any extreme. Agile provides a middle ground between rigid up-front planning and no planning. Well-planned and measured software projects enable you to make better choices, which ensure that time is spent on valuable effort and that technical debt is effectively managed. Addressing design up front reduces cost, but complex problems need to be broken down into smaller, manageable pieces to enable teams to better estimate and prioritize their work as changes occur. Properly done, Agile facilitates both. Additionally, metrics from a structured process encourage accountability that enables teams to tackle technical debt, improve estimates, and communicate changes.

Avoiding compromise

The waterfall model has the right goal but is impractical when faced with real-world changes. Ad hoc development may seem productive and flexible in the short term, but quickly breaks down into unmanageable waste and deployments without targeted value. On the other hand, Agile (via a process framework such as Scrum) provides a practical middle ground that facilitates project success, as long as fundamental principles of Agile and the process framework are maintained. Compromising these principles quickly drags development into a hybrid waterfall model or an ad hoc approach.

The “End Note” in the 2020 Scrum Guide is clear about compromise:

The Scrum framework…is immutable. While implementing only parts of Scrum is possible, the result is not Scrum. Scrum exists only in its entirety and functions well as a container for other techniques, methodologies, and practices.

Agile may be the golden mean between planning and flexibility, but it doesn’t give you everything. You certainly cannot always have perfect quality alongside the ideal scope, budget, and timeline. And as unexpected changes happen, not even shorter development iterations will always enable you to perfectly balance all constraints. However, Agile development processes do include mechanisms that enable better adaptation to change. With Scrum, for example, a well-tended backlog will enable teams to prioritize work and better negotiate constraints, leading to better quality and value-driven deliverables.

Ultimately, Agile development is about working in smaller increments and following the scientific process with measurable results to continually improve scope, budget, and timeline estimates and balance based on real-world evidence. Agile processes are the best solution we have for managing change in software projects while staying focused on what is valuable and leveraging metrics to continuously improve quality.