Let’s analyze some differences between two iterative and incremental processes: Scrum and UP (or RUP). One significant difference is that UP has the notion of Iteration Phase, or Iteration Type. This means that not all iterations are equal. And this makes a lot of sense to me. Not all iterations are equal because when you start a software project there are many unknowns and things you have to analyze and think about. And when you are about to deploy a system to a production environment or release it to customers there are many other things you need to do that are different from what you do during construction. This doesn’t happen all the time, but it happens often. And then you start seeing agile teams that do “Stabilization Sprints”, or “Release Sprints”. I’m sure those are forbidden words for many agilists, but at least I was relieved to read a post by Mike Cohn about release sprints where he describes a Bank that manages a 13 million LOC core system that needs them. Believe me, they are not alone.

I think Agile methods should put more focus in these issues, which are very relevant in many projects. And give practical and specific advice on how to do all this while being agile. In other words, I don’t think minimizing the potential differences in sprints is a good idea.

So, in this case, the problem I see in Agile methods is not related to what they say, it’s to what they don’t say.

“Don’t change Scrum” is something that Ken Schwaber wrote in his book Scrum and the Enterprise. Here’s his rationale for this: “Scrum isn’t a process that you can modify to fit your enterprise. Instead, it exposes every dysfunction in your enterprise while you build products… Whenever people change Scrum, it’s because they’ve run into a problem, dysfunction, or conflict that they do not want to face and fix. Instead, they change Scrum so that the problem remains invisible and remains deadly to the enterprise. If you allow this to happen, you will have lost Scrum’s primary benefit”.

I know that I can’t change the speed of light… But… why can’t I change Scrum? And what exactly is it that I can’t change? Are Scrum advocates so sure that it doesn’t make sense to have standups twice a day or once every two days or that they can last less or more time? Or that I can’t use any other estimation method?  Or that being a Scrum Master is a full time job? Or that always the team has to be self organized? Or that scheduling tasks is not needed?

The rationale I imagine for this position is that being flexible will lead to misuses and ineffective processes. And that makes a lot of sense. In the past, for example, this is what happened to CMM and its original appraisal method “Software Process Assessments”. If you’re flexible, as that method was, people that don’t deserve a CMM Level will get it. So let’s make the evaluation method more rigid. And they created CBA-IPIs. But that method was also too flexible, and so they created SCAMPIs. But from what I see, this increasing rigidness is causing more problems than solutions to CMMI.

So what I understand is that some agilists say “If you’re flexible with Scrum, some people will say they use Scrum when they don’t and they will spoil its good name, so, don’t change it”. I really can’t agree with this. Scrum has very few rules, but I see no rules there that can’t be changed if it’s done with expert judgment and based on empirical evidence.

I also want to say in advance that in general I have a positive view on Agile methods. I think they are an interesting packaging of useful practices and that they have made a huge contribution to development practice in general. I also value some new practices like TDD and Continuous Integration that have provided smart answers to increasing challenges of software development.

So… here goes my first critique. It’s to the idea that every sprint has to generate a “Potentially Shippable Increment”. To be honest, even after reading some posts by Ken Schwaber and other experts about this, I still don’t get why they are so rigid about this. There are many reasons why you would want to do something in a sprint that is not even close to shippable, no matter if it has the word “potentially” in front of it. One of them might be that what you have delivered is a proof of concept of a feature that needs more work, or you are working on a very complex functionality and are only showing to your product owner something that does not make “business sense” yet but is pointing to the right direction, or your sprint is so exploratory as you are very early in the development cycle that the presence of bugs is not a problem, or that you’ve decided to partition a complex functionality in two or more sprints. I would be more flexible here… saying something like “Test and make your product as shippable as you need to in each iteration, according to business needs”, combined with “Try to have your product make business sense as soon as possible in the development cycle”. This shouldn’t be an excuse for delivering poorly tested code. It’s just something more flexible and realistic. If a product owner needs shippable increments in each sprint, then go for them. For me, it’s just another business goal that has to be discussed in the corresponding sprint planning meeting. Another thing related to this that should hold true in sprints is something like “If you are developing a potentially shippable functionality in a sprint, it shouldn’t be accepted unless it’s really shippable (DONE) when the sprint is over”.

Bottom line is that I don’t like rigid positions to respond to common development mistakes, as they can induce to other dysfunctional practices.

In other methods that adopt the iterative and incremental lifecycle model, the word “change” has a very strong meaning. In particular “requirements change”. These methods assume that you can define initially what you require, even if that’s only for one iteration, and they also assume that what you define initially tends to be right. If it’s not right, it only needs some “changes” that you can specify in a change request form. Well… that makes sense: for a final state to be different from an initial state there must have been a change in the middle. But in the co-evolutionary model you don’t express the new requirements as changes to the former. You just define the new problem. You rethink it now that you have a partial solution to it that gave you more insight. So, talking about a “requirements change” is difficult… it’s easier to talk about the “new problem”. Brooks tells a funny story about some changes to his house that he finally solved “thinking in a different frame of reference”. Some call it “thinking outside the box”.  “Requirement changes” mentality makes you think inside the box, in the same frame of reference.

Sometimes when we develop software, and particularly in those environments where creativity is vital, we should open our minds to rethink our problem every time we reach a partial solution. That’s like asking yourself if you want to change your question every time you get an answer for it. If you follow the “requirements change” mentality, it will be hard to do it. You will also be assuming that your first description of the problem was correct, and that it will only need to make some adjustments.

Finally, I want to say that I’m not denying here the importance of change control and, in some cases, maybe in most of the cases, of requirements change control. I can think of many types of projects where not having requirements change control can transform them into a nightmare. I can also think of cases where all this I’m proposing just doesn’t make sense due to the nature of the problem. But in many cases it does. And maybe rethinking your problem will lead you to build a great software product. So don’t forget this: when you do incremental and iterative development, consider also using a co-evolutionary approach.

In other words, if they are not compatible, so what? Isn’t software development an incredibly wide field so as to need different approaches to different problems? Didn’t for example Management benefit during its evolution from new theories that presented different points of view about how organizations should be managed? CMMI and Agile methods present many incompatibilities (at the CMMI practice level and not at the goal level), and one of the reasons is that they follow different management theories.

I always say that the best software engineer is the one that knows many different theories, tools, techniques, management approaches and so on. And has the judgment to recommend one or the other depending on the specific needs of the project. Off-shore outsourcing and distributed teams, for example, present unique challenges.
With such a wide field, the search for a unifying theory is wrong and doomed. I think that the future of Software Engineering is in specialization. Once specializations become clearer, it will be easier to recommend where each approach works better, and this applies both to technical and non technical issues.

Why are they compatible? Because agile methods propose a set of practices that represent a fairly significant coverage of the practices requested by levels 2 and 3 of CMMI. Scrum in particular covers very well the the planning, monitoring and control, and requirements management process areas, among others. Some Extreme Programming practices such as continuous integration cover very well some Level 3 engineering process areas, such as Product Integration.
In Hexacta, particularly, we are about to present several “agile” projects for a CMMI Level 3 SCAMPI, hoping to demonstrate that these projects comply with all the practices of levels 2 and 3. So from that standpoint, Agile and CMMI are compatible.

Why are they incompatible? Because what the agile methods propose is not enough to be a Level 3 organization. Therefore, it is necessary to add some additional practices. The problem is that many of these practices are things that agile advocates do not recommend, or in some cases even hate, like everything related with most of the documentation often required by CMMI practices or that is sometimes needed as evidence to pass a SCAMPI. So, if some “agile purist” saw one of our “agile” projects, she would say they are not agile (I even think some would say much uglier things).

Another comment I want to make, which is closely related to the above, is something that surprises me: the lack of flexibility shown by some agile advocates on their methods. In particular, there is a paragraph from the book “Scrum and the Enterprise” by Ken Schwaber that calls my attention. He says: “Do not change Scrum. Scrum isn’t a process that you can modify to fit your enterprise. Instead, it exposes every dysfunction in your enterprise while you build products. It is your canary in a coal mine. Whenever people change Scrum, it’s because they’ve run into a problem, dysfunction, or conflict that they do not want to face and fix. Instead, they change Scrum so that the problem remains invisible and remains deadly to the enterprise. If you allow this to happen, you will have lost Scrum’s primary benefit.”
I just can’t understand this position, unless it is an exaggeration to avoid the appearance of “pseudo Scrums”. Flexibility is key in developing software. The best software engineer is the one who knows more methods, techniques and tools, and knows which to use and when to use them, and even how to adapt each method, technique and tool to a particular project and context. You cannot be inflexible with “soft” things like Scrum. This is not an exact science. There are critical mission projects, there are product development projects, there are 5 people and 35 people projects. The variety is endless. My recommendation for all is: “Do not remove fundamental practices to hide weaknesses, like Schwaber says, but adapt the methods to the projects’ and organization’s needs.” Presenting inflexible methods with the “either this or chaos” statement has already done a lot of harm in Software Engineering history.