Hexacta Blog

Some of the problems I see in Agile Methods – Part 3

Santiago Ceria — Fri, 11 Dec 2009 20:19:09 +0000

A few days ago I was talking to a colleague and he told me “I’ve convinced my boss to start using Scrum”. “Great!”, I replied. But then he started to describe his situation and the problems he is having. And the main problem was that they still didn’t have a product backlog. And his question was “What do I do before I’m ready to do my first sprint?”. This is not exactly the issue I would like to discuss, but it reminded me of another problem I see in agile methods, and it’s the idea that “all sprints are equal”.

Let’s analyze some differences between two iterative and incremental processes: Scrum and UP (or RUP). One significant difference is that UP has the notion of Iteration Phase, or Iteration Type. This means that not all iterations are equal. And this makes a lot of sense to me. Not all iterations are equal because when you start a software project there are many unknowns and things you have to analyze and think about. And when you are about to deploy a system to a production environment or release it to customers there are many other things you need to do that are different from what you do during construction. This doesn’t happen all the time, but it happens often. And then you start seeing agile teams that do “Stabilization Sprints”, or “Release Sprints”. I’m sure those are forbidden words for many agilists, but at least I was relieved to read a post by Mike Cohn about release sprints where he describes a Bank that manages a 13 million LOC core system that needs them. Believe me, they are not alone.

I think Agile methods should put more focus in these issues, which are very relevant in many projects. And give practical and specific advice on how to do all this while being agile. In other words, I don’t think minimizing the potential differences in sprints is a good idea.

So, in this case, the problem I see in Agile methods is not related to what they say, it’s to what they don’t say.

Google DevFest 09

HAT — Wed, 25 Nov 2009 15:12:38 +0000

17th of November was a very cool day, i could enjoy the Google DevFest 2009 in Argentina and i will share you some exiting thinks from this event.

Some news about Google Apps

Google is working in a lot of new stuff, one of them is the improvement of the GEO API. The 3th version of Javascript API for Google Maps is growing up faster and its focused on speed browsing and will bring us some new cool feautres.

Android is hitting the 2.0 version and this week Motorola is bringing to the market a new cellphone with this operative system.

Others topics they talked about are Google App Engine, Google Wave, Open Social, Google Chrome Extensions and HTML 5.

DevFest09 documents

Some of the problems I see in Agile Methods – Part 2

Santiago Ceria — Sun, 22 Nov 2009 13:16:51 +0000

Another problem I see in some Agile Methods is a lack of flexibility about the rules they propose. I see that as some kind of contradiction: for example, Scrum is empirical only for things that it does not define beforehand.

“Don’t change Scrum” is something that Ken Schwaber wrote in his book Scrum and the Enterprise. Here’s his rationale for this: “Scrum isn’t a process that you can modify to fit your enterprise. Instead, it exposes every dysfunction in your enterprise while you build products… Whenever people change Scrum, it’s because they’ve run into a problem, dysfunction, or conflict that they do not want to face and fix. Instead, they change Scrum so that the problem remains invisible and remains deadly to the enterprise. If you allow this to happen, you will have lost Scrum’s primary benefit”.

I know that I can’t change the speed of light… But… why can’t I change Scrum? And what exactly is it that I can’t change? Are Scrum advocates so sure that it doesn’t make sense to have standups twice a day or once every two days or that they can last less or more time? Or that I can’t use any other estimation method? Or that being a Scrum Master is a full time job? Or that always the team has to be self organized? Or that scheduling tasks is not needed?

The rationale I imagine for this position is that being flexible will lead to misuses and ineffective processes. And that makes a lot of sense. In the past, for example, this is what happened to CMM and its original appraisal method “Software Process Assessments”. If you’re flexible, as that method was, people that don’t deserve a CMM Level will get it. So let’s make the evaluation method more rigid. And they created CBA-IPIs. But that method was also too flexible, and so they created SCAMPIs. But from what I see, this increasing rigidness is causing more problems than solutions to CMMI.

So what I understand is that some agilists say “If you’re flexible with Scrum, some people will say they use Scrum when they don’t and they will spoil its good name, so, don’t change it”. I really can’t agree with this. Scrum has very few rules, but I see no rules there that can’t be changed if it’s done with expert judgment and based on empirical evidence.

Some of the problems I see in Agile Methods – Part 1

Santiago Ceria — Tue, 10 Nov 2009 18:20:44 +0000

First of all, I admit I do not consider myself an expert in Agile Methods. Yes… I’m a Certified Scrum Master and I’ve been involved in several agile projects. But let’s say I don’t know enough and have enough experience to consider myself an expert. Even with this, I will dare say which are the issues that I don’t see addressed correctly in these methods, or at least in the typical “Scrum + XP” combination that used in most cases in Industry. The risk here is that someone will quickly reply with a link to explain how these issues can be solved. Oh well… I’ll take that risk.

I also want to say in advance that in general I have a positive view on Agile methods. I think they are an interesting packaging of useful practices and that they have made a huge contribution to development practice in general. I also value some new practices like TDD and Continuous Integration that have provided smart answers to increasing challenges of software development.

So… here goes my first critique. It’s to the idea that every sprint has to generate a “Potentially Shippable Increment”. To be honest, even after reading some posts by Ken Schwaber and other experts about this, I still don’t get why they are so rigid about this. There are many reasons why you would want to do something in a sprint that is not even close to shippable, no matter if it has the word “potentially” in front of it. One of them might be that what you have delivered is a proof of concept of a feature that needs more work, or you are working on a very complex functionality and are only showing to your product owner something that does not make “business sense” yet but is pointing to the right direction, or your sprint is so exploratory as you are very early in the development cycle that the presence of bugs is not a problem, or that you’ve decided to partition a complex functionality in two or more sprints. I would be more flexible here… saying something like “Test and make your product as shippable as you need to in each iteration, according to business needs”, combined with “Try to have your product make business sense as soon as possible in the development cycle”. This shouldn’t be an excuse for delivering poorly tested code. It’s just something more flexible and realistic. If a product owner needs shippable increments in each sprint, then go for them. For me, it’s just another business goal that has to be discussed in the corresponding sprint planning meeting. Another thing related to this that should hold true in sprints is something like “If you are developing a potentially shippable functionality in a sprint, it shouldn’t be accepted unless it’s really shippable (DONE) when the sprint is over”.

Bottom line is that I don’t like rigid positions to respond to common development mistakes, as they can induce to other dysfunctional practices.

Is Semantic Web the Web 3.0? (I)

HAT — Thu, 29 Oct 2009 15:06:08 +0000

People have been talking about web semantic and web 3.0 and we need to understand that they are two very different thinks. The term of the semantic web is coined by Tim Berners-Lee and there is no full consensus about what Web 3.0 means.

We can say that Web 3.0 is defined as the creation of high-quality content and services produced by gifted individuals using Web 2.0 technology as an enabling platform. At its core, the semantic web comprises a set of design principles, collaborative working groups, and a variety of enabling technologies.

OK, I understand, but why we need a Semantic Web?

Today there is information overload produced by heterogeneous sources. That is, the Internet information is scattered and the most cases it is impossible to mix these data because the systems automatically not know how to communicate with each other.

Semantics refers to aspects of the meaning or interpretation of the meaning of a particular symbol, speech, language or formal representation. That is, understand the Web from the point of view of meaning and not words. The idea of the semantic Web is to add meta data describing the content and the relationship of data to make it possible to be evaluated by machines automatically, expanding the Internet to improve interoperability between computer systems and reduce the mediation of human operators.

And what about Semantic Search?

Semantic search is a process used to improve online searching by using data from semantic networks to disambiguate queries and web text in order to generate more relevant results. A Semantic Search Engine could provide an overview that lists semantic search systems and identifies other uses of semantics in the search process.
The deal is that the semantic search is going to be big and it is going to help us answer questions that we simply cannot answer today.

The Problem of the Semantic Web

The Semantic Web defines a new paradigm and it will bring radical changes in how people interact with Internet. On the one hand, users have to learn to use semantic search methods and put aside the literal search that is currently used.

In addition, Web developers will have to learn new technologies to generate and maintain the semantic structure on the Web. This involves adding the development team with another responsibility.

So, will the Semantic Web accepted by users and developers?
Another question to ask, what happens if the contents are poorly documented? Or the meta data information of a web page is wrong?

Co-evolution – Brooks did it again

Santiago Ceria — Wed, 21 Oct 2009 21:34:40 +0000

In a previous post I commented about Fred Brooks’ great conference when he received the Turing Award. He named it “The Design of Design”. In that conference he spends a lot of time discussing life cycle models and explaining why the Waterfall model is wrong (“dead wrong” as he calls it). And he also presented what he thought was a better model: the co-evolutionary model, attributing it to Maher and Cross. In this model, from a Problem P1 you get a Solution S1. And from P1 and S1 you get a new problem, P2. And from P2 and S1 you get a new Solution, S2. This simple concept explains that the problem and the solution evolve in parallel. And it’s one of the good things that Agile Methods have adopted. If you use agile methods, your lifecycle model is not only iterative and incremental. It’s also co-evolutionary. Let me explain this better.

In other methods that adopt the iterative and incremental lifecycle model, the word “change” has a very strong meaning. In particular “requirements change”. These methods assume that you can define initially what you require, even if that’s only for one iteration, and they also assume that what you define initially tends to be right. If it’s not right, it only needs some “changes” that you can specify in a change request form. Well… that makes sense: for a final state to be different from an initial state there must have been a change in the middle. But in the co-evolutionary model you don’t express the new requirements as changes to the former. You just define the new problem. You rethink it now that you have a partial solution to it that gave you more insight. So, talking about a “requirements change” is difficult… it’s easier to talk about the “new problem”. Brooks tells a funny story about some changes to his house that he finally solved “thinking in a different frame of reference”. Some call it “thinking outside the box”. “Requirement changes” mentality makes you think inside the box, in the same frame of reference.

Sometimes when we develop software, and particularly in those environments where creativity is vital, we should open our minds to rethink our problem every time we reach a partial solution. That’s like asking yourself if you want to change your question every time you get an answer for it. If you follow the “requirements change” mentality, it will be hard to do it. You will also be assuming that your first description of the problem was correct, and that it will only need to make some adjustments.

Finally, I want to say that I’m not denying here the importance of change control and, in some cases, maybe in most of the cases, of requirements change control. I can think of many types of projects where not having requirements change control can transform them into a nightmare. I can also think of cases where all this I’m proposing just doesn’t make sense due to the nature of the problem. But in many cases it does. And maybe rethinking your problem will lead you to build a great software product. So don’t forget this: when you do incremental and iterative development, consider also using a co-evolutionary approach.

Significant Design Up Front, When?

Santiago Ceria — Tue, 25 Aug 2009 15:39:11 +0000

So in my previous post I said that nowadays there are fewer cases when building models before building the actual code of an application makes sense. In which cases does it make sense? More explicitly, in which cases should we spend a significant amount of time creating architecture and design models before we run to code? Let’s ignore for a while safety critical applications and “for other reasons very critical applications”, the easy answer to the question (I know that more and more applications are becoming critical, but still I want to focus on the other ones, at least for this post). My question is equivalent to asking “when do we need significant design up front”?

I thought I had the answer to this question. It goes something like: “The more you know about the domain you are working in and the reference architectures that make sense to that domain, the less time you need to devote to design up front”. When you’ve built 20 web applications using .NET that have similar functionality, you won’t need to design a lot before you start working on the 21^st.” This all seems very obvious, right? Now, let’s suppose you’re facing a problem in a completely new domain. You’ve built 20 web applications and now someone asks you to build an application to control the movement of a robot’s arm. No database, no front end, a real time operating system… all new. I had a similar problem a long time ago, and I remember how terrified I was when facing the empty whiteboard in our first design session. So, what do you do in this case? My answer goes something like “In these cases you can’t jump to build something because you don’t even know what languages or communication mechanisms you should use. What you have to do is to look for reference architectures for the domain, learn about the domain and how the solutions for that domain are usually structured, and try to come up with a specialization of that reference architecture that can solve your problem. Until you figure all that out, there’s not much you can code. It would be a waste of time” So, according to this theory, Design up front seems to be related with the knowledge of the domain and the technologies used in it. In the first case you don’t need to spend a lot of time on design because you already know the reference architectures for that domain. You already have a design in your head.

In a meeting last week I said all this to a group of software professionals and, when I proudly finished my explanation, one friend who has a lot of experience in software development, much more than I do at least in the technical side, said: “I’m sorry to say this, but I disagree completely with what you say. I think it’s exactly the other way around. When you know a domain and the technologies that work well in it, you can spend time on design. There’s not a lot you can do wrong designing. But if you’re working in an unknown domain, you have to start building something very small, as the chance of making mistakes is much higher. In the case of the robot, I would start by building a very simple functionality, something like moving the arm one inch. And then I would grow the system from there”. Long silence…

I won’t say if I changed my mind after hearing that. And you, whose side are you on?

Team Foundation Server from within Eclipse

HAT — Tue, 04 Aug 2009 19:15:39 +0000

I was trying to integrate a Java project with TFS, using Eclipse and Teamprise. Teamprise is a suite of client applications that includes an Eclipse plug-in for accessing Team Foundation Server.

In general the plug-in works remarkably well (in fact, I think it works better than Visual Studio with Team Explorer!). It’s really stable (thanks to Eclipse) and several options are easier to access than in Visual Studio.

Pros

Integration with the source repository is complete and limited only by TFS (you can still use the Team Synchronizing perspective or you can use the new ‘Pending Changes’ view). Some particularly interesting characteristics are the shelving feature, check-in policies, and the ability to associate changesets with specific tasks, bugs or any other work item type, providing a higher level of traceability. You can develop and deploy your own policies, although you have to install them in all clients.

Using reports and documents is exactly like in Visual Studio. The latest version (3.2) includes improvements to the way of connecting the project with TFS and supports the Maven 2 framework (previously it only supported Ant). You can define, launch or queue Java builds from within Teamprise using Ant or Maven – you can also install any framework on the build server such as JUnit, Checkstyle, Javadoc, etc.

Cons

It’s difficult to find labels in the code, you can’t look for labels by date (that’s actually a TFS limitation) or search a label for a specific folder or sub tree.

One of the most interesting components of TFS is the data warehouse. It seems to still need to incorporate some concepts such as build information and test coverage, among others – in general, the build reports are too brief for developers. It would be interesting if Teamprise could interpret files in TRX format to display test results (at the time you have to produce your own report using JUnit).

By the way, you can’t create a team project, set security properties, configure alerts, etc. You still need Visual Studio Team Explorer or command-line tools for those features.

The challenge

The main challenge for Teamprise is to keep up with the state of the art and to be able to evolve in two different ways at the same time: on the one hand, by adding features from the Java world and improving its integration with Eclipse and, on the other hand, by including new characteristics of the constantly improving TFS. So far I think they are doing a good job. In addition, it seems that TFS 2010 is bringing a lot of new and exciting features: will Teamprise be able to keep the pace?

A change in focus for Software Engineering

Santiago Ceria — Wed, 29 Jul 2009 19:39:42 +0000

My friend Hernán Wilkinson sent me a link to a blog post that included a reference to a very interesting article by Tom DeMarco. Just as a reference, De Marco is one of the classic authors in Software Engineering. When I studied systems analysis more than 20 years ago, his book on Structured Design was like a Bible.

I agree with almost everything he says there, but still I think that he takes it too hard against his own work, and I want to relate this to my previous post, because what he proposed in “Controlling Software Projects” is something that even in 1982 didn’t work in all software development projects, although from what he says it looks like he didn’t realize that at that time. But the key is that they worked in many software projects: in the types of projects that DeMarco was involved in then. So, probably his mistake was not to mention in what types of software development projects the techniques he was proposing had proved to be effective or were important in relation to what they were trying to achieve. Also, it’s easy to say now that some things proposed 27 years ago are not true anymore. Just as an example, “The Mythical Man Month“, Brook’s classic essays book that is a must for any software professional, has many things that are plain wrong if tried to applied nowadays, as Brooks himself recognized when he said “Parnas was right, I was wrong”.

So, why are so many things that were once right now wrong? Here’s a possible reason: For 40 years the software community (academia and parts of Industry) has been struggling to produce technologies to let us build more flexible software. David Parnas and the initiators of the Object Oriented Community (like Alan Kay) had a lot to do with this. New development environments, debuggers and configuration management tools also produced great improvements. Increase in processing power also made a huge difference. The funniest way of showing this is what Boehm recalls that his first boss told him when he was a programmer: “Now listen. We are paying $600 an hour for this computer and $2 an hour for you, and I want you to act accordingly.”But now that software is easier to change, the models that we strived to produce that were easier to change than the software itself have changed their cost-benefit equation. The other side of this coin is that the “increasing cost of fixing errors” curve is more flat now than a few years ago (not for the case when a bug is found in a production environment, though). So now the methodologies we use are a bit closer to a “code and fix” development lifecycle. Now… does this imply that code and fix is right? Does it imply that it was right 10 or 20 years ago? The answer is no in both cases, but the second no is stronger than the first one. Does this imply that it was always wrong to invest a lot of effort in software models? No! And does it imply that it’s wrong to build models now? Another no! But maybe today there are fewer cases where a significant investment in models is justified. The same can be said of other techniques, like the need for “measures and control” that DeMarco pushed for when he said “You can’t control what you can’t measure”, although he is right in that maybe there was too much a focus in management as a reaction to missed deadlines, and that now software engineering should concentrate on the more technical aspects of building software.

Our discipline is immature and a moving target. Our tools, techniques and methodologies differ from project to project, from domain to domain, from company to company. And they change over time. There are a few universal truths that affect everyone: those are the ones that Brooks mentioned in No Silver Bullet. That’s about it.

I wanted to conclude by saying that the authors of the Agile Manifesto did an extremely good job by choosing the words: “We are uncovering better ways of developing software” to start the manifesto. Old technologies and paradigms, missing tools and lack of processing power among others were covering those ways of developing software. So some things we did then seem evidently wrong now, but in many cases they were not, and DeMarco should still be proud of his work.

Why do we keep searching for unifying theories for software development?

Santiago Ceria — Mon, 29 Jun 2009 17:24:22 +0000

I want to come back to the discussion about Agile vs. CMMI. After a presentation about the subject in UTN (National Technological University) in Córdoba a couple of weeks ago, I started thinking a bit differently on the subject. I thought that instead of asking the question “Are CMMI and Agile Methods compatible” the important question is “Should CMMI and Agile Methods be compatible?”

In other words, if they are not compatible, so what? Isn’t software development an incredibly wide field so as to need different approaches to different problems? Didn’t for example Management benefit during its evolution from new theories that presented different points of view about how organizations should be managed? CMMI and Agile methods present many incompatibilities (at the CMMI practice level and not at the goal level), and one of the reasons is that they follow different management theories.

I always say that the best software engineer is the one that knows many different theories, tools, techniques, management approaches and so on. And has the judgment to recommend one or the other depending on the specific needs of the project. Off-shore outsourcing and distributed teams, for example, present unique challenges.
With such a wide field, the search for a unifying theory is wrong and doomed. I think that the future of Software Engineering is in specialization. Once specializations become clearer, it will be easier to recommend where each approach works better, and this applies both to technical and non technical issues.