Prototyping is SCRUMptious?

Something that has been floating in and out of my awareness for a while now was driven home recently during a round table after a viewing of Kent Hudson's 2010 GDC presentation The AI of BioShock 2: Methods for Innovation and Iteration. It is primarily a process talk - and the process is, roughly speaking, prototyping with SCRUM.

To condense an excellent 45 minute talk into a single paragraph: Mid-project the team declared the flagship AI character a failure. The team decides to radically adjust their process, adopting a "SCRUM-like" process. The team begins to iterate on a rough prototype performing successive refinements. The team assumes group ownership over the feature and tackles "finding the fun" as a cross discipline effort, providing good cross discipline feed-back. Over time the feature converges to fun, and the team converts the final prototype into the production final. Profit.

The reaction that several of us had after the viewing was "wow, that is totally different than most prototype with SCRUM stories I know!" While some of us could relate similar success stories, there were also a good number of failure story with a set up that was not so different from what Kent talked about. Later I bugged some other people and when back over my own experiences to collect more data. Focusing on just the combination of prototyping and scrum a couple of archetypal stories emerge.

Stories

Story one is similar to Kent's initial failure story. Rather that iterating to find the fun, the team develops various prototypes covering a broad spectrum of features. Prototyping is declared done when all the proposed features have a representation in game, and the team moves on to production. At some point people realize that while there are a lot of features, none of them is fun either individual or in aggregate.

Story two is where the team creates a prototype, but rather than iterating on features in progressive refinement on the rough grey-block, the team iterates on visual polish. Over time the features approach the visual completeness of production-quality, but the mechanics never really progress. For this team there is a confusion between the visual polish and the fun.

Story three is a kind of antithesis to Kent's success story. The team creates a prototype, but rather than entering a good feedback loop that converges on a fun feature, each iteration is like a completely new prototype. Rather than smaller changes to the existing prototype, each iteration makes such sweeping changes that it is effectively a new prototype.

There was lots of variations to these stories, but these were the three archetypes that emerged. 

So the question that I began asking myself was, "what can you extract from these stories? Is there an identifiable set of patterns?" I am not sure, but I had some interesting observations.

Some observations

One was that Kent's success was success stolen from the jaws of failure. In most of the stories that related a failure, it was a failure stolen from the jaw of success. These stories didn't generally begin "we knew we were screwed, so we went back to the drawing board and really looked at what we were doing." They began more like "so when we started the project, we all felt pretty good about what we we doing." It isn't a part of the process, but it seems a common issue around the process. Confidence or just complacency keeps people from looking at what the actual outcome is over time and at what the process is providing.

The second general pattern is around the definition for success. For Kent's team the definition for success was team driven and fairly clear. Create an enemy in Bioshock that provides a melee-centric mid-boss feel. For most of the other stories, there were two problems. One was that the "real" success criteria were vague: "create a action/adventure melee combat system", "create AI for a FPS", "create sandbox AI". The real success criteria can't be evaluated by the team when they are vague, they can only be evaluated by "experts" -- the designer or producer whatever the local term is. 

In the vacuum left by the real success criteria, the teams do one of two things: stop evaluating as teams, or define stand-in success criteria such as "make our stories go away" or "burn down to zero". The team effectively takes itself out of a buy-in position from a process point of view. I don't mean that on an individual level people stop trying to do their best, but as a team there is no process directed feedback.

Some thoughts

It seems to me that there are two activities that we call "prototyping". One is exploratory in nature; we don't know exactly what we are after, so we build examples to help define the solution. For the other, we know what we are after, but the details are not clear or are novel. While we don't differentiate between these two activities, they have very different implications on process. It seems that the second is much easier to execute successfully in the SCRUM model. It is an almost perfect fit. 

Exploratory prototyping I think tends to work poorly with scrum, precisely because it looks like it ought to just magically work with SCRUM. A group of people working flexibly and iteratively on  a feature? Great! Except that exploratory prototyping isn't "a feature", it is series of features that are connected by a theme. It can be hard to differentiate between iteration on a feature and the iterative evaluation of candidate features. There is probably added difficultly because sometimes people think they are prototyping towards a concrete goal, but the goal is vague enough that in reality they are doing exploratory prototyping.

On a separate note, it is also very difficult to evaluate prototypes because you have to "look around" the grey block, to see past the incompleteness of the prototype to the essence of the feature. And frankly it is a true multidisciplinary talent. You have to see through half baked code, art, and design. Not many people have the ability to handle the analysis on all these levels, and I think this contributes greatly to the problems that many groups have with prototyping. When the role of evaluation devolves from the whole team to an "expert", that expert then has to evaluate the viability for all disciplines.

Conclusions?


I am not sure that I have strong conclusions other than the obvious. There are obviously wrong ways to prototype with SCRUM. You need to be continuously vigilant about your process, because it can always go wrong. 


It should be obvious that you need to keep your success criteria real and interpretable by the team implementing the feature. It should be obvious that the team needs to be part of the feedback loop on the feature, not just the process.


If you are prototyping try to be aware of whether you are doing exploratory prototyping, and if so account for that in the process.