Interview: From Apollo Mission to agility manifesto

The twelve principles of agile software development were laid down by 17 good men and true on a February day in 2001 at the Snowbird ski resort in the state of Utah. Like any software programming methodology, it resonates with flavours borrowed from other approaches, but agile’s positive willingness to embrace changing requirements, even late in development, has made it more popular than ever. One of those 17 original authors of the agile manifesto was Jim Highsmith (pictured) who, after a previous tenure at NASA, now works as an executive consultant at ThoughtWorks.

CW: Can you give us an “elevator pitch” explanation of agile that encapsulates its basic ingredients?

JH: The way I define agility is the ability to create and respond to change in a turbulent business environment. Ultimately, agile is about responding to change as it now happens even more rapidly, particularly over the last decade or so. I think it is very difficult for a lot of people to really embrace change. So this idea of continuous change and adaptation is a really important piece of agility. The idea of collaborative and cross-functional teams - which are, to a certain extent, self organizing - is another big piece of agility; and the inclusion of customers right inside the process rather than just at the beginning and end of a project.

CW: Why do you think the need for agile came about in the first place? What problem were you trying to address?

JH: Well, I've been in the software development game for a long time and through the 1980s I did a lot of work with more traditional technologies and methodologies including the waterfall model. I decided that I didn’t want to work in this very sequential and heavily documented fashion where projects were just getting longer and longer. It just didn't appear responsive to business needs. So, I started in the early 90s doing something at that time we called “rapid development”. I had my own form of it called “radical software development”. We would go into clients and we would do projects with one week innovations and it just felt like that was a more productive and customer-friendly way to do projects.

CW: Will the velocity created by web-based apps and rapid consumption mobile apps be the killer factor that now keeps agile at the fore?

JH: I think there are actually four technologies coming to the fore that are going to make an impact on business in the next few years and everybody is talking about them.  The move toward mobility and mobile apps is one, the move towards cloud computing of course, the whole phenomenon of social media applied to business situations, and then finally the data analytics side where we are just capturing so much data and now we're having to figure out what to do with that it. Where agile really shines is in situations where you have a lot of uncertain and changing or ambiguous variables - and this is the situation we have now.

CW: Among your core tenets for agile you state the best architectures, requirements and designs emerge from self-organising teams. Has social media helped this?

JH: Well, first of all I'd like to say that yes, agile has embraced this idea of self-organising teams, giving the team more autonomy while empowering them to make certain decisions. But this thought has kind of gone off the rails a little bit, recently. I wrote an article a couple years ago called No more self-organising teams because I think to some extent this has been defined inappropriately. Figuring out what kinds of decisions teams get to make and what kinds of decisions teams don't get to make is still the role of the management. We still need strong team leadership, we just need a different type of leadership. For the record, I think the use of social media has definitely been a plus in terms of a lot of agile projects, particularly the distributed projects.

CW: Popular opinion suggests NASA used a lot of the more sequential waterfall approach to programming rather than agile. Given your work on the Apollo space programme, wasn’t moving to write the agile manifesto a big change for you personally?

JH: Yes it was, but it happened over a four-year period. When I worked at NASA back in the 1960s, it was really a different world. I worked on really unstructured approaches to software in the 1970s and very structured waterfall approaches in the 1980s, before moving to a more rapid, more agile way in the 1990s. One of the things that's very interesting was that NASA is not as waterfall as you might think. In fact, one of the guys who really did a lot in terms of project management for large NASA projects had quite an agile view of the world. At one point, it was even reported to Congress that sequential approaches were not working well. Obviously the Apollo space programme involved a lot of documentation, but really, from a software development perspective, it was much more iterative and probably more ad hoc than we would have liked.

CW: What kind of computer system was used on the Apollo space programme? Did you feel like you were wielding immense computing power at the time?

JH: I worked on communications. The Apollo spacecraft was due to land in the middle of the Pacific Ocean and we didn't really have a communications system to connect with ships that ran in the middle of the ocean at that time. They built five ships that were basically miniature Houston control centres. They were going to send them out into the middle of the Pacific to track the spacecraft as it came back into the atmosphere. These ships had radars, central navigation systems and systems for gathering data. The computers were Sperry Univac 3012 militarised versions.

One of the interesting things is that they actually had a red “battle button” on them, so if you were in the middle of battle you pushed the button and just let it go until it burned up. They were 32-bit machines and had 32K of memory - they were very powerful for the time. When we were out on the ships, most of the programming input and output was done by paper tapes so it was a very slow methodical process. But yes, we did feel like we had a lot of computer power at the time. But obviously, compared to anything today, it was really minuscule. It's kind of hard to believe what actually was accomplished, on what today we would call an immensely underpowered computer.

CW: You formed the agile alliance as a group of 17 people. Did you have the feeling at the time that you were creating something special, that people would be talking about ten years on?

JH: No!  One of things we did at the agile conference this year in August was a celebration of ten years of the signing of the agile manifesto. I don't think we had any idea of what would happen because, at the time, there were several of these methodologies around - extreme programming, adaptive software development and future driven development - which were all becoming very popular. But it was kind of few and far between, it wasn't at a level of what I call "rogue teams in organisations" i.e. the kind of guys that could convince their managers to try something new. The problem was that “organisational anti-bodies” would come out and attack and it was very difficult to get new methods implemented in bigger organisations. The big change over the last few years has been that we've moved from looking at rogue teams to having vice-presidents of engineering and CIOs who come in saying, “We really want our entire organisation to be agile”.

CW: Tell us about your work with ThoughtWorks and what drives your passion for technology right now.

JH: Well, one of the things I have been doing since I came to ThoughtWorks is really working more with managers and executives on how to implement agile in a more effective way in organisations at all levels. agile excels in delivery and actually getting projects out the door. What we've been less successful at is really getting agile accepted and promoted into the wider organisational processes as opposed to just in the software development area - so I am to change that.

It's really important that you drive a real understanding of the business need for agility, which I call “enterprise agility”. Companies should ask themselves what level of agility they really need in their IT or their product development or organisation as a whole. Many people view agile software development as an operational thing, an engineering thing or a departmental issue, but it really can make an impact on the whole organisation.