Tuesday, 10 March 2009

A call for agile museum projects (a lunchtime manifesto)

Yet another conversation on twitter about the NMOLP/Creative Spaces project lead to a discussion of the long lead times for digital projects in the cultural heritage sector. I've worked on projects that were specced and goals agreed with funders five years before delivery, and two years before any technical or user-focussed specification or development began, and I wouldn't be surprised if something similar happened with NMOLP.

Five years is obviously a *very* long time in internet time, though it's a blink of an eye for a museum. So how do we work with that institutional reality? We need to figure out agile, adaptable project structures that funders and bid writers can also understand and buy into...

The initial project bid must be written to allow for implementation decisions that take into account the current context, and ideally a major goal of the bid writing process should be finding points where existing infrastructure could be re-used. The first step for any new project should be a proper study of the needs of current and potential users in the context of the stated goals of the project. All schema, infrastructure and interface design decisions should have a link to one or more of those goals. Projects should built around usability goals, not object counts or interface milestones set in stone three years earlier.

Taking institutional parameters into account is of course necessary, but letting them drive the decision making process leads to sub-optimal projects, so projects should have the ability to point out where institutional constraints are a risk for the project. Constraints might be cultural, technical, political or collections-related - we're good at talking about the technical and resourcing constraints, but while we all acknowledge the cultural and political constraints it often happens behind closed doors and usually not in a way that explicitly helps the project succeed.

And since this is my lunchtime dream world, I would like plain old digitisation to be considered sexy without the need to promise funders more infrastructure they can show their grandkids.

We also need to work out project models that will get buy-in from contractors and 3rd party suppliers. As Dan Zambonini said, ''Usability goals' sounds like an incredibly difficult thing to quantify' so existing models like Agile/sprint-esque 'user stories' might be easier to manage.

We, as developers, need to create a culture in which 'failing intelligently' is rewarded. I think most of us believe in 'failing faster to succeed sooner', at least to some extent, but we need to think carefully about the discourse around public discussions of project weaknesses or failures if we want this to be a reality. My notes from Clay Shirky's ICA talk earlier this year say that the members of the Invisible College (a society which aimed to 'acquire knowledge through experimental investigation') "went after alchemists for failing to be informative when they were wrong" - " it was ok to be wrong but they wanted them to think about and share what went wrong". They had ideas about how results should be written up and shared for maximum benefit. I think we should too.

I think the MCG and Collections Trust could both have a role to play in advocating more agile models to those who write and fund project bids. Each museum also has a responsibility to make sure projects it puts forward (whether singly or in a partnership) have been reality checked by its own web or digital specialists as well as other consultants, but we should also look to projects and developers (regardless of sector) that have managed to figure out agile project structures that funders and bid writers can also understand and buy into.

So - a blatant call for self-promotion - if you've worked on projects that could provide a good example, written about your dream project structures, know people or projects that'd make a good case study - let me know in the comments.

Thanks also to Mike, Giv and Mike, Daniel Evans (and the MCG plus people who listened to me rant at dev8D in general) for the conversations that triggered this post.

If you liked this post, you may also be interested in Confluence on digital channels; technologists and organisational change? (29 September 2012) and Museums and iterative agility: do your ideas get oxygen? (21 November 2010).


  1. The Swedish Open Cultural Heritage Web Service (http://www.kulturarvsdata.se/english.html) went from idea to public beta in about a year. Quite quick for a heritage project. The first half of 2008 was spent getting stakeholder buy-in and producing protocols and formats. The central index, Java and .NET harvesting nodes and the public API was developed in about six months and the service went public beta a few weeks ago after some alpha testing.

    I think one of the success factors was that the development team were well isolated from the primary stakeholders by the project manager. There was also some "guerilla development" involved i.e. develop first, ask for permission later...

  2. Hi Mia, the self promotion thing always makes me feel uncomfortable!... but the DigitalNZ project may be of interest. We recently used Agile Scrum to develop an aggregation and API service for NZ digital content. We managed to get the whole system live with aggregation across 30 institutions in just under 16 weeks. The stars kind-of aligned for us in that there was much more to this than just using an agile methodology. We had the complete support and trust of our advisory board to run with things, we also had ring-fenced budget, and we were carful to get the right kind of people involved. We also opted for almost no tech specs... the whole effort was kicked of with user-driven stories and wireframes, and the dev effort was driven off that front-end. Seb has a nice write up of what we did.

  3. Andy, DigitalNZ has been an inspiration to the Swedish project so self-promote with pride!

    Another factor in a short time to delivery is I believe to adopt a data first strategy, rather than a metadata first. Many museum professionals have this almost platonic idea of the digital collection with perfectly modelled metadata and that publication should wait until that state is achieved, which means they never publish. Publish first, order later!

    And finally that depressing subject of copyright. Adopt a when in doubt publish anyways stance. You´ll be damned by more important stakeholders than copyrightholders if you don´t i.e. the end users or perhaps the lack of any end-users.

  4. Detailed and articulate post Mia. Not in a museum environment, but using DSDM Atern, and I know my old colleagues at the NHM are considering it. User stories, moscow, wireframes, component behaviours, and timeboxes, good project processes - all this has helped us ride some major business direction change as the projects get going.
    Issues for Museums? you need a darn good project manager dedicated to it (the last part rare in Museums), and there are a number of vital roles in DSDM - not sure many Museums could support the team structure(s) required, but I've been told its adaptable. Its about far more than iterating code. Much more about collaboration with the 'business' e.g curator, creative, editor.

  5. A late reply - sorry! Assignments ate my spare time this weekend.

    @Andy - I did wonder about the wisdom of calling it 'self-promotion' - maybe a simple 'share your experience' would be better. I guess I'm particularly interested in stories from people who've managed to do it well, though of course the people who are reluctant to talk themselves up are often the real heroes.

    @David - that's brilliant. And it's interesting that you did it with tools that aren't traditionally seen as supporting quick web development.

    Did your project manager have the power to make judgement calls or did they still have to go through sign-off loops for some things?

    @Andy - I'm very jealous of your ability to just run with things. Did you find that user-driven stories and wireframes helped with that, or did they come afterwards?

    @David - by "adopt a data first strategy, rather than a metadata first", do you mean 'just get in there and do it'? Like a 'You Ain't Gonna Need It' for data structures?

    The spectre of copyright is really holding back the UK museum sector. We also have concerns about damaging the commercial prospects of picture libraries, but I think those can generally be addressed.

    @_mikeL_ - you're right, it's the collaboration that actually makes the difference; iteration just enables better collaboration because everyone can see how their content, user stories and ideas are being interpreted in implementation.

  6. Agility seems to be a luxury in the museum web world. Wonder why that is? It certainly doesn't cost more.

    In my recent experience, it seems the barrier to flexible development is mainly managerial; it's sometimes as simple as 'decision makers' not allowing people with real knowledge and tech skills to be heard properly in the chain of development.

    As Mike Ellis observed concerning the development of Launchball, when people in-house are trusted to go away and develop, good things happen.

    Agile development *can* happen almost anywhere, if managers really trust dev teams. Delegating responsibility to teams to fail if necessary (as commented by others here) is vital.

    I think we can work towards better lines of responsibility and delegation: what's harder is to grapple with the way projects are funded, and the way funders expect some very rigid sign-offs and processes.

    These structures lead to rigid dev paths that grind to a halt, as 'upstream decision-makers' get their hands on important conceptual or technical decision making points, and then sit on them, or vacillate for ever.

    Finally, Agile becomes difficult when you're managing multiple funding streams or diverse stakeholder groups - they expect rigid and predictable signoffs and deliverables.

    Which is crap really. Keep projects small, in-house, incrementally developed, single source funded, evolutionary not revolutionary. I see this as the reason Seb (and team) have done so well.

  7. It seems to me the key to agile development is trust. The head honcho must trust the primary stakeholder who must trust the product owner who must trust the developer team. In a lot of more traditional waterfall methods there´s too much of an emphasis on control and predictability. Does the museum enterprise culture lack in trust?

    In the Swedish project I believe the project manager/product owner had pretty much the right to make any decision or change in priorities he wanted as long as the agreed upon budget roof wouldn´t be affected or final delivery postponed. Some of those changes made were. e.g. a change from federated search to harvested index and using RDFs rather than XML as the main format for harvested data.

    On metadata: This guy says it better, http://www.betaversion.org/~stefano/linotype/news/93/

    Also in Sweden ther have been earlier attempts to create some sort of joint collections search and until now they´ve faltered and failed due to two main issues: 1. They never got past the vocabulary and ontology discussion they felt was necessary before building a joint index or a federated search and 2. lack of commitment from the primary stakeholders, perhaps because of the former point - no concrete early results, just minutes from meetings.

  8. Oops, saw now that Jon had already made the agile takes trust and accept unpredictablity points.