Measuring change: who decides what’s measured?

I have spent the summer and early autumn helping two organisations improve their ability to reflect on and learn from their experience. Both pieces of work have involved how to measure change in the real world.

The first was designing an impact assessment framework for International Service. They do capacity-building with local organisations in five countries around the world, and they are interested to know more about the difference that this capacity building is making in the lives of their partners’ beneficiaries. That’s quite a long impact chain, from sending a development worker to discerning how (hopefully) improved services are improving individual lives.

International Service is fifty-six years old this year. A much younger organisation, the Newcastle Conflict Resolution Network, is planning its activities over the next twelve to eighteen months, and I have been invited to help design their logframe. Ultimately we’re hoping this will help evaluate their impact, because of the attention they are giving now to:

• What are our realistic objectives?
• What are the changes that we want to see and measure in the future? (indicators)
• Where will we look to find any evidence of those changes? (the means of verification)

My input into these organisations has been informed by the latest CDRA Nugget – if you’re not signed up, these occasional nuggets of insight and information are well worth reading.

This latest nugget I paraphrase here as: those involved in designing impact assessment frameworks (oops! – that includes me…) must remember the most important lesson of all: initiatives work best if they are designed and implemented with those who will be the recipients. Note the word ‘recipients’ rather than ‘beneficiaries’ – let’s not assume that everything our organisations do turns out for the good.

And so I am helpfully reminded that thinking how to assess the effect of one’s work requires you – me – to ask the recipients: what would you value as a positive impact? And how might that impact be measured? That I am not asked these questions as a recipient of local or government services here in the UK does not diminish the obligation to ensure they are asked through the organisations I work with.

More on collaborative working

In my previous post, I used the analogy of star-formation to discuss one aspect of consortia-building. I want now to turn to poetry as a means of highlighting another side of working in closer partnership.

Try these lines from an e.e.cumming’s poem (the lower-case letters are the poet’s conventional style):

love’s function is to fabricate unknownness

…

how lucky lovers are (whose selves abide

under whatever shall discovered be)

I understand these lines to say that one purpose of a relationship is to make space in which something as yet unknown may emerge. And having emerged (‘whatever shall discovered be’), how lucky lovers – friends – organisational partners are, whose relationship is strong enough to survive, be strengthened by, and to make use of what they have created between them.

Put this way, the link to consortia becomes clearer: we do not know what will emerge from the process, but the starting intention is to find out if we have a relationship strong enough to hold the fruits of collaboration.

What becomes possible to achieve, may in the end be what no-one chooses to sustain. But if one begins with an adventurous “It makes sense to be in the same room for the moment, let’s see what happens as we continue…”, then I believe the partners will have much greater capacity to abide under – and develop further – their mutual creation.

Framework’s core membership is meeting later this month to explore the world of collaborative working and consortia-building. If you want to know the outcomes of our discussion – or to suggest poetry or other resources we should be paying attention to! – please do be in touch; your ideas would be warmly welcomed.

The poetry quoted above is taken from “love’s function is to fabricate unknownness” in Selected Poems 1923-1958 e.e. cummings, Faber and Faber 1969, page 34.

Star formation and partnership-building

The Reflection Nebula

My current desktop photo is an image of the Reflection Nebula, taken by the Hubble Space Telescope. The description from the www.hubblesite.org says in part: “The bright, young star left of center gives the nebula its brightness. The gas and dust of the nebula is left over from the star’s formation.”

I love the concept of stars-in-the-making: an ageless incremental process of gradually combining dust together which might eventually gather enough mass to collapse and ignite.

I’m using this image to represent a process I observed last week: the potential forming of a new voluntary sector consortium. In the room were three people from a particular area of service delivery, acting in the belief that they needed to work closer together with a view to winning statutory contracts which may emerge in the future.

As the discussion deepened, they began facing the questions which confront all consortia-in-the-making:

• Are we designing something for our wider network to engage in, or will it be just us? What are our loyalties to other network members?

• What do we do about organisations we don’t want to join – are we an open or a closed group? Are we open to organisations outside the Third Sector?

• What standards should we expect of each other as members?

• Are we in this process for the benefit of service users, or is our main motive to ensure the survival of our organisations?

These are tough questions, and if properly explored would help towards strong relationships between those ultimately involved in the consortia. In Framework we have long experience, in the UK and abroad, of supporting and creating quality relationships. A strong basis for partnerships is open communication, particularly about the issues which are likely to be most difficult. This means risk-taking, accompanied by an inward search to be clear about one’s own motives and priorities.

All this is easier to write about than to embody in daily life. Ultimately, Framework is a one kind of consortia – a network of potential competitors joining together for mutual benefit. Which isn’t to say we get it right internally all the time!; but we’ve learnt never to stop working on the relationships between us.hs-2000-10-a-pdf

From parenting to management (3): seeing the future

From parenting …

Some of the more unsettling moments as a parent are when I’ve looked into my child’s face and seen what they might look like ten years or more in the future.

Have you ever had that too? It’s as though a window of insight has opened, and I’m seeing the shape of their face, or an expression, which will in time become very familiar.

Though the surprise of seeing them in the future is odd, what’s most unsettling is the accompanying shock of recognition, as if I was expecting what I’m only now just seeing.

I find I’m saying to myself, “Oh, so that’s what s/he’ll look like then”. The shock is of something apparently already known but only now being confirmed.

And then the window closes, and we move on: the child unaware, and me busy recovering and reordering my internal resources!

… to management:

Managers – especially when they are responsible for bringing changes about – need to have a vision of the future. It may be intangible, or incomplete; but they know inside them what needs to be different and how the new order ought to be.

Between the vision and the new reality is the change process. And whereas my insight into my child’s future appearance is strictly for my own internal reflection, within organisations if you’re implementing change then your job is to articulate the vision of the future as often as you can. Clearly, consistently, and frequently. It’s like the promised land: if we keep going, and keep acting to get there, we’ll get there – or somewhere just as useful.

But first there is the vision; an unvisited insight, perhaps, or one worked towards through an awayday or strategic planning exercise. “Oh, so that’s what we can be like. So now, what do I do first to get us towards there?”

The story behind the picture

In my presentation to Mediation North’s November 2008 conference, I aimed to set out a simple framework to help community mediation services monitor, and then evaluate, and then assess the impact of, their conflict resolution activities.

The presentation itself – this version can be opened, with apologies for earlier difficulties – can be found at http://www.framework.org.uk/MODULES/NEWS/framework_NEWSmoduleASP/NEWSMOD_newsitem.asp?type=News&itemid=73

Civil wedding ceremony, Burundi 1996

The second slide of the presentation is a photograph of a civil wedding ceremony in Burundi in 1996. The couple are Quakers, and after the formal marriage they will head back to the Quaker church at Kibimba for a religious celebration.

I chose this photograph because it highlights so beautifully the complexity of conflict – and of trying to monitor and evaluate an organisation’s work.

The couple are Tutsis; the registrar on the right is a Hutu. As it happens, all three physically conform to the traditional stereotype of the conflict: the Tutsis are thinner, lither; the Hutu is of a stockier build. It’s important to note that these are stereotypes and I met people of all shapes and sizes from different ethnic backgrounds whilst I was working in Burundi.

What’s unusual for the stage of the conflict is that the official is a Hutu; at that time, most political jobs went to the ruling Tutsis – as did jobs in the army, police, civil service and other significant institutions. Out of sight, outdoors, are the soldiers who guarded every state building in Burundi. The soldiers are Tutsis, guarding their enemy – the Hutu official inside.

Thus the old quotation about intervening in conflict applies: if you think you understand what’s going on, you’ve not been paying enough attention.

From intervening in conflict, to judging whether we make a difference: the complexity remains. It can be easy to monitor the quality of an intervention – did we work effectively, did we help them explore the situation, did the intervention proceed as we’d intended? Less easy is to evaluate the difference we made: does everyone agree whether a full settlement was reached, and how did relationships change? Hardest of all – not least because it takes time to become visible – did we make a positive long-term impact on the people we worked with, on their conflict and on their community?

Each stage, monitoring, evaluation, and impact assessment, requires a slightly different mindset and a different set of questions and tools to gather and assess evidence, and that is the detail contained in the plenary presentation.

Learning for effective practice

I was the plenary speaker at Mediation North’s autumn conference in November this year.

My intention was to help staff and practitioners be clear on the difference between monitoring, evaluation, and assessing impact – and to show how mediators can bring in the evidence that services need to show their success.

It was something of a risk to make a plenary speech on the subject of monitoring and evaluation!, so I was pleased with the positive feedback fom the plenary and from the subsequent workshop. If you want to download my powerpoint, please visit the Framework site – this is the link.

My sense was that people welcomed a structured framework in which to think about how they demonstrate effectiveness – and the slightly different mind-sets that are needed when approaching monitoring, evaluation, and impact.

A recent Charity Evaluation Service report stressed the importance of evaluating for learning, not just for accountability. This matches a theme close to my heart. How far can an external evaluation have as an outcome the development of the organisation’s capacity to monitor and evaluate itself?

In other words, how can the evaluation leave the organisation in a better place to continuing its self-assessment and learning? Here is a paper, Evaluation for learning, in which I have put some of these ideas together.

Experiential learning

I was yesterday part of a group looking at how to promote whole person learning.

I’ll say more about whole person learning in future posts, but for the moment I want to report on the process we used. My sense is that it could be used by any group of reflective practitioners who are looking for ways forward to develop a work area or interest.

I use whole person learning in the methodology below, so substitute WPL for whatever issue you’re interested in.

The process took about two and a half hours, with a break as well.

We started with the decision that until we know what we’ve got within us on WPL (“whole person learning”) we can’t know what we can offer.

So on an individual basis, take up to 30 minutes to answer these questions:
1. What is my first/an early experience or inkling of WPL. When, where, what – build a sense of that experience. What were its implications or consequences – lasting value or only seemed mundane at the time.
2. How is that experience similar to, and different from, WPL if we were to be developing it now.
3. What was involved in the experience that makes it WPL?
4. What are my questions about WPL?
5. How would I use WPL in our work together?
6. What areas of theory and practice do I need to attend to?
7. If I’m more able to promote WPL in my work, what would it help me accomplish?

Share in pairs, 10 minutes each.
Back in the whole group: Share what came out of the pot.
Then whole-group discussion: What’s our resulting agenda, and priorities, and agreeing action.

The primary agenda item was for us to work on a two-day course for facilitators of whole person learning; again, I’ll keep you informed about this.

As with some experiential learning experiences, it was easy to feel at times that it was all going too slowly! But the depth of our sharing in the whole group would not have happened without that individual and pairs sharing which underpinned our later discussions.

If you try out this process in your own work or groups, do let me know how you got on.