Tag Archives: monitoring

Critical ALNAP

The  Active Learning Network for Accountability and Performance in Humanitarian Action (ALNAP), established in 1997, is probably the most respected inter-agency body looking at accountability in the humanitarian sector.  So, you can imagine how I responded to this update on the Humanitarian Performance Indicators Working Group in their annual report:

Launched in 2009, the ALNAP Working Group  on Humanitarian Performance Indicators set out to provide a forum for members to share their experiences and thoughts on approaches to organisational and programme performance indicators within the humanitarian system. After discussions in the group about the difficulty of establishing a clear direction, it was decided to dissolve the group for the time being and to incorporate discussions on performance indicators into the development of the methodology for the next iteration of ‘The State of the Humanitarian System: Assessing performance and progress.’ In 2011/12, ALNAP will decide whether there is additional value in reconvening the Working Group

If you can stifle your laughter or manage not to choke on your coffee then you’ve done well.  The group convened, discussed difficulties and the dissolved within two years.  Reminds me of another critical issue.

Advertisements

Leave a comment

Filed under Uncategorized

Can’t give, won’t give

In the last few years a lot of effort has gone into evaluating the effectiveness of charities. The organization Giving What We Can emphasis that cost-effectiveness in charities is essential:

…even restricted to the field of health programs in developing countries, research shows that some are up to 10,000 times as effective as others.

The site divides charity giving into categories, such as health, education and empowerment.  Within the health section,  the site gives a nice breakdown of disease priorities according to the effectiveness of their treatment (largely based on the DCP2 from the Disease Control Priorities Project), where effectiveness is measured in disability adjusted life years per dollar spent.  All well and good.  However an interesting thing happens when you click on the Emergency Aid section:

It is a very difficult intervention type to assess, because it is so wide-ranging, there is little quantitative data, and because each emergency is unique. At this point in time, we have no information on the cost-effectiveness of different types of emergency aid, but we hope to rectify this in the future.

For a more detailed analysis of an intervention-type, see our pages on health.

For all their work, this charity cannot find any data to estimate the effectiveness of emergency relief.

There is no basis on which to separate emergency and health programs.  Both are concerned with saving lives, they have the same outcome of interest.  Emergency conditions appear differently to us, the giver, but to the patient who is dying the conditions don’t seem that different.  Yes, we see/hear/feel more about emergencies than ‘normal’ health problems and emergencies have a sort of ‘Biblical’ significance, but from the point of view of a child dying of malaria, or the mother of a malnourished baby, I don’t imagine there is a distinction.

What this site and others show, in high contrast, is the peculiar position of humanitarian aid in our thinking.

Leave a comment

Filed under Uncategorized

Measuring

Outcomes that require contact with the people being helped are rarely measured during or after the delivery of humanitarian aid.

Consider this from the opening paragraph on ‘Impact’ in Lord Ashdown’s Humanitarian Emergency Response Review (HERR):

The emergency response sector does not routinely assess impact. What this means is that for some operations it is almost impossible to say how the assistance helped people, or even whether it helped them.

After noting that measuring anything in a crisis is indeed more difficult than normal, he suggests that much of the resistance to proper monitoring and evaluation is because,

it is convenient not to have to do it; the need for high visibility action, shaped by donors and the media; the competition between agencies and the lack of serious consultation with the people in need; the failures in leadership. Perhaps most important, impact assessments require time and resourcesthatare not trivial, and to date few donors have been willing to support these efforts on the humanitarian side of the system

And finally, just to complete the picture, he says,

…evaluations tend to evaluate activities rather than impact and do not often enough include a participatory element so that disaster survivors can hold agencies to account.

Which really takes us to the crux of the issue.  Where outcomes are measured, they are in terms of output – food, medicine, shelter – and not results.  Imagine that you are running a sales business and you have to decide which of your sales people to give the most resources.  They are allowed to negotiate on prices and make a hard sale.  Carefully, you count up the number of units each of your 20 salespeople have sold; and look at that, there’s quite a range!  So you give more and more of the best products to those whose sales figures are the highest (and even a little bonus).  Then you sit back and wait to get rich…

But you don’t get rich, and in this context the reason is obvious: revenue.  Your salespeople are driving down prices to shift resources beyond the point of breaking even.  But who can blame them, after all, shifting resources is what you measured and used as the basis of reward.  You’re bankrupted.

The banality of this example is most of the point; outcomes should be measured on a combination of revenue and costs (the not so mysterious idea of profit) and not on either alone.  But it goes further.  Businesses do not choose to suffer under the tyranny of the revenue/costs balance; that balance is the definition of the proper use of resources.  It is not something cold and ruthless, it is the simplest rational measurement of outcome that we have.

Perhaps humanitarian agencies think that they are measuring results: food delivered, check.  Shelter built, check.  They’re not.  And we know this because the outcomes measured do not require actual contact with the people the agencies are there to help.  Weight-gain? Health? Well-being? Alive? These are the outcomes we should be interested in, not processes, not our costs.

The next stage is to explore why things are like this, why we’re not doing things properly.  Lord Ashdown has gone some of the way to explaining it but I believe that it is a lot more complicated.

Leave a comment

Filed under Uncategorized