Outcomes that require contact with the people being helped are rarely measured during or after the delivery of humanitarian aid.
Consider this from the opening paragraph on ‘Impact’ in Lord Ashdown’s Humanitarian Emergency Response Review (HERR):
The emergency response sector does not routinely assess impact. What this means is that for some operations it is almost impossible to say how the assistance helped people, or even whether it helped them.
After noting that measuring anything in a crisis is indeed more difficult than normal, he suggests that much of the resistance to proper monitoring and evaluation is because,
it is convenient not to have to do it; the need for high visibility action, shaped by donors and the media; the competition between agencies and the lack of serious consultation with the people in need; the failures in leadership. Perhaps most important, impact assessments require time and resourcesthatare not trivial, and to date few donors have been willing to support these efforts on the humanitarian side of the system
And finally, just to complete the picture, he says,
…evaluations tend to evaluate activities rather than impact and do not often enough include a participatory element so that disaster survivors can hold agencies to account.
Which really takes us to the crux of the issue. Where outcomes are measured, they are in terms of output – food, medicine, shelter – and not results. Imagine that you are running a sales business and you have to decide which of your sales people to give the most resources. They are allowed to negotiate on prices and make a hard sale. Carefully, you count up the number of units each of your 20 salespeople have sold; and look at that, there’s quite a range! So you give more and more of the best products to those whose sales figures are the highest (and even a little bonus). Then you sit back and wait to get rich…
But you don’t get rich, and in this context the reason is obvious: revenue. Your salespeople are driving down prices to shift resources beyond the point of breaking even. But who can blame them, after all, shifting resources is what you measured and used as the basis of reward. You’re bankrupted.
The banality of this example is most of the point; outcomes should be measured on a combination of revenue and costs (the not so mysterious idea of profit) and not on either alone. But it goes further. Businesses do not choose to suffer under the tyranny of the revenue/costs balance; that balance is the definition of the proper use of resources. It is not something cold and ruthless, it is the simplest rational measurement of outcome that we have.
Perhaps humanitarian agencies think that they are measuring results: food delivered, check. Shelter built, check. They’re not. And we know this because the outcomes measured do not require actual contact with the people the agencies are there to help. Weight-gain? Health? Well-being? Alive? These are the outcomes we should be interested in, not processes, not our costs.
The next stage is to explore why things are like this, why we’re not doing things properly. Lord Ashdown has gone some of the way to explaining it but I believe that it is a lot more complicated.