A few weekends ago my friend Sam, who works with Save the Children, was talking about his experience of working in Yemen. Sounded like a bit of a nightmare (‘kat’ or ‘chat’ a particular problem) and he said that it really didn’t help that many Yemenis have a perception that they are in a developed country (brought on, he thinks, by the proliferation of smart-phones and other superficial baubles of modernity). Because of this, they’re not too fussed about changing how they’re living. In response, he suggested the following two-point development index:
1. What do you eat?
2. Where do you sh*t?
If the answer to the former is ‘nothing but [insert low-nutrient carb]’ and the latter is ‘in a field’, then you’re not living in a developed country. In fact, when I mentioned this to some Nigerian colleagues they thought that question 2 might be sufficient.
My tongue is only lightly in my cheek in saying this. In India more than 50% of people defecate in fields, and yet India parades itself as an emerging nation – not developed, but taken seriously on the world stage. Of course, India also fails on a number of other metrics, but open-defecation is unequivocally bad for people and pretty easily solved. All nations, and maybe all concerned westerners, should strive to meet my friend Sam’s two point index. It’s a good place to start.
Michael Clemens and Gabriel Demombynes recently published this summary paper of what happened and what they thought about the Millennium Villages Project debacle. It makes interesting reading, and expands to cover their approach to impact evaluation and its limitations. Something that is not covered, however, is a discussion about data.
Simon Brooker – an epidemiologist working in Kenya with experience of working with economists on trials – once said to me that ‘epidemiologists care about data, economists care about analysis’. The more that I read, the more that I think that this is broadly true. If this dichotomy is the case (and I have not taken a systematic approach to this, so, ironically, my data is very poor) then I am sure that it has emerged from the different human genres that the disciplines have grown up in: epidemiologists dealing with messy things like health, and economists with more precise and less subjective things like money. Nowadays the disciplines are overlapping. Economists who are sharp as tacks when it comes to analysis, such as Clemens and Demonbynes, seem conspicuously obtuse on discussions of sampling, measurement bias, recall biases, and missing data. Such biases, especially in the context of an impact evaluation, can be sufficiently large as to make subsequent fiddling with statistical models all but irrelevant.