Centre for Development Studies

Assessing Rural Transformations - ART

October 2012 - September 2015

Principle Investigator: James Copestake

Funding Body: ESRC-DFID

Total Value of Award: £365,656

Background to the ART project

The ART project set out to design and test a credible way to assess the impact of multifaceted development activities in complex contexts where other approaches to impact evaluation, such as randomised control trials, are not appropriate. The result, the qualitative impact protocol - known as the QUIP, is a relatively simple and cost-effective way of finding out directly from intended beneficiaries of a development activity what they think are the most significant drivers of change in their lives, livelihoods and wellbeing.

The ESRC/DFID funded part of this project is now completed, and the QuIP methodology is now being developed and supported by an independent research organisation founded by staff members who worked on the ART project. To find out more about application of the QuIP methodology and keep up to date with new developments and publications please see the Quality Social Impact website.

The Principal Investigator of the ART project, Professor James Copestake can be contacted at: j.g.copestake@bath.ac.uk

Publications

Briefing papers and guidelines

Example QUIP reports

Blogs

Articles

Other

Research Methodology and Outputs

The ART project worked with two international NGOs, Farm Africa and Gorta Self Help Africa, to develop and trial the methodology on four rural livelihood projects; two in Ethiopia and two in Malawi. The aim was to add a qualitative strand to the NGOs' evaluations, complementary to the collection of quantitative monitoring data. The quantitative monitoring data allows the NGO to know what has changed, and the QUIP aimed to tackle the question of why that change had occurred, using beneficiary feedback.

A quantitative monitoring tool called the IHM, developed by NGO Evidence for Development, was used alongside the QUIP to measure changes in household level disposable income relative to basic food needs. A baseline study was conducted for each project, followed up by two rounds of monitoring studies a year apart (Strand 1).

The qualitative monitoring (Strand 2) was carried out using the QUIP methodology. Field researchers used semi-structured interviews and focus group discussions to assess impact based on self-reported attribution. Two rounds of impact studies were conducted, a year apart.

There are strong ethical grounds for asking people directly about the effect of actions intended to benefit them, and doing so can contribute usefully to learning, innovation and public accountability. But for responses to be credible it is necessary to address potential response biases. The QUIP tackled this by ensuring that interviewers and respondents were given no information about the project being evaluated. We also developed simple software to make analysis and reporting timely and auditable, as well as accessible to small NGOs. 

The approach recognises that by working alongside routine monitoring of key indicators of change it is also possible to estimate the magnitude as well as the nature and direction of the main causal drivers of change. By relying on self-reported attribution rather than statistical inference to generate evidence of causation it also avoids the need for a control group. Underlying this design is an emphasis on generating evidence that is both credible and cost-effective.

The four projects all aimed to strengthen household level food security in the context of both rapid commercialisation and climate change. A large number of interconnected, uncertain and hard-to-measure confounding factors (Z) affected the casual links between project activities (X) and impact indicators (Y). The QUIP generated evidence of attribution through respondents’ own blinded accounts of links between X and Y alongside Z rather than relying on statistical inference based on variable exposure to X. This was used to generate standard tables showing the frequency of unprompted reference by respondents to different drivers of change, cross-analysed against project theories of change and exposing both obvious gaps where links were expected, and unintended consequences.

Two example reports, one from Ethiopia and one from Malawi, are available to download. These demonstrate the type of information the QUIP can provide alongside quantitative monitoring data.

The outputs of the project were discussed at workshops in Malawi, Ethiopia and the UK between June-September 2015, and work is ongoing to produce articles synthesizing the two strands of data collection.