The post-2015 debate has kept alive the issue of finding new alternatives to attack poverty. Meanwhile, policy wonks in the blogosphere are quietly but fervently discussing the proliferating effect of randomized control trials (RCTs), a method used to evaluate the impact of ideas, in deciding the shape, colour and size of future social policies. In India, RCTs have found one of their largest laboratories.

 

To give a glimpse, 3ie, a leading proponent of impact evaluation methods, lists over 200 experiments in India, most of them RCTs. MIT’s Poverty Action Lab (JPAL), another leading advocate of RCTs, lists having done about 90 experiments in India. With their impressive networking skills and the claim of providing rigorous evidence, the ‘randomistas’ could have substantial influence on Indian policymaking.

 

What are Randomised Control Trials?

 

Without further ado, let’s attempt to demystify RCTs. Inspired by their pervasiveness in medicine, a section of economists have re-discovered RCTs as a potential game-changer in social policy. A proposed policy, say giving out uniforms to increase school enrolment or distributing double-fortified salt to reduce anaemia, is given out to a sample of people to evaluate its impact. This is the ‘treatment’ group. An equivalent number of people with the same basic features is made the ‘control’ group that doesn’t get the treatment. The impact of the policy on the two groups is monitored and evaluated for a fixed period of time and the results are used to draw recommendations on what works and doesn’t work for policymaking. The list of people selected for the study is shuffled so that their allocation into treatment and control groups is random. Depending on the study, the unit of analysis could be people, or clusters of schools, firms, neighbourhoods or villages.

 

For instance, in an experiment conducted by the World Bank to study the impact of a Total Sanitation Campaign on children in Madhya Pradesh, 80 Gram Panchayats (lowest tier of local government) from 19 blocks were equally allotted to a treatment group and a control group. Around 5000 children under five were covered. A follow-up was done after 21 months. Although toilet coverage increased by 19%, nothing appreciable was achieved in terms of child health. The study says that this could be due to delays in implementation. RCTs, then, are not immune from real world challenges.

 

Randomised trials like these are carried out accounting for many such challenges. For one thing, the participants could simply drop out from the experiment when they feel it doesn’t serve them well. For another, participants in the control group could learn of the benefits of a policy to the treatment group and switch over. Keeping the two groups geographically apart, then, becomes a challenging task in designing the experiment.

 

Non-economist social scientists might be disconcerted by the thought of Stata-results of RCTs becoming state policies. Others might frown that these experiments treat people like, well, rats. The ethical issues have been charged to the randomistas repeatedly, but the rigour of evidence that RCTs claim to provide has so far outweighed these concerns.

 

It would also be worthwhile to see if RCT recommendations are indeed the best. In an experiment, JPAL appointed balsakhis or young teachers in Vadodara and Mumbai, who worked for half the number of hours that permanent teachers did, in order to give remedial education to primary school children. As the balsakhis came at a cheap wage – which JPAL points out is ‘a fraction of the cost of civil-service teachers, one among the projected policy lessons was cost-effectiveness. Nevertheless, a policy replication would mean paying the paltry amount at a national level. The question that then arises is whether appointing cheap wage teachers is the way to go, and also if it is okay to pay paltry amounts to temporary teachers.

 

The External Validity Question

 

The key question that remains is if RCT pilot studies can be replicated in new settings and scaled up at the level of the mammoth that India is. This issue is known technically as the external validity question. “We have all played this game of questioning methodology. No methodology is without its problems,” a British economist said answering a question I raised in a public lecture on his latest RCT results from India.

 

Harvard scholar Lant Pritchett has been a critic of RCTs, with a focus on the aspect of external validity, pointing out that the results obtained in one experimental location are not generalizable per se in different contexts. Pritchett considers the success of the randomistas as ‘entirely faith-based’. He echoes what the anthropologist Andrea Cornwall and colleagues once called ‘development myths’, which drive the industry of development practice in the form of fads that take spotlight at various points in time.

 

‘Context matters’ has been a longstanding slogan in development studies. Thus, when a RCT claims that unconditional cash transfers can successfully replace grains doled out through ration shops in India, old-fashioned social scientists are bound to frown as contextual differences, such as access to bank accounts and share of health expenditure in income, mean a lot to this group.

 

Although the ethical implications of using knowledge as technologies of power deserve serious reconsideration, some RCTs have attempted to reduce the guilt through quirky methodological designs. For instance, in an RCT conducted by the International Labour Organisation to evaluate the impact of providing health insurance in rural India, both treatment and control groups were given access to health insurance cards; simply, the control group was not given any discounts on the insurance premium. The treatment group was found to visit health centres more often, incentivised by the decrease in their out-of-pocket expenditure.

 

RCTs have received the additional push of that crucial player in the development industry – money – with a large number of aid agencies, such as USAID and the Gates Foundation, recognising randomised trials as valuable tools to find evidence. DFID, the British aid agency supported an experiment in 2010 that found community mobilisation through women’s groups to have reduced neonatal mortality by 45% in India. DFID has justified its stance saying that RCTs need not be kept separate from other qualitative and quantitative methodologies, and the critics of RCT might do well if they engage with these experiments more closely. Fair point.

 

 

It is not news for anyone who knows the world of academics that effective networking is considered quite a legitimate method of selling ideas – euphemistically called research dissemination – through conferences, workshops, consultations, publications strategically placed in the right journals, mutual citation, partnerships with governments and forming alliances of like-minded organisations. Randomistas are surely not wrong in following the tradition.

 

As things stand in India, the sarkari (government) sector is yet to be penetrated by RCTs, while the non-sarkari universe, which includes all things NGO, academics and think-tankish, are increasingly being drawn to the popularity of rigorous impact evaluation.

 

The point is not that RCTs are diabolic. Perhaps RCTs could be improved and refined by taking into account the criticisms that have emerged. The point, rather, is that the development sector is hungry for methodological alternatives to attack poverty and has found relief in this particular technique in the absence of others. Small wonder, then, that impact evaluation is steadily becoming synonymous with conducting RCTs.