Evaluating ColaLife Operational Trial Zambia – COTZ

[The following article was written by Rohit Ramchandani & Jane Berry and was published this week in UNICEF’s internal M&E Newsletter EvalNews. It looks at the issues we have faced designing the evaluation framework for the ColaLife Trial in Zambia (COTZ)]

COTZ is funded by DfID, J&J Corporate Citizenship Trust, COMESA and Honda with in-kind support from SABMiller
For more information, see https://colalife.org and the blog https://colalife.org/blog
By Rohit Ramchandani & Jane Berry

The COTZ project combines innovative design of a new product – an Anti-Diarrhoea Kit – with private sector delivery, to test to what degree access to essential diarrhoea medication may be improved in underserved rural communities. The process of designing the evaluation for this pilot has presented a number of interesting challenges thus far, given the relatively new/unique health market approach being tested. Continuing with the projects “mantra” of open innovation, we wanted to share some of our early experiences with our UNICEF colleagues, who have played an instrumental role in the project’s development.

ColaLife is a UK charity that has brought its innovative concept to Zambia for a first operational trial. The design, planning and baseline phase began in December 2011 with a 12-month implementation phase due to start in September 2012. UNICEF Zambia has been involved since the early stages in supporting the development of the project, and has supported the design and commissioning of the Monitoring and Evaluation (M&E) work package, working closely with ColaLife’s Public Health Advisor, Rohit Ramchandani, Principal Investigator and Doctor of Public Health candidate at Johns Hopkins Bloomberg School of Public Health.

The COTZ project builds on 3 years of innovative concept development and design involving local and international stakeholders, including 8 months’ co-design with implementation partners in Zambia. It will test the feasibility of piggy-backing Anti-Diarrhoea Kits (ADKs), for children under five, on SABMiller’s (Coca-Cola’s bottling partner in Zambia) secondary distribution chain using ColaLife AidPods, plastic triangular containers that fit into the empty space between crated bottles of cola.

WHO recommendations (2009) for a ‘diarrhoea treatment kit’ for all new mothers with guidance on use, and using ‘market forces’ and ‘innovative delivery strategies’ to widen access. The purpose of the trial is to increase access to simple medicines (Oral Rehydration Salts and Zinc – ORSZ) in underserved rural communities. ORS has been widely known for many years in Zambia, but Zinc as a combined therapy for diarrhoea has not. Whilst health posts and clinics serving rural populations distribute ORS for free, it may not be in stock and many mothers/carers may need to walk long distances, incurring various opportunity and financial costs. Pictured below, the ADKs will contain 8 sachets of ORS (which each make up 200ml of solution – the first of their kind in Zambia), 10 paediatric formulated zinc tablets, IEC materials, a small bar of hand soap for prevention messaging and as an incentive, as well as a small bendable straw. The attractive container itself can be used to measure and mix the solution, and can be re-used later as a simple storage device.

Early design plans for the evaluation were to pursue a cluster randomized trial (CRT), widely accepted as the gold standard for evaluating the effects of interventions to improve health where groups of individuals (e.g. communities) are randomly allocated to the experimental conditions under study (i.e. treatment and control groups), as opposed to individuals. It soon became clear that the degree of control we would require to conduct a CRT would be in contention with the free market aspects of the supply chain system we were attempting to test. Our intervention – of supplying ADKs through private, community-level retailers via district-level Coca-Cola wholesalers – meant that our highest level of control was at the district/wholesaler level. Attempting to select clusters below this level (at ward, or Standard Enumeration Area level for example) would have been difficult, since we will have little control over which retailers will come to purchase ADKs from the Coca-Cola wholesaler, and were hesitant to try and control this too much.

With only two intervention districts and two control districts (which were narrowed down using pre-defined inclusion/exclusion criteria), randomly allocating the intervention would have added little statistical advantage. Under this scenario, claiming true randomisation would have been somewhat artificial, and the fact is that there were important, practical reasons for pre-defining two of the districts for our intervention arms. The willingness/level-of-participation of the independent, district-level Coca-Cola wholesaler, for example, has played an important role in the project design and set-up phase. While this will inevitably have implications for external validity, it is an important lesson learned, and something we felt would have to become part of the program design during any potential scale-up. One of the main lessons learnt in this regard, with reference to making design choices, has been that it is more important to be transparent, capture and share ones reasoning, and pursue the process in such a way that it can be replicated, rather than to artificially attempt to adhere to rigorous methodologies with poor adherence. This is particularly important in public health research.

The COTZ pilot will thus utilise a plausibility design. These appraisals go beyond adequacy assessments by trying to rule out potential confounding factors which could potentially cause any observed effects . These types of assessments attempt to control for influence of confounding factors by choosing control groups before an evaluation is begun or during analysis of the data. In this case, two districts have been selected as comparators for our two intervention districts, and have been matched on a number of pre-defined criteria. Using a pre-post comparison design, the intervention and control districts will be compared at the beginning and end of the program cycle. This helps result in much more plausible conclusions than if no control was used. In many ways, plausibility assessments are akin to the ‘natural experiments’ most common in public health (Habicht et al.).

An impact model was developed early on in the evaluation process. This instrument continues to serve as an important communication, planning, management and measurement tool. The model is divided into three key streams of work which reflect the multiple approaches that make up the intervention, namely: A) the supply chain stream; B) the knowledge, attitudes and practices stream; and C) the knowledge translation stream. Taken together, activities from each stream are expected to contribute towards various levels of impact, namely improving availability of ADK contents, improving access of those contents, improving utilisation of the contents, and ultimately, decreased child mortality from diarrhoea, the second leading known cause of child mortality worldwide. Deciding how far up the impact chain to measure was another important evaluation design decision that had to be made. Given the required timelines and resources associated with effective measurement of mortality impact, we felt that analysis at this level would be out of scope for the current pilot. Measurement will therefore end at the utilisation level (e.g. proportion of children with diarrhoea who used ORS and zinc).

We are now preparing for baseline collection before the end of August, and then beginning full rollout of the ADKs in September. Key methods will include household and retail surveys, review of sales/voucher-redemption data captured through mobile phones, key informant interviews, mystery shoppers and more. With these new stages of the project cycle come whole new sets of decisions that must be made. How to best define our primary sampling unit (we will use the Standard Enumeration Areas defined in the last national census)? How to randomly sample households (we will employ a random walk sampling method, which is a probability sampling approach where every kth household is selected)? How to appropriately capture relevant contextual factors (e.g. other diarrhoea-related programs operating in the districts, seasonality, water sources, etc.)? We are also utilising Geographic Information Systems (GIS) for spatial decision support leading to faster, better, and more robust understanding and decision-making capabilities (i.e. location of households, private retailers, SEA boundaries, etc.). This is a relatively new application in public health and is still considered to be in its infancy. Accordingly, we expect the challenges, opportunities, and lessons learned to continue flowing. Like most new applications, there is a lot of promise, but also a number of potential pitfalls that must be avoided along the way.

You can follow our progress along the way by visiting www.colalife.org. We will continue to share our experiences, challenges and triumphs as we go. We welcome your evaluation-related comments, thoughts and suggestions as well. They can be directed to: roramcha@jhsph.edu.