In recent years, there has been growing acceptance of impact evaluation methods that are not based on a counterfactual model of causality. This provides fertile ground for a range of rigorous, non-experimental approaches, such as Contribution Tracing.
Contribution Tracing, conceptualised by Barbara Befani and Gavin Stedman-Bryce, is based on the principles of Process Tracing and Bayesian Updating. Process Tracing is an established social science method that enables strong causal inferences to be made within a single case, by ‘tracing’ the observable implications of causal mechanisms.
The evolution of Contribution Tracing was spurred by several observed weaknesses in currently available approaches to validating contribution claims. Such approaches are methodologically neutral, providing no guidance to the evaluator on what evidence to seek out, or how to assess the strength of evidence, if observed, in relation to a contribution claim.
In response, Contribution Tracing uses the principles of Process Tracing combined with a branch of mathematics called Bayesian Updating. Contribution Tracing takes the guess work out of applying the Process Tracing tests and provides a rigorous way to quantify confidence that an intervention has contributed to an outcome; something evaluation commissioners have been increasingly calling for.
Based on Bayes Theorem, Bayesian Updating is a method of statistical inference which we use in Contribution Tracing to update our confidence in a contribution claim, as we observe (or do not observe) items of evidence. The Bayes Theorem is used to calculate our posterior confidence in a contribution claim based on our prior confidence - set at 0.5, the Bayesian “no information” tradition – and a likelihood function which relates to the difference between the true positives rate (Sensitivity) and the false positives rate (Type I Error).
In Contribution Tracing, the Sensitivity of an item of evidence relates to the probability of observing it, IF the contribution claim is true. From Process Tracing, Hoop Test evidence is an example of evidence with high Sensitivity. Our expectation of observing Hoop Test evidence is high, assuming the contribution claim is true. Therefore, not observing such evidence, lowers our confidence in a claim.
The Type I Error of an item of evidence relates to the probability of observing it, IF the contribution claim is NOT true. The higher the Type I Error (value closer to 1), the less unique that item of evidence is in relation to the claim under investigation. In Contribution Tracing, therefore, we focus on identifying evidence with low Type I Error (value closer to 0). This is akin to Smoking Gun evidence in Process Tracing, as evidence with low Type I Error is unique to the claim under investigation.
Contribution Tracing applies the principles of Bayesian Updating to focus the evaluator on identifying evidence with the highest probative value. The larger the difference between Sensitivity and Type I Error for an item of evidence, the higher its probative value. Thus, the search for evidence is highly focused, making best use of finite resources.
It is early days for Contribution Tracing but we are hopeful that it can bring a greater level of clarity and rigor to impact evaluation.
You can read more about Contribution Tracing here.