tom aston

How to avoid 'toolsplaining': thinking differently about social accountability

Guest blog by Tom Aston

On the plane to Accra just over a week ago I read Rebecca Solnit’s Men Explain Things to Me (the origin of the term “mansplaining”), and it struck a chord with me. A colleague from Kenya who hadn’t heard the term before asked if there was such a thing as “white-splaining”. And, indeed, there is. But, recently, I’ve been concerned with another phenomenon: “toolsplaining”.

 

“Toolsplaining” is, as far as I can see, the phenomenon where we over-explain how clever a particular tool is, but forget to explain how (in reality) it interacts with context, and works together with other strategies and processes to achieve change. We often assume, usually due to lack of information (or lack of careful investigation), that whatever tool we used must explain the change – that this tool (the scorecard) or that tool (the citizens’ charter), for example, was the cause of the change.

In practice, especially for social processes, how that change happens is generally more complicated, and nuanced. There are typically multiple causal pathways even within our own strategy that work together to influence change (and of course, various complementary factors driven by other actors). And it’s often the informal micro-politics that matters, rather than your formal process.

So, we need to think differently.

How is our intervention contributing to change?

I was in Accra to support the five-year USAID-funded Ghana Strengthening Accountability Mechanisms (GSAM) project which aims to “strengthen oversight of capital development projects to improve local government transparency, accountability and performance.” In particular, what CARE wants to understand better is how our intervention is contributing to district assemblies’ responsiveness to citizens’ concerns in relation to the planning and implementation of capital investment projects.

We used contribution tracing to define a hypothesis and identified causal pathways for how change happened, rather than merely what the log frame says, or what a process map suggests ought to happen. To do this, the team looked at the process that was designed (see the graphic below), but then traced back real changes (e.g. district assemblies replacing inadequate building materials) as a causal chain.

Scorecards formally hinge on a public meeting (an interface meeting). But, on various occasions, we believed that changes had been triggered even before we’d held a public meeting (6 or 13 in the graphic above), but after we’d conducted site visits to monitor the quality of infrastructure (2). We’d established District Steering Committees composed of government actors, community leaders, and engineers (invisible between 1b. and 2b.) which were seemingly able to resolve some (but not all) problems without district town hall meetings, or even scorecard interface meetings.

Tracing the real process has therefore helped us think again about how, when, and where we really might have influenced change.

Inter-related pathways to change

Rather than a single pathway of information sharing, or knowledge transfer, it was clear we had at least four inter-related change pathways for social accountability:

  1. providing financing to civil society organisations who prepared a district scorecard to get district assembly members to respond;
  2. getting district assembly members to release data and participate in the process;
  3. supporting citizens to monitor priority infrastructure projects and presenting their findings to authorities, and;
  4. creating new spaces for dialogue between citizens and district assemblies about capital projects.

The team are now going to go out to find evidence to support their claim about how their strategies influenced change. But, I just wanted to underline some of the learning:

  • Define terms (eg transparency, accountability, responsiveness) precisely so you know what change you’re actually going to measure and what data is relevant to your hypothesis.
  • Interrogate your assumptions periodically. Allow different staff members to challenge your logic. Don’t just rely on proposal writers or project managers.
  • Don’t bundle everything together. Or else, how will you understand the relationship between different components of your hypothesis?
  • Make sure your hypothesis is in order. Remember, logical steps follow chronologically...
  • Don’t toolsplain. Don’t get distracted by your hypothetical process maps or steps in your tools: in other words, consider the evidence, not what you hope your tool influenced.

Capturing complex change: is it really all about confidence?

Guest blog by Tom Aston

For those of us that sing the praises of social accountability (citizen-driven initiatives to hold those in power to account), making a claim about “impact” (or transformative change) is a challenge we face on a daily basis. And CARE’s not alone. The title of the first session at an NGO political economy analysis working group at which I’m presenting this week (“Building the Evidence-base for Social Accountability”) speaks to the same concern.

Some argue that we need more longitudinal studies. Others advocate the use of Randomised Control Trials (RCTs). And a recent study CARE conducted on the influence of Community Score Cards on reproductive health-related outcomes in Malawi shows that RCTs have a place, and do demonstrate that social accountability makes a difference.

But, once you consider that outcomes are behavioural changes of real (complicated) people, you quickly see, as Marina Apgar recently suggested, why we need to move “beyond measurement of linear pre-defined change and intervention-effect alone and [use] mixed-methods to help us understand emergent complex social change.” Social accountability outcomes (such as mayors changing budgets to benefit poorer areas, or even procuring a new ambulance) don’t fit neatly into boxes. They rely on our capacity to influence behaviour, and this is behaviour we can’t (fully) control. So, we need to better explain HOW change happened, not merely to assert that it did. 

Recognising this has led CARE to explore various theory-based methods such as Most Significant Change and Outcome Mapping. With a particular emphasis on the change process, we are now piloting Contribution Tracing with Pamoja Evaluation Services in Ghana and Bangladesh to help us better understand CARE’s contribution to social accountability outcomes.

Contribution Tracing is all about increasing your confidence in making claims about impact. Essentially, you make a “claim” about your intervention’s role in achieving an outcome that really happened (your contribution), and then find evidence to defend your claim.

To do this, like other theory-based methods, you need a hypothesis (a proposed explanation) about how you think change happened. You then review the connection between different steps (or components) in that process

You identify evidence that would help support (or undermine) your proposed explanation using the four tests of Process Tracing (Straws-in-the-wind, Hoops, Smoking Guns, Doubly Decisive).

What matters is not how much evidence you have, but how good that evidence is to help confirm that each part of your proposed explanation for your claim really exists (“probative value”).

In Contribution Tracing, you use Baysian (Confidence) Updating to assign a probability (how likely it is) that the various components of your contribution claim exist; and ultimately whether your claim holds true. You then update your confidence after gathering data precisely tailored to your claim (increasing or decreasing the probability using the four tests), compare this against rival explanations, and then put it up for “trial”, inviting others in to peer review your claim.

We’re right at the beginning of the journey, but to me, what our learning already suggests is that:

  • You can show your contribution, even when change processes are complex;
  • You can make credible impact claims, without a counterfactual;
  • You can tighten up your loose theory of change as you go along, and;
  • You may not need to gather as much data as you think you do to prove it.

But don’t take my word for it; listen to some reflections from staff on the experience so far. And watch this space for more to come.

Advocacy evaluation: art of the possible

by Gavin Stedman-Bryce

I was asked by Médecins Sans Frontières (MSF) to attend a staff conference in Geneva earlier this year. I had been asked to speak about the importance of monitoring and evaluation of advocacy - a real passion of mine. Alas my diary was full the day of the conference but thankfully the organisers allowed me to attend via video.

This short film, featuring Tom Aston, a Governance Advisor from CARE UK International, touches on why advocacy M&E is not only increasingly important, but possible thanks to a range of innovative methods and tools.