Guest blogger Lindsay Hodgson from Catch Impact reframes data using the Music Mentor case study and associated Theory of Change, and asks us to think about the data we're collecting and the questions we're asking it.
What comes to mind when you think of ‘data’? Numbers, spreadsheets, graphs?
In this blog we’re reframing data to think about ideas, stories and expectations. We’ll follow the data journey of an organisation we’ve called Music Mentors (loosely based on our experience with a real charity).
Through a series of short animations, we'll see how they:
- used theory of change to explore their ideas of how they bring about change
- thought about different types of data they need to collect to check their expectations
- analysed the data they collected to see what story it told
Theory of change is a great tool to map out the changes you think you’re making (Outcomes) and how you make them (Activities and Enablers). There are some great resources out there to guide you through using theory of change.
Theory of change
Music Mentors run music sessions in prisons where people can learn to play instruments, create music together and build relationships with mentors.
Their vision for doing this is to help people lead positive and meaningful lives, creating an identity away from that of an offender.
But how does this happen? What are the subtleties of their programme? What makes it work and how can they capture information on that?
Watch Part 1 to see:
- What Music Mentors do
- What they hope to achieve
- How they used theory of change to make decisions and map out short, medium and long-term changes
Music Mentors created a comprehensive theory of change detailing all their activities and how they thought these led to positive outcomes.
We’ve simplified it here because we want to focus on the stories that emerge when you undertake this process. Follow the highlighted outcomes to reveal an interesting thread in their story.
Music Mentors – simple theory of change (often described as a logic model in this format)
Something they uncovered was how important it was for people to make mistakes in a safe environment and the value of sitting with that uncomfortable feeling and learning from it.
All too often they saw people reacting with disproportionate anger over making a few small errors. When they dug a little deeper they heard stories like “I’m just rubbish at everything, I might as well not bother” and “everyone’s laughing at me because I’m not good enough”.
The internal tape that was playing in these people’s minds was about being a failure, a lost cause, a disappointment. Many of them had adopted an approach to life which said “if I don’t try then I can’t fail”.
Moving people away from these thoughts and using music as the tool to do that became one of Music Mentors’ top priorities before any other positive changes could occur.
Once they understood this golden thread of their story, Music Mentors explored how they could create an environment that fostered learning from mistakes.
Key ingredients included:
- a non-judgemental environment
- a supportive bond with a mentor and
- the opportunity to try something new
By trying something new they were likely to make mistakes and the mentors could guide them through their emotional reaction to that, teaching them regulation and resilience skills along the way.
Later, we’ll see how they used musical instruments as the tool for change but take a minute to think about your own services and what tools you have at your fingertips. Are there some less obvious ways that your activities are leading to outcomes that you might need to explore?
Music Mentors spent years focussing on providing instruments that people knew how to play so that they could express themselves.
And while self-expression is an integral part of their theory of change, they were missing an opportunity to capitalise on their resilience outcomes by enticing people outside their comfort zone. As someone wise once said: life begins at the end of your comfort zone.
Types of outcomes
To help you with the process of thinking about change, it's useful to consider different types of change (or outcomes).
The table below shows six of the main types of outcomes that we’re often working towards. We’ve included examples from the Music Mentors theory of change to get you thinking.
So, now you’ve seen how theory of change works and how you can think about change, let’s ponder some questions.
Think about the changes you’re trying to make. Ask yourself:
- What do we know / think about the way our service works?
- Can we describe our short, medium and long-term changes like a story with a beginning, middle and end?
- What data can we collect data on this that might help us understand what’s happening?
Types of data
Once you’ve created your theory of change you’ll need to collect data to test your ideas. Are things happening as you thought they would in your theory? Is something else going on?
There are four great types of data that will give you a full picture of your service:
- User data: who is using your service
- Engagement data: how they participate
- Service data: what they think of it
- Outcomes data: what has changed for them as a result
It’s ideal to have all this data linked together (in a database for example) so that you can explore patterns and ask yourself questions about what might be going on.
Watch Part 2 to explore:
- What data Music Mentors collected to show progress towards achieving their vision
- What their data told them
- How they can maximise impact based on what they’ve found out
By organising your data in this way, you can see different stories emerging.
For example, you might find that people over 30 years old repeatedly attended the programme but those under 30 only came sporadically.
Why was this: can feedback in your service data tell you anything or perhaps something in your user data like length of prison sentence?
Did it have an impact on whether short, medium and long-term outcomes were achieved?
Can you set up your programme differently so that it’s more effective for different age groups?
Think about the data you’re collecting. Ask yourself:
- Does it tell me a story?
- Is it giving me information that I can use to test my ideas in my theory of change?
- Is there any other data I could be collecting that tells our story of change better?
Analysing the data
At the beginning of this process, Music Mentors thought that making mistakes was a formative part of achieving success. They:
- built the idea into their theory of change
- then collected data on all of the things they thought went into it
- asked participants to create ground rules that fostered a non-judgmental environment
- tracked repeat attendance and the ratio of mentors to participants to determine whether that was enough to establish a supportive bond between them
- created opportunities to make mistakes by introducing unfamiliar instruments into the sessions.
Once they had this data, they could ask it questions to see what story it told.
Watch Part 3 to see:
- How Music Mentors tested their ideas with the data they collected
- Made changes to their service based on data
- Looked at the subtleties of the mechanisms that created change
Often, we forget to ask our data questions and simply end up accepting and reporting it.
Music Mentors set up processes to collect and analyse data so they could continually listen to the story the data is telling, adding new chapters as interesting plotlines revealed themselves.
They looked at the data they had collected and asked three simple questions:
- What is it telling us?
- Why does that matter (in relation to our theory of change)?
- What more do we want to know (so we can make good decisions and improve effectiveness)?
The example below shows Music Mentors at the start of their journey.
They had always collected data about what instruments people had experience of playing but this time they looked at it in relation to their theory of change.
Lots of people had experience of playing drums so they provided drums. That ticked the box for people to express themselves.
The data showed no one had experience of playing banjos so they used these to engineer a situation where everyone was making mistakes, creating a level playing field and providing a learning opportunity.
Below is an example of the process that Music Mentors went through with some of their data. We’ve included an example from each type: user, engagement, service and outcomes.
Music Mentors asking questions about their data:
Going through a process like this keeps your data collection alive and makes sure you’re looking at the most current and relevant story of your service. You’ll find you begin to collect new data to answer questions that the new stories pose.
It can be difficult to make time and space for storytelling with data but in doing so you can bring your ideas and expectations to life, gathering evidence along the way. This can help you with building your strategy, communicating who you are and what you do and demonstrating your impact to stakeholders.
But most importantly, it means you’re giving the people you support the most effective service possible.
Think about what you ask your data:
- What story is your data telling you?
- Why does that matter (in relation to the changes you expected in your theory of change)?
- What more do you want to know?
Training
Datawise London run training to help small charities and community organisations to unlock the value of their data. This blog is a synopsis of Using Your Data to Evaluate Your Impact, Explore Options and Drive Change.
You can watch a recording of this session here: