Funders guidance - Assess

Assess

This page covers data analysis – the process of bringing together the evidence that has been collected, and making sense of it in order to understand how much change has happened, for whom, and why. This collation and analysis will be carried out at two levels:

  • By grantees or evaluators working directly with them and reporting data and findings to you
  • By you as a funder collating and analysing the data you collect directly from grantees or beneficiaries and in collating and analysing the data reported to you by grantees across your funded programmes

In addition to covering key steps in analysis, such as comparing groups and making sense of change, this section also explains the questions that will need to be asked in order to get an objective picture of overall impact. This includes thinking about the other factors that could have contributed to outcomes and impact, as well as considering how long changes last for, and how much change might have happened independently of the work you have funded. It also involves considering whether bias by those collecting data or the way data was collected could have influenced results.

For funders there are specific difficulties in getting proof that it is their funding that is causing changes and therefore in ‘attributing’ impact. Firstly, most funders are dependent on the standard of evidence provided by their grantees. ‘Before and after’ data that strongly suggests a link between the work carried out and the changes that beneficiaries experience may be sufficient evidence.

However, if you need to demonstrate the link between the funded work and changes observed, for example in order to influence policy change or to demonstrate successful models of practice to other projects or organisations, you may need higher standards of evidence and to fund grantees to engage more technical methods.

Secondly, it may be difficult to distinguish the precise type and quantity of outcomes that can be attributed to your specific funding in a situation where your grantee may be receiving resources from a number of different sources, either as part of core funding or for specific services.

3.1 We store and use people’s data safely, respectfully and legally and encourage our grantees to do the same

As part of collecting evidence, you may handle sensitive personal information about individuals and your grantees are even more likely to. This personal data must be managed in accordance with the Data Protection Act if you or your grantees are to fulfill legal responsibilities in terms of keeping data safe and using data appropriately.

There is a large amount of useful guidance available on how should handle personal data in a way that meets the Data Protection Act. Briefly, the main points of the Act state that personal data should be:

  • Kept accurate and up to date
  • Used only for the purpose for which it was originally collected
  • Not transferred outside the European Economic Area without adequate protection
  • Kept for only as long as is necessary
  • Only gathered if strictly needed
  • Kept safe from loss, damage and unauthorised access

This criterion is met in full if:

The way you store and use the data you collect for impact measurement meets the legal requirements of the Data Protection Act in full. Your grantees are aware of their obligation to meet the same legal requirements and supported to do so if collecting information on your behalf.

What next?

If you’ve met this criterion in full, you could improve your practice by:

  • Developing a written policy on data protection and its relevance to impact measurement
  • Developing and sharing a written policy on data protection will create a helpful resource for making sure that everyone in your project or organisation understands the procedures in place around the legal storage and use of personal data. You could encourage your grantees to the same.
  • Routinely sharing your data protection policy with the people you collect information from
  • Explaining your policy to individuals will help them to understand your responsibilities towards them as the data handler, as well as their own data rights as impact measurement participants.

3.2 We have IT systems in place that allow us to input, collate and analyse the data we require quickly and easily

Making sure that data about your funding, your grantees, the work they are doing and the difference it makes to beneficiaries, can be analysed and reported quickly and easily is a crucial part of ensuring impact measurement is supported by staff across the project or organisation. This is particularly important for funders who deliver a range of projects and programmes. You will need strong IT systems for your own work; as a funder, you can support your grantees to have appropriate IT systems in place to manage the information that they are collecting as well.

There are a growing number of project management systems and outcome reporting platforms that have been developed to support projects or organisations to help manage and measure the impact of their work. These include bespoke databases and IT systems that have been developed for specific types of project or organisation and online platforms that integrate with third party data collection, management and reporting tools. Funders should decide which is most appropriate for them and ensure staff are supported to use them to manage their work so that data is readily available for the wider purposes of demonstrating impact and contributing to service improvement.

Charities and social enterprises delivering frontline services should be prepared to invest at least one per cent of their operational budgets in their monitoring and evaluation infrastructure (IT and database development and management). This will ensure that they have the best chance of becoming proficient in managing data, reflecting on it and using it to inform their practice. You can consider this cost in any applications for funding from your grantees.

This criterion is fully met if:

Your own data entry is quick and easy, secure and can be aggregated and analysed to demonstrate impact and inform service improvement. Grantees are encouraged and supported to put appropriate IT systems in place to store and analyse information in the same way.

What next?

If you’ve met this criterion in full, you can improve your practice by:

  • Storing your data in a way that makes checking for missing or incorrect data quick and easy.
  • Checking for missing or incomplete data is far easier to do once your data is all collated in one place. This will help you to understand and improve your data quality.
  • Storing your data in a way that means you can automatically generate reports

There are obvious time savings involved with automatic reporting. Automatic reporting will help you to prepare the information for analysis and to work with your data much more easily.

3.3 Different types of information are compared to make sense of how and why changes occur

This criterion focuses on the way in which evidence is brought together during analysis to explain how and why your funding makes a difference. This is important for the data analysis that takes place at the level of your grantees and any data analysis that you carry out yourself of collated data that is provided to you or any data you collect directly.

Explaining why changes occur is vital to understanding your impact and you should encourage your grantees to present this sort of analysis where possible, as well as looking for such explanations in your understanding of your broader impact. Using data to explain change requires two steps to analysis. Firstly, each type of data should be analysed separately to make sense of it. Secondly, different pieces of information should be brought together during analysis to try and explain why changes happen.

This might involve bringing together different types of data about the same beneficiaries. For example, you could encourage your grantees to compare quantitative data about how many people experienced a particular outcome with qualitative data from interviews where the same people provided in-depth information about how they experienced different outcomes and why they thought those outcomes were achieved. If you are carrying out impact measurement across a funded programme, a similar analysis would apply and data could be compared from different grantees working with the same beneficiary group.

You may also want to compare different perspectives on change to try and understand why outcomes occur. For example, if you were funding young people, you might ask for data to be collected from the young people themselves about outcomes, and from their parents and carers as well. Examining the way in which the two groups give different or similar explanations for change will help you to get a more in-depth picture of why outcomes occur.

This criterion is fully met if:

Different types and sources of data are compared and contrasted in a way that helps you to understand why beneficiaries experience outcomes as a result of your funding.

What next?

If you’ve met this criterion in full, you can improve your practice by:

  • Encouraging grantees and those collecting data for you directly to assess the strength or weakness of their findings by making explicit which different data sources support or contradict each other
  • Reviewing the way in which different sources of information support or contradict each other will give a sense of which findings are strongest. It will also help to identify where data is still inconclusive and where more information might be needed.
  • Encouraging grantees and those collecting data for you directly to explore initial findings with key stakeholders during analysis
  • Exploring contradictory or inconclusive data with stakeholders at different levels of data collection will help to fill in any gaps and develop analysis and interpretation.

When you are analysing and interpreting your overall impact, discussion with your staff, grantees, partners and experts can be useful to improve your understanding. This process of presenting and discussing your findings to build meaning during analysis is called iteration and is a useful tool for adding quality and depth to your findings.

3.4 We check to see if different groups of beneficiaries experience different amounts or different types of change as a result of our funding

This criterion looks at the way outcome data for different groups of beneficiaries is compared and contrasted during analysis, to find out whether different groups experience different changes as a result of your funding.

Trends and patterns in outcome data for the whole of your beneficiary groups may hide important differences between different groups, so it will be helpful to encourage grantees and others collecting data for you directly to make any differences explicit. Separating out data for different groups in order to compare and contrast the amount and type of change that they experience – also described as disaggregation – is an important step in analysis, since it is crucial to understanding whether or not your work is reaching everyone in the target group.

Disaggregation will often throw up important questions and learning points. For example, if particular ethnic groups or age bands are achieving better outcomes from the funded work, does this indicate learning about the work itself, highlighting a particular strength, or a weakness? Services may not be relevant or appropriate for everyone in the target group.

This criterion is fully met if:

Outcome data for different types of beneficiary is compared and contrasted in order to assess whether different groups experience different amounts or types of change.

What next?

If you’ve met this criterion in full, you can improve your practice by:

  • Encouraging grantees and others collecting data directly for you to draw in different sources of information and different perspectives to explain any differences between groups
  • Including disaggregation in the way you and your grantees analyse data to understand change (see 3.3) will add depth to the way you understand your impact.
  • Discussing findings about the differences in outcomes experienced by groups with staff, grantees, partners and experts

Taking unexplained differences in outcomes achieved back to those who have experience of working with the target group will help you to collect additional perspectives and information to explore and explain how and why groups experience different outcomes.

3.5 Negative and unexpected outcomes, as well as positive outcomes, are carefully considered

Considering findings objectively is a tricky but fundamental part of data analysis. This criterion focuses on whether or not analysis gives equal weight to positive and less positive findings.
The danger is that, without objectivity, the data can end up reflecting a rosy picture of what you and your grantees hoped to find, rather than an accurate picture of how things really are. This can mean that any negative or unplanned outcomes are ignored during analysis. Grantees may be reluctant to share difficulties, setbacks and disappointing results with the funder as they might feel it could adversely affect the relationship. Challenging this assumption involves creating an atmosphere of trust and encouraging honesty in reporting.

The most important learning may rest with negative or unplanned outcomes that are discovered. It is important that these outcomes are given due weight during analysis, so treat negative or unexplained outcomes with the same amount of care and attention as positive outcome data. This means spending the same amount of time on analysis, and asking the same key questions about why changes occur, who or what else is involved in creating change, and how experiences differ between groups. This applies whether you are measuring the impact of capacity-building funding to community groups, for example, where you may be collecting outcomes data directly, or whether your grantees are doing this analysis in relation to their end beneficiaries. Specifically asking your grantees to report on this can begin to build this fuller understanding.

This criterion is fully met if:

Negative and/or unexpected outcomes are analysed just as carefully as positive and planned ones and your grantees are asked to report on this.

What next?

If you’ve met this criterion in full, you can improve your practice by:

  • Encouraging careful thought about how methods of data collection can influence findings

Reviewing the way information was collected will help objective judgements to be made about how important the findings (negative or positive) really are. This includes considering any technical problems during data collection or issues with missing data, as well as reflecting on how the bias of those collecting the data might have influenced the way data was interpreted.

3.6 We recognise that there are other factors that might have influenced the outcomes that beneficiaries experience

In order to fully understand your impact, your analysis, and that of your grantees, you will need to consider the question of attribution – that is, make an assessment of how much change was down to the funded work, and how much was down to other factors, including the services provided by others. This may be difficult to do; practical considerations and available resources will be important in decisions about using experimental methods, such as randomised control trials. Many funders may look for strong evidence of a link between funded activities and outcomes rather than evidence of direct cause and effect.

Change is complex: in the context of your funding, there will be a number of other agents involved in creating outcomes, including other organisations, other professionals and even beneficiaries’ families and friends. Your funding may also be only part of a complex resourcing of a piece of work or inter-related services. This makes it difficult to come up with a precise figure or percentage of the change that is due to funding alone. However, being able to recognise and describe the role that others play in achieving outcomes for beneficiaries is an important part of data analysis. Considering which other players were involved in creating change will allow you to estimate your contribution to overall impact.

It is possible that some of the changes that beneficiaries experience would have occurred ‘naturally’, that is, independently of the funded work. The amount of change that would have happened without the intervention is also known as deadweight. Understanding how much change would have occurred anyway will get a more accurate picture of the contribution to change made by the intervention. This analysis can be done more realistically at the level of individual examples of funded work, rather than across a funded programme.

This criterion is fully met if:

You encourage grantees to use what they know about the involvement of others in creating change in order to draw sensible conclusions about what other variables have affected their outcomes. You also encourage them to consider what might have happened anyway, without their intervention.

When you draw together information on your funding you consider the contribution you make to positive impact.

What next?

If you’ve met this criterion in full, you can improve your practice by:

  • Encouraging your grantees to collect detailed information from beneficiaries about how other people or factors support or work against positive change; when your grantees are your direct beneficiaries, you ask similar questions.
  • Asking beneficiaries about what proportion of the changes that they experienced was down to the specific services, compared to the influence of other people or factors, will help to develop a more in-depth understanding of attribution. You and your grantees might be able to use this information in the future to develop partnership working and design of funding the development of services.
  • Considering the way in which broader socio-economic factors support and hold back change
  • Considering wider socio-economic factors which are out of your control – for example, widespread unemployment or climate change – will help you to understand your impact in its wider context.
  • Providing the resources to compare your outcome data with a control group.

Collecting data from a control group for specific funded work will help you to establish a more accurate picture about how much change would occur without the intervention that you have funded, and to gather a higher quality of evidence to demonstrate the impact of the work.

3.7 We can describe how outcomes from our funding relate to economic, social and environmental issues (for example, climate change or unemployment)

Depending on the type and amount of your funding, it may be important for you to demonstrate to your trustees and other stakeholders that you get good value from your funding. In some cases you may want explicit evidence of the return in social outcomes of the funding ‘investment’ made.

You may wish to encourage and support grantees to make the connection between the outcomes achieved for groups of beneficiaries as a result of your funding, and the resulting offset to ‘costly’ societal problems, for example unemployment, environmental damage or offending. If grantees present outcome data in this way it will help demonstrate how the funded work could save money for taxpayers and the government. This can be a powerful tool for explaining the value of the funded work and your funding itself to your board but also to government, partners, and the public.

Economic evaluation is not a new idea and there are many different approaches to choose from, some of which are more complex and time-consuming than others. Simply considering potential cost savings during data analysis and describing how and where funded work impacts on costly economic, social and environmental issues – rather than coming up with a specific number for costs saved or avoided – will add depth and value to findings on impact.

For example, if the outcomes from funded work include reduction in landfill waste, you could describe the cost of landfill and highlight how these outcomes have contributed to reducing this cost. Similarly, if your funding contributes to reduced reoffending or a reduction in serious offences, findings on impact could include a summary of the costs involved in court proceedings and custodial prison sentences, and a description of how the funded work can result in some of these costs being avoided.

However, if you wish to measure all of your impact – including social, environmental and economic changes – in financial terms, or you want information about the potential savings that result from your funding, you will need to consider resourcing a more sophisticated methodology – for example, cost effectiveness analysis or Social Return on Investment (SROI).

This criterion is fully met if:

You are able to describe how the work you fund relates to economic, social and environmental issues, and where the outcomes you achieve for beneficiaries could potentially contribute to cost savings.

What next?

If you’ve met this criterion in full, you could improve your practice by:

  • Describing the value of individual examples of funded work or discrete projects in financial terms by using monetisation

Monetisation involves setting a proxy financial value for each outcome area achieved – whether economic, environmental or social – in order to be able to describe overall impact in financial terms. This analysis can be done more realistically at the level of individual examples of funded work rather than across a funded programme. This methodology is used in SROI.

Resources for this section

Using IT systems to input and analyse data quickly and easily

Inspiring Impact’s Impact Hub lists database and case management sources for voluntary organisations and of relevance to funders.

Thinking through how much change might have occurred without our work

The Cabinet Office’s Guide to Social Return on Investment is a guide to undertaking SROI. This resource also provides useful guidance about thinking through the other factors that might have influenced outcomes (attribution) and the changes that could have occurred without your work (deadweight).

Useful websites

Managing ICT is a section on the NCVO website dedicated to effective ICT for voluntary organisations which is also applicable to funders. This includes information on budgeting for and buying a system, training and support, and planning for IT. There are also a number of resources to download such as ICT publications, research and blogs.

 

Sign-up to the principles

We need your help to put principles into practice

Sign up to our newsletter

Stay up to date with all of the latest from Inspiring Impact by signing up to our monthly newsletter