DCED Webinar Measuring Results in Challenge Funds

WebinarFrontPicOn 23 January 2014, the DCED organised a Webinar on Measuring Results in Challenge Funds. The webinar addressed questions such as: Why measure the results of challenge funds? How can challenge funds measure their impact and learn from experience, while keeping results measurement costs proportional? What has been learnt from the results measurement experiences of recent challenge funds?

The DCED Secretariat presented guidance on Measuring Results in Challenge Funds using the DCED Standard for Results Measurement, and Amanda Jupp from Coffey International shared the experience of the Enterprise Challenge Fund (ECF) in the Asia-Pacific. This was followed by a Q&A session. On this page, you can access the full recording of the webinar, Powerpoint Presentation, as well as additional useful materials. We have also answered Frequently Asked Questions, which were raised by participants before and during the webinar, below.


Webinar resources

You can view the full recording here, or by clicking on the video below.


 


 

Frequently asked questions

1. How to design results chains for a challenge fund project?
You can find guidance on how to articulate results chains in challenge funds as well as a number of practical examples in the DCED Guidelines.The DCED has also published a separate Guide on How to make results chains, which can be downloaded here. For further resources on results chains, including case studies, click here.

2. Do you suggest designing the results measurement system and requirements with your client in order to create buy-in from the client?
Yes!

3. How do you develop logframe targets at this early stage or do you wait until you have the first stage of proposals so you have more ideas of type of projects?
The problem here is that we don’t have enough data from different challenge funds to be able to establish any benchmarks. Ideally, we would be able to say that, in a given context, we could reasonably expect xxx outcomes. That means that logframe targets can be arbitrary; the projections based on the specific proposals are going to have a better foundation.

4. Has the ECF’s use of the DCED Standard been audited?
Yes, the ECF’s use of the DCED Standard was audited in 2013. You can download an audit report here. ECF also published a paper on their experience of using the DCED Standard and being audited, available here.

5. What is the minimum timeframe to monitor development impacts of a challenge funds? 
This is likely to be different for different businesses. For example, suppliers to an agricultural processing business might experience increased incomes in the first growing season. However, users of financial services might take much longer to see a benefit. In order to monitor this, develop a results chain that shows how you expect the customers/suppliers of the business to benefit, and discuss this with the grantee to understand when impact is likely to be observable. In all cases, longer-term monitoring might be necessary to assess sustainability and systemic changes.

6. If you are focusing on stars for measurement, how do you ensure that you are learning from the failures and you are using the learning to make management and steering decisions?
You need to monitor all projects, not just the stars; however, you monitor them in different ways. From ‘failures’ (or projects that turn out not to be commercially viable), you want to understand why they failed, and what lessons can be learnt about market development. You will need a good understanding of the business activities and outputs, and why these were or weren’t achieved. For the stars, you will probably want the same information but also to understand what impact these business had on their customers/suppliers.

7. What are we required to measure, and how can we measure results without over-burdening the businesses and while keeping our costs in budget?

This is a matter between the challenge fund and the donor; the important thing is that the donor appreciates the actual cost of collecting the information that they would like to have.

8. How can results be measured in a cost-effective, or even low-cost, manner; and truly involving partners and beneficiaries?
Results measurement needs to be budgeted for; perhaps the biggest problem in the past has been expecting too much from under-resourced monitoring systems. Challenge Funds typically base much of their results measurement on data provided by partners, but are necessarily somewhat removed from beneficiaries. Engaging regularly with beneficiaries costs too much for the typical CF budget, especially when projects are spread out geographically or located in remote rural areas.

9. Would you be able to specify the approximate costs associated to results measurement in relation to the overall programme spent?
There isn’t much published by challenge funds on cost data which might help us answer that question. Looking at experience from other private sector development projects, 5-10% of the total budget is recommended to be spent on monitoring and evaluation. In challenge funds, the percentage spent on results measurement will vary according to the size of the grant. Results measurement costs are often fixed (for example, visiting the business), and so will be the same regardless of whether the grant is large or small. Consequently, smaller grants have proportionately higher results measurement costs.

10. How to standardize results measurement among different projects and sectors?
A standardised results measurement system risks being ineffective. If you standardise all indicators across a diverse portfolio, you will end up asking the wrong questions and failing to ask the right ones.

However, you can’t reinvent the wheel for each new grantee. Consequently, you need to use a standardised framework which can be tailored to allow effective results measurement of each project. In the DCED Standard, results chains are a standardised tool; but a new results chain for each business will show exactly what you need to monitor in each case. Indicators should be systematically developed, using the same basic principles each time; however, the exact indicators used will be different for each business (although there may be some universal indicators which all business report against).

11. How small is a small sample. 25+25 or 200 households?
The answer depends on a number of factors, such as the size of the change you are looking for (small change requires bigger sample). For more information, please see here (and in particular the Practical Guidelines for Conducting Research, Chapter 6 on Sampling Strategy).

12. What approaches are being used to tackle attribution and are any of them any good?
There are various types of attribution, and many ways in which they are being tackled! There are probably two key attribution questions that every challenge fund needs to address:

  • Project-level attribution: This is the question of whether the observed benefits would have occurred without the project.
  • Input Additionality: This is the question of whether the grantee would have made the same investment without funding from the challenge fund.

For more about project-level attribution, consult the DCED Guide to attribution, and chapter four in the DCED Guidance to Measuring Results in Challenge Funds. To summarise: it is difficult, there are lots of possible approaches, and in most cases it will be possible to estimate attributable impact.

Input additionality is much harder to demonstrate – but equally essential if you want to show that challenge funds are an effective mechanism. The DCED is developing guidance to assessing the additionality of public-private partnerships, which is currently in draft form and will shortly be published online.

13. Can challenge funds trigger market distortions?
All subsidies distort markets; the question is how to maximise the beneficial effects for the poor, while minimising the negative effects.

14. How to maintain market based principles in the CF? How to right size the expectation in relation to systemic change for challenge fund programs? Challenge Funds are often launched to harvest the innovative ideas of business (rather than simply to obtain co-financing). Sometimes, they are launched as the development agency is explicitly seeking new ideas. Developing a comprehensive strategy, based on a sound analytical framework, may not be conducive to maximising creativity. The issue is how to get the best of both worlds (creativity and holistic understanding) while avoiding the worst.

15. How to measure the indirect results from challenges funds, both positive and negative?
This is a fascinating issue, and there is a long-running debate about whether challenge funds should expect to have significant indirect results at all. In general, clearly define what indirect results you expect to have, and reflect this in results chains. Consider how you might monitor each step in the results chain, and who might be able to give you the relevant information. Try to triangulate data from several sources; for example, key informant interviews with secondary data. When assessing indirect impact, take an investigative approach; ask qualitative questions and follow up emerging themes to get a fuller understanding of the types of change that are occurring. There is more information in chapter five of the DCED Guidance to Measuring Results in Challenge Funds.

This is certainly an area which needs to be explored further in future, and it is encouraging to see more challenge funds examining their indirect impacts. In particular, it would be great to see more challenge fund results measurement system exploring potential negative effects, such as distortion in the financial markets.