12 January 2023 10:00-12:30hrs CET (GMT +1)
The third PIE online dialogue focused on the partnership challenges and opportunities in assessing effectiveness and impact for accountability and learning purposes. During the dialogue, representatives of Dutch Institutes for post-secondary education and TVET and partners from around the world discussed these challenges (and solutions) under the guidance of resource persons from donor organisations, international consultants and experts in monitoring & evaluation. To further stimulate diverse perspectives, particularly from francophone colleagues, simultaneous translation was provided (English – French), and the spoken language in one of the breakout groups was French with English translation.
The programme was as follows:
Findings from the survey
As a stepping stone for the online dialogue, all participants received an invitation to fill out a short survey. A total of 73 respondents started the survey, and 35 (largely) finished it. During the dialogue, the main findings from this survey were briefly presented.
1. Countries
Respondents are stationed in ten different countries, with the Netherlands (8 respondents) and Myanmar (3 respondents) being the most important.
Figure 1. Countries where respondents are stationed
2. Relevance of assessing impact/effectiveness
Overall, the relevance of assessing the impact/effectiveness is considered higher for own learning and improving the way of working than for being accountable to donors.
Figure 2. Relevance of assessing impact/effectiveness for … (in # respondents)
3. Ways to assess effectiveness/impact
But also: ‘we often don’t assess the impact or effectiveness’
4. Challenges
5. Confidence in assessing impact/effectiveness?
Nearly 50% of the respondents scores a ten (very confident) on the confidence ladder of assessing the impact/effectiveness of their work.
Breakout groups
From the survey among partners and a preliminary discussion among PIE members, we selected five topics to organise breakout group discussions around. Resource persons (experts) were assigned to these groups based on their experience, skills and organisational/institutional exposure.
Five breakout groups
| Field | Short description | Expert & facilitator |
| a) Developing the evaluative agenda and dealing with conditionalities | This relates to the demands ‘imposed’ by others on your programme and how this might influence the assessment process. These others may be your (back)donors or your direct partners with whom you implement the programme to be evaluated. Still, they may also include your ‘target group’ (e.g., students, researchers, lecturers), governments, or institutional actors (e.g., university boards, NGOs). | Paul de Nooijer (IOB) & Konstantina Velkushanova (IHE Delft) |
| b) Implementing | This relates to all things needed to execute a high-quality impact/effectiveness assessment. As such, it covers elements such as capacity, preventing biases, ensuring a clear purpose is defined, ways to deal with attribution, having a clear methodology, ………. | Janat Lukwago Namirembe (IFDC TBC) & Coen Buvelot (KIT) |
| c) Learning from | This relates to the internal learning process within an organisation and/or partnership based on the outcomes of an impact/effectiveness assessment. How can such results be taken up in future programming and planning? How to prevent negative results from being downplayed or ignored? | Cobi Mars (MFA) & Joost Teuben (UT/ITC) |
| d) Communication on | This relates to the communication about the outcomes of impact/effectiveness assessments internally (within the organisation and/or the partnership) as well as externally (among those directly involved in the evaluation or those affected (positively or negatively) by the results of the assessment). How important is it to, for instance, broadcast evaluation findings and to whom should you broadcast them? What to do with negative results? What are good ways of communicating (e.g., social media, official meetings, reports, ….). | Beatrice Safi Kalumbi (MDF DRC) & Mike Zuijderduijn (MDF NL) |
| e) Creating support | This relates to defining and obtaining support within and outside your organisation for doing an impact/effectiveness assessment and how it will be conducted. It includes resourcing, financially and in terms of how people are involved in planning and executing the assessment and the process of reflection following the evaluation. | Landry Fanou (WUR/CDI) and Simone van Vugt (WUR/CDI) |
Outputs of breakout groups
Each breakout group discussed the assigned topic through a short introduction by the expert, which was then be validated and enriched by all participants in the breakout group. The following questions acted as guides for the group discussion:
1. what are the main challenges or problems participants see or experience in this field? And
2. in what concrete ways can we tackle these challenges and problems?
All challenges and solutions are logged in an online tool and a maximum of two solutions were presented in the following plenary (see overview on the following page). First reactions in this plenary came from experts from IHE and MDF, and a representative from the Dutch Ministry of Foreign Affairs
Main challenges and solutions from breakout groups
| Group | Challenge | Solution |
|---|---|---|
| 1. conditionalities | Focused too much on quantitative evaluations (KPIs) but not on qualitative (long-term impact) | The impact evaluation framework should be set in the beginning of the project, not in the end (as the main driver of the project) It is more than setting an evaluation agenda => design of a new cooperation agenda |
| 2. Implementing | Natural separation between M&E and the project implementation | Train the implementing staff to implement the M&E framework. Design, technical support and quality control can still be done by a dedicated ‘expert’ but day to day implementation by project staff |
| 3. Learning | How to assess impact if not designed properly in advance? | Continuous learning and adaptation during the project implementation phase should complement mid- and end- evaluations and impact assessments |
| 4. Communication | How can you present assessment results of a complex programme in a way that people understand and accept those results? | Add a sense-making step in between data collection and reporting to ensure recognition and joint analysis of findings => stimulates understanding / acceptance / internalisation of results |
| 5. Creating support | Need to recognise evaluation as a learning process and address cultural perceptions and religious issues related to the process | Need conscious leadership, inclusive, respecting diversity (way of speaking) across the globe |
Plenary
During the panel discussion the main challenges and solutions were discussed with Jeroen Kelderhuis (Head Civil Society and Education Division, Dutch Ministry of Foreign Affairs), Damir Brdjanovic (Professor of Sanitary Engineering at IHE Delft Institute for Water Education), and Beatrice Safi Kalumbi (trainer and consultant at MDF Training & Consultancy in the Democratic Republic of Congo).
Beatrice Safi Kalumbi states that the M&E theme is challenging internally and externally. Crucial for her is availability: not always the right persons who have access to information and are in the position to share can be consulted. M&E team and management team have different goals. (Mis)communication and power dynamics play an important role.
Jeroen Kelderhuis declares that donor and implementers are in the same ship. Donor institutions are often seen as the ones to please, but that should not be the priority. Base line is important. He also emphasises the need to ‘be more relaxed’. As ‘we don’t know what will happen during the journey […] we need room to change things during the programme/project’. Besides, it is not only numbers that count and the focus should shift more to the journey itself. And while he emphasises the need for external assessors/evaluators, he also states that reaching an impact ‘in 5 years’ time is quite difficult’ and that ‘long term programmes of 10 years may be needed’. Other issues he touches upon:
- Students are seen of ambassadors for the Netherlands, but that was not the goal of the programme. It’s an important outcome though and should be recognised as such
- Shifting the power in North-South relations requires donors to give more space to local partners and their needs. Besides, we should be mindful of our language and listen better. An Internal working group within the Ministry (chaired by Jeroen) is working on locally led development/localization.
- Local context, looking further than the capital of the country is a challenge in that regard. It starts with the design.
- We should realise that we are working in framework given by politicians: they are the ones who can make real changes.
Damir Brdjanovic is enthusiastic stating that ‘this is the fastest learning experience in my life’. Participating in this dialogue felt like being ‘Alice in Wonderland’. In the breakout groups he noticed misunderstandings between projects and programmes. He states that development should be seen as a process during which we should be able to change direction. Whereas deadlines are important, impact assessments should be considered for the short term, midterm and long term. Perhaps we should also change the image of impact and the wording and find more friendly wording. Donors and implementers are partners and are in the same ship and should work together. Damir has experience with the Gates foundation and is inspired by the network, but feels there is not enough capacity. Capacity building is thus needed, and he will devote 2023 to increase impact skills and would like to make a guidebook on impact assessment: process and implementation with a glossary of terms. As a group we could make this together.
Lindy van Vliet closes the third online dialogue of PIE by inviting participants reach out to PIE if they want to be involved in potential follow up actions of individual PIE members. PIE will make an attempt to link people to each other as much as possible. Notes of the meeting will be shared as soon as possible. We thank all for their active participation and conclude that we share a strong motivation to learn from our own work and that of others to strengthen future interventions.