Verena Wagenhofer has been part of AVL since 2018, where she focuses on corporate research coordination, developing programme and policy recommendations, fostering R&D networks, and managing strategically important Coordination and Support Actions. In this interview, she shares her perspectives on Work Package 1 of the LeMesurier project, discussing how the team is assessing 2ZERO’s Direct Key Performance Indicators (KPIs) and offering a glimpse into some of the key findings to date.
As the lead of WP1, you are establishing a framework to assess 2ZERO’s direct KPIs. Could you explain the key aspects of this framework and how it aligns with the objectives set out in the SRIA?
We set out to create a framework for the assessment of the Direct KPIs of the 2ZERO Partnership. Direct KPIs can be measured by referring to single-count parameters. When you look into the 2ZERO SRIA, you will find out that this applies to more than 40 of the listed KPIs. So we had our work cut out for us. We knew that for the assessment we would need to collect information from three different types of sources: public, around 30 active 2ZERO projects & CINEA/EC. So, there you have it, assessing more than 40 KPIs by gathering inputs from different sources, which included more than 30 funded projects, required both a well-thought-out methodological approach and process.
In order to deepen our understanding of the KPIs and scout for public information that is broadly available, we started out with some desk research and gathered what relevant information we could from public sources. With what we now knew, we spent some time creating a comprehensive project questionnaire that would enable project coordinators to provide targeted feedback about their projects’ contributions to the Objectives, measures of success and targets defined in the 2ZERO SRIA. With the help of project clusters and Project Officers at CINEA, we distributed the questionnaire to the 30+ projects, gathered their feedback and evaluated it over the course of a couple of months. We clarified some remaining open questions with projects during online interviews. One of the final pieces of the puzzle was obtaining aggregated data from periodic and continuous reporting from projects. This was needed to determine the value of a few statistical data KPIs.
This quantitative approach was complemented by us doing a deep-dive analysis on the example of a group of six projects, where we qualitatively assessed their impact pathways and contributions to the 2ZERO objectives.
LeMesurier is analysing 2ZERO projects to determine their contributions to the Partnership’s SRIA. What has this analysis revealed so far? What challenges have you faced in gathering and interpreting project-specific data?
Generally, we are very pleased with projects’ responsiveness. Everyone has been very eager to contribute to our mission and some have even stayed in touch with LeMesurier teams for continuous exchange and validation of findings. Now that the initial assessment has been completed, we are quite happy to see that 2ZERO projects have significantly contributed to the achievement of the 2ZERO Objectives and KPIs. Around 85% of the assessed Direct KPIs have either been achieved or made significant progress!
Of course, when we got started, we were also aware of some of the challenges that would be ahead of us. Understandably, projects were a bit hesitant to provide us with preliminary results and data. To eliminate doubts, we clearly pointed out that LeMesurier was not assessing project progress or assessing project impact but instead translated the feedback to aggregated data coming from the entire 2ZERO project portfolio. Another challenge was the potential mismatch of timing between a two-year project, such as LeMesurier and many other 2ZERO projects with different durations in different years, producing their results at different times. LeMesurier could only assess the status where it overlapped with active 2ZERO projects and oftentimes we were referred to future months and even years where relevant results were expected.
What are the next steps for WP1 in the coming months? How do you see the KPI framework evolving, and what impact do you hope it will have on shaping future 2ZERO projects?
We are currently working on making the high amount of information from the initial assessment a bit easier to digest by visualising the feedback and progression of Direct KPIs. This is a rather exciting step for us, because we can see the past months of data collection and interaction with projects come to life in graphs and summary statements and transport a message that is easily grasped and understood.
In the next few months, we will spend so time and effort to transfer our methods, procedures and findings to the 2ZERO Partnership, so that future monitoring actions may be facilitated and eased. The close and continuous collaboration with 2ZERO throughout the work on LeMesurier has been a fruitful platform to exchange ideas and develop them further, review drafts and improve methods and procedures, so that they meet future requirements.