Measuring Innovation in the United Nations Development Programme

Funding innovation for sustainable development; what have we learned…

This post shares some lessons from the UN Development Programme (UNDP) on how to design a lean innovation fund that doesn’t create additional hurdles for innovators. UNDP works in more than 160 countries on some of the world’s toughest challenges: poverty, climate change, gender equality, democratic governance and peacebuilding. With partners across the globe, we work toward achieving  the 17 Sustainable Development Goals (SDGs), which came into effect in January 2016 and will guide funding until 2030. We have been investing in innovation since 2012. Through the generous support of the Government of Denmark, UNDP offices were able to test and scale innovations that, for example, helped increase the effectiveness of the global response to the Ebola crisis through mobile payments for relief workers, scaled a shared-value solution for electronic waste in China, supported and governments with launching public policy labs; which have successfully influenced policymaking and re-designed public services. 

Along the way we learned some lessons, and iterated our funding approach and metrics.

The speed of delivery is not a useful proxy for agility

We aimed at supporting our offices with testing their hypothesis quickly through direct interactions with users, through co-designing concrete prototypes to de-risk investments. To accelerate this, we requested proof of disbursements of funds within eight weeks. This rather short time frame was meant to inspire early action with users. However, in many cases activities were designed that were not embedded in longer, outcome-oriented processes such as one-off hackathons. Realizing our misalignment, we decommissioned the speed of delivery as an indicator.

“The speed of delivery is not a useful proxy for agility” -@bkumpf Click To Tweet

The single intrapreneur might go fast, but usually not far

In the early days of our fund, we created demand by presenting new ways of working in country offices. As a result, entrepreneurial colleagues approached us, a cohort of early adopters emerged and many successfully led proof-of-concept initiatives. But in many offices their work remained isolated, and some failures happened due to the general lack of operational dexterity solo intrapreneurs often experience. From venture capitalists we learned the importance of ventures: diverse teams with complementary skills-sets. For us, the main capabilities a team needs to bring to the table are: the ability to navigate power and politics, to market the concept and to raise follow-on resources, to lead user-centric processes and design experiments, and to articulate a business model to deliver the solution at scale.

“From venture capitalists we learned the importance of ventures: diverse teams with complementary skills-sets.” -@bkumpf Click To Tweet

There is no universal metric to measure the impact of innovation

Every innovation fund needs to clarify how success can be measured. We agreed from the get-go to not report on what we consider to be vanity indicators such as the number of proposals received or ideas proposed at a co-design event; the number of staff participating in innovation activities including trainings; the number of toolkits developed or the number of innovation blogs published.

One of our key lessons is: there is no universal set of metrics to measure the impact of innovation investments across all UNDP’s thematic areas. Many development and humanitarian players focus on “actual/projected number of lives saved and improved.” In our case, this metric would only capture part of the impact. For instance, UNDP Georgia supported the government to redesign the national emergency services to make it accessible for persons with disabilities. Our office led a co-design process with many stakeholders, which identified pain-points and developed testable prototypes. Fast-forward one year, the emergency services were redesigned on a national scale. This led to the launch of Georgia’s Service Lab, which has contributed to the redesign of numerous public services, and the establishment of digital service centers. There was a profound change on how the government operates. Merely counting the number of people directly affected by the redesigned service would not reflect the full impact of this institutional change. Therefore, we agreed on the additional indicator: the potential to influence policy/systems change.

“One of our key lessons is: there is no universal set of metrics to measure the impact of innovation investments across all UNDP’s thematic areas.” -@bkumpf Click To Tweet

Metrics that put a focus on the novelty component of innovation are not sufficient

Until last year, we split our investments along a 40-40-20 framework, as depicted below. This, however, is insufficient as it doesn’t adequately budget for mainstreaming new ways of working organization-wide. A team or fund with the mandate to inspire new ways of working needs to measure the success of mainstreaming innovation, to get an organization in a state of permanent beta.

Innovation investments should support incremental and radical innovation

Our Strategic Plan calls for innovation beyond incrementalism, gizmos and gadgets, but our innovation metrics don’t adequately reflect this. They don’t fully measure progress toward fostering systems-change processes requiring a multitude of interventions. Many innovation funds focus on supporting single innovations, and support novel technologies such as blockchain, or approaches such as behavioral insights. This can be a hindrance to supporting systems change and redefining paradigms.  

Based on our lessons, here are some iterations moving forward:

Measure mainstreaming

We will test metrics that monitor the institutionalization of innovation. Our office in Rwanda just institutionalized a mandatory budget for iterative design and qualitative inquiry for all new projects. Yet, no global metric captures such mainstreaming success thus far.

Below are a few indicators we will test:

• Percentage of country offices that institutionalize foresight and horizon scanning functions
• Number of operational policies and financial vehicles launched based on results from country-level experiments
• Percentage of (senior) managers who have a dedicated goal in their annual performance assessment that is predicated upon ‘risk expectation’ and which outlines a highly ambitious experiment as a ‘if, then’ statement

Test right-fit monitoring models

Currently, data on impact is primarily self-reported by UNDP offices. Gradually offices are adopting more rigorous mechanisms such as randomized control trials (RCTs) and other quasi-experimental methods. While we are building institutional infrastructure for this, we acknowledge such rigorous methods are not always the right thing to do.

Moving forward, we will test the applicability of M&E models to gain (near) real-time insights in order to iterate quickly. This includes testing IPA’s Goldilocks Toolkit and the Lean Data model, designed by the Acumen.

Also, most data collected describe “backward-looking impact” or near real-time impact. This very same data can be harnessed to enable forward-looking learning. We will test simulation and modelling approaches to be more responsive to emerging needs and more cost effective in meeting those needs.

Develop metrics to measure systems-transformation

For us innovation means combining systems thinking with ethically designed experiments. One pathway to translate this into action is to formulate ambitious, yet measurable missions derived from the SDGs. Several of the incremental innovations we support have the potential to contribute to systems change, yet we cannot claim causality as one player in a complex system. Mechanisms to contribute and monitor systems change include sensemaking, network building across sectors, experiments, co-creating movements and political bases for policy changes.

Moving forward, we will revise our investment framework to reflect dedicated budget lines for incremental and transformative innovation, as well as for mainstreaming. Knowing there is no golden rule to split innovation portfolio investments along core, adjacent and transformational initiatives, we will start with 30% of the fund for transformational innovation. A specific window will be earmarked for innovations that target previously unreached individuals at the base of the pyramid. No one player, nor sector alone can solve the social, environmental, political and economic challenges of our time. Get in touch if you are interested in co-conspiring.

“No one player, nor sector alone can solve the social, environmental, political and economic challenges of our time. Get in touch if you are interested in co-conspiring.” -@bkumpf Click To Tweet

Lean Impact RSS

Thanks to Benjamin Kumpf for contributing this piece. He leads the UNDP Innovation Facility. Follow him on Twitter @bkumpf. This post is an abbreviated version of the blog Innovation metrics for human development – what have we learned and was inspired by the essay Developing Humanitarian Innovation Impact Metrics: where do we start by the UNHCR Innovation Team.