How to Move the Needle on Moving the Needle

Measurement and evaluation is nothing new to public diplomacy. Nicholas Cull holds that America’s public diplomats have always understood the need to move the needle. However, most historical accounts also agree that the topic has gained considerable traction since 9/11. A glance at the 2014 Public Diplomacy Evaluation Resource Guide shows that especially in the last decade measurement and evaluation has moved to the top of the agenda for researchers and practitioners alike. And the recent works of, for example, Barbora Maronkova, Efe Sevin, James Pamment, Carissa Gonzalez, or the U.S. Advisory Commission on Public Diplomacy all strongly suggest that the issue is still gaining momentum. Much like in neighboring fields, such as public relations and corporate communication, measurement and evaluation is now a key area of exchange between academics and practitioners, opening up opportunities for education, mutual exchange, and joint research.

So far, the field has mostly started to build much-needed knowledge of best practices, models, tools, and metrics for public diplomacy measurement and evaluation. However, this growing literature isn’t just preoccupied with developing solutions for showing impact, proving efficiency, or showing strategic alignment. Rather, some skepticism has accompanied it from the start. Some overarching critical points are:

• The often selective evaluation of few programs (and neglect of evaluation especially of more reactive programs where evaluation is never really part of the greater scheme)
• The tendency to measure output (broadcast hours, column inches, etc.) rather than outcome (impact on strategic goals)
• The strong focus on justification of individual programs rather than doing evaluation for weighting and deciding about a broader strategy
• The common divide of operational and strategic levels though exclusion of public diplomats from the formation of foreign policy

In the more recent literature, these points of critique are often the beginning of an emphatic call to action. The common message in the direction of the public diplomacy professionals is that they need to step up their game, that they should focus more on sophisticated models, measures, and metrics. Some say the field is in need of an “evaluation revolution” to provide stronger evidence of impact, stay ahead of competition over budgets, and clearly link PD activities with strategic goals in foreign affairs.

These calls for change are now a common element of the debate. However, after showing why change is necessary, the interesting question becomes how such change can best be achieved. This is the point where the call for change needs to be supplemented with a close look at the specific challenges that have to be overcome to move the needle on moving the needle. For this, numerous points have been raised in the recent debate:

• Difficulty of providing evidence for impact on long-term intangible objectives (such as relationships, reputation, or trust)
• Extant negative attitudes and reluctance of PD professionals towards sophisticated evaluation; i.e., prevalence of a culture of reporting rather than a culture of evaluation
• Heterogeneity of approaches and activities within the wider PD domain and accordingly specialized and segregated cultures of and approaches to evaluation
• Lack of resources for performing sophisticated evaluation and increasing pressure to do more with less
• Professionals’ lack of knowledge and skills in evaluation models and methods
• Lack of consensus and standards on what to measure
• Lack of interest and support from superior officials

The thing is, though, that so far all arguments about these challenges have been based purely on anecdotal information. What we are lacking are empirical insights on how PD professionals actually deal with these alleged challenges in their day-to-day work. Specifically, we lack evidence on how these challenges affect everyday behavior of practitioners. What is, for instance, the influence of budgeting or support from superiors on professionals’ intentions to focus more on measurement and evaluation? How important are budgeting and support compared to practitioners’ own attitude towards evaluation? What role do practitioners’ knowledge and skills in measurement and evaluation play as a driver (or barrier) to deliver sophisticated evaluation of their programs?

These drivers and barriers need to be explored empirically. If we want to move the needle on moving the needle, we need data from actual PD practice. Based on these insights we can then derive some systematic answers on what needs to be done in the shot, mid, and long term to address the existing challenges. 

Photo by Tim Sheerman-Chase | CC BY 2.0