Building an Evaluation Approach for Evaluating Arts and Soft Power

Note from the CPD Blog Manager: The author is recipient of a 2018–20 CPD Research Fellowship. Learn more about the CPD Research Fellowship here.

The impact of soft power is notoriously difficult to measure due to the subtle ways in which cultural influence and attraction operate over time. Through my USC Research Fellowship, I have been exploring how we might improve the measurement of arts and soft power through developing two evidence tools that could help with the challenge of soft power measurement and evaluation, which could be useful to public diplomacy practitioners in the field.

Public diplomacy practitioners and cultural organizations are inhibited by an inability to conceptualize and measure the outcomes of diplomacy due to the lack of an overarching measurement framework that hinders soft power measurement. My fellowship has explored the development of an "Arts and Soft Power Ecosystem Framework" supported by taking a "What Works" approach to improve soft power measurement.

Arts and Soft Power Ecosystem Framework

A soft power ecosystem refers to complex networks operating within and across a range of scales and time. The ecosystem sees soft power as a process that enables a range of public diplomacy outcomes to be created. This is not a straightforward linear process, but an ecosystem with multiple feedback loops influencing and impacting each of the individual outcomes over time.

Through the ecosystem there is a growing network of organizations that are able to create international dialogue, exchange of information, and values through exchanges of art and culture such as through a festival or season.

The Arts and Soft Power Ecosystem comprises:

• Core inputs, such as soft power assets like arts and cultural organizations, the soft power-enabling environment with a country;
• Soft power as a process that enables core public diplomacy outcomes, such as increased trust, new and deeper relationships, and perception change;
• Outlines for how each aspect or element of the soft power ecosystem could be measured / evaluated; and
• Sources of the emerging, underpinning evidence base for each outcome of the ecosystem.

 
The ecosystem is an intricate web of connections, feedback loops, and a dense network of interactions both internally—with movement of ideas, people, products and money around the whole system—and externally through ripple effects from the ecosystem, which may lead to other non-soft power related outcomes.

Developing a "What Works" Approach for Soft Power

Supporting the measurement and evaluation of the Arts and Soft Power Ecosystem there is a need to share learning and data better across organizations and countries to further develop the evidence base around soft power and the robustness of the evidence base together with the need to develop long-term approaches and long-term, larger data sets to support the evidence and learning.

Most of the available current evidence is at an individual program level held by a multitude of different organizations across the globe. Better comparable data at a system level would aid public diplomacy practitioners, researchers and policymakers alike by furthering progress on the measurement challenge. In that same vein, some deep-dive studies on cause and effect of attempts to leverage specific soft power assets would hugely benefit the field.

Public diplomacy practitioners and cultural organizations are inhibited by an inability to conceptualize and measure the outcomes of diplomacy due to the lack of an overarching measurement framework that hinders soft power measurement.

The What Works approach aims to improve the way government and other public sector organizations create, share and use (or 'generate, translate and adopt') high-quality evidence in decision-making. They act as ‘bridge’ institutions between the producers of evidence (often, but not always, in academic institutions) and the consumers of evidence (public service commissioners and professionals).

A What Works approach would allow us to further build and deepen the evidence base around soft power including exploring ‘how’ soft power works over time in different circumstances, building on the soft power ecosystem framework. This approach would also support the strengthening and the development of soft power measurement and evaluation capabilities of public diplomacy practitioners through the sharing, learning, approaches and emerging evidence.

An aim of a What Works approach is to build a critical mass of knowledge and the capacity to apply this to practice and policy such as around soft power. The creation of a Soft Power What Works center or approach could further examine how soft power works in different contexts, such as different political and economic contexts, and join up public diplomacy evaluation approaches, datasets and evidence across the soft power agenda.

Further research is needed to improve understanding of the precise routes to impact through soft power, the strength and nature of causality in different audiences and contexts, and the durability of changes in perceptions and influences on behaviors. Key to this is obtaining both longitudinal and comparative data, applying analytical rigor and theoretical insight.

Evaluators in this space have to be systems thinkers; view matters from multiple perspectives; respect the ecological interdependence of things; and tell a story that connects what is being evaluated with the wider picture, analyzes and judges in context—and with respect for context, connecting the “micro” with the “macro,” and recognizing order within the apparent chaos.