Measuring Peace: Indicators of Impact for Peace Practice

July 2001

This paper is based on preliminary findings from the case studies and consultations carried out by the RPP Project/Phase I. It is not intended as a final product of the Project. Rather, its purpose is to elicit further thinking, experience, ideas and suggestions from the community of people and agencies engaged in peace practice. From May 2001 through May 2002, a series of consultations and workshops will be carried out to encourage this kind of challenging interaction with the ideas presented here. This process will produce much greater clarity and agreement around the issue and the results from this will be available by July 2002.

I. THE ISSUE

At present, there are no commonly agreed indicators for agencies to measure the impact of their peace and conflict-focused programming.

In other programming areas, such as emergency assistance or development, quantifiable macro-level indicators provide important information on whether a program 'is working'. These include things such as number of houses or schools built, reductions in the rates of hunger, disease, or unemployment. Many peace practitioners believe that similar indicators for 'peace practice' could pinpoint key areas or sectors of society where change can be monitored (and measured) in order to gauge whether their activities have led to progress on peace.

However, most such programs operate on a micro-scale. They work on a specific issue (at a specific level), which is related to but more limited than the problem of 'war' or 'conflict' writ large. The question is, how can we develop indicators that measure the impacts of specific agency peace programming and that also link these to an impact on peace writ large in that context?

1. Impulse for Indicators

The impulse to develop indicators comes from a variety of sources.

• Peace agencies and practitioners want to monitor the progress of their work, to improve it, and also to demonstrate its importance to others.
• Donor agencies want to make sure they are maximizing the impact of the funds they devote to peace by supporting the most effective approaches.

2. Impulse against Indicators

At the same time, many agencies are concerned about the high priority being put on developing impact indicators. They say that, since no one knows 'the right way' to make peace and peace work is undertaken through a wide array of approaches and activities, there is a danger that standardized indicators will be:

• Rigid: cementing certain concepts of how peace is made and delegitimizing others.
• Simplistic: oversimplifying the complexity of the factors and actors in any given context and the complexity of social change processes.
• Diverting: resulting in the energies of peace agencies being diverted to address a 'cost-effectiveness agenda'. Some ask "is this taking energy away from working for peace itself?"
• Limiting: constraining peace agencies to use (and donors to fund) only those approaches that are related to specific indicators, and refrain from pursing other 'untried' approaches that may be more promising (again, since no one yet knows 'the right way' to make peace).
• "Means-Blind": suggesting that any activity which produces the desired outcome is a good thing (potentially including force or threats of force). Some ask "is a focus on impacts obscuring the importance of the means used to get there?"

3. It's Happening anyway

Despite the above concerns, many efforts are focusing on developing impact indicators for peace work. These efforts have taken a variety of forms:

External evaluations of peace interventions
In recent years, many evaluations of peace and conflict-related projects have been undertaken. Some use indicators pre-established by donors. Others use indicators derived from the humanitarian assistance field (sometimes with adaptations). Still others avoid the use of explicit indicators and base recommendations on the evaluator's personal experience of how peace work should be done.

Agency codes of conduct
Some peace agencies have recently developed formal codes of conduct to guide their work. Such codes lay out specific ethical guidelines for peace work, focusing on advancing certain issues or standards. Such codes imply that progress on these issues is an indicator of progress towards peace.

Interactive evaluation methods
Some peace agencies engage participants in an interactive process of setting goals and developing appropriate indicators at the beginning of a project. The goal is to build a consensus around objectives and later standards for monitoring and evaluation.

II. UNDERSTANDING INDICATORS

The term "impact indicators" has become accepted jargon. Yet it is not always clear how an "impact indicator" of peace work differs from an outcome of peace work?

Webster's Dictionary defines "indicator' as something that:
a) points out or points to
b) is a sign, symptom of index of
c) suggests or demonstrates the necessity and advisability of
d) states or expresses briefly

Thus while an outcome is a statement of fact, an indicator points to something else (another process). In this case, it points to impacts of a particular peace/conflict program conducted by an agency. But such programs have a wide range of impacts - positive and negative, intended and unintended. Which impact indicators are we concerned with?

In practice, the drive to develop impact indicators is focused on measuring positive impacts, or whether such programs achieved stated goals. Most peace projects have two levels of goals:

1. The project's micro-level goals in terms of specifically who and what the project attempts to affect (and in what ways).
2. The project's stated (or often just implicit) underlying premise that achieving such micro-level goals will contribute to peace writ large (the macro-goal).

Thus, as the term is currently used, an indicator suggests/points to a particular project outcome and suggests that it signifies progress on peace writ large (or not).

III. DILEMMAS OF MEASURING PEACE

Several distinct but related dilemmas make impacts of peace work difficult to measure.

1. Attribution of Impacts to any Given Peace Effort
Proving that peace activity A caused social outcome B is extremely difficult. At any given point, many activities are simultaneously underway and few peace agencies claim that peace is the result of their efforts alone. More often, an outcome is seen to result from the cumulative efforts of many actors, each "playing a role". Singling out any one intervention as "the cause" of an outcome is often an unhelpful distortion of reality. In fact, the success of any one intervention may depend on the other things going on too!

2. Credibility of Reports of Impact
A related question is - how credible are reported impacts? They often concern changes in people's attitudes, values, relationships and ideas that may help them undertake actions that promote peace - things which are mostly intangible. In the RPP cases, there are two basic 'ways of knowing' about impacts (with different levels of credibility).

Verbal Reports
'What people say', and 'what kind of people say it'. This includes self-reports by agency staff, personal testimonials by participants, leaders or influentials, observer reports, or public opinion surveys. If more people (and different kinds of people) report a certain outcome it "rings truer" than if fewer people do. Sometimes many such reports coincide and there is a common opinion on the impact of an activity. However, sometimes there is disagreement over who or what is responsible for a certain change.

Observable Deeds and Facts
These are visible, tangible actions or steps people take, or events which happen, as a result of a given peace activity. As such, they are usually more credible than mere reports because they demonstrate that 'talk' was followed up by concrete action or change. However, the reasons why people undertook these often are traceable only through verbal reports.

3. Significance of Impacts for Peace Writ Large
Even if it could be conclusively and credibly determined that a peace program was the main cause of a certain outcome, how do we know how significant that outcome is to the achievement of peace? It is especially difficult to come up with common judgments about the significance of impacts because:

• Different peace agencies define peace (the end goal) differently. For example, for some peace means an end to open hostilities, for others it means redress of basic injustices that fuel violence.


• There are many different and competing approaches to how peace is made. Any judgement about the significance of certain activities is rooted in an underlying theory of social change that prioritizes certain developments over others as critical. One agency may prioritize creating new institutions to manage conflict while another may see "building attitudes of tolerance" as the critical first step.

IV. WHAT THE RPP CASES SHOW

Numerous Indicators Exist
Despite the prevailing assumption that impact indicators are hard to identify for work on conflict, the RPP cases showed the many indicators that people are already using. When a wide variety of people in each RPP case was asked "how do you know a given effort is making the situation better?" they all had answers. It is clear that most peace practitioners and agencies implicitly work with indicators of impact in mind all the time. It is how they understand if their strategy is working or not.

Indicators are Context-Specific and Dynamic
The impact indicators reported in the RPP cases were inherently context specific. It was difficult to understand an indicator and its connection to peace without understanding the particular context, and what people in that context saw as 'progress' towards peace.

For example, the signing of peace agreements could be an indicator of a progress towards peace in one setting, but signify stagnation or even a setback in another.

In the first-ever mass peace meeting, a covenant was signed and sealed with traditional reconciliation rituals. The agreements made were subsequently carried out, and further peace meetings and confidence building measures between the warring sides followed.

The signing and celebrating of agreements between elites had occurred repeatedly over many years, yet almost none of the agreements had been implemented. The signing of a new agreement only made the parties more distrustful of each other's intentions. Popular opinion on both sides saw the peace process as just a stalling mechanism.

The RPP cases also show that indicators are not only context-specific, but they are dynamic. Even within one context, what people see as "progress towards peace" can change dramatically over time, and indicators can be specific to a particular "moment".

The amnesty for war crimes was seen by many local people and agencies as necessary to stop the fighting, and achievement of this as a step toward peace. At a slightly later point, the resulting impunity enjoyed by high officials was seen by many people to be an obstacle to lasting peace.

Weak Links between Impacts on Individuals and Impacts on Peace.
Often the cases reported indicators of impacts on individuals, usually participants in programs conducted by peace agencies. Even if such indicators were credible, and clearly attributable to the agency, the significance of such impacts for the dynamics of the conflict overall was often not directly addressed. The cases showed that while sometimes impacts on individuals led people to take actions that had larger effects on the society, more often their larger significance was unclear.

The training changed my life. Suddenly very different issues have become important and my perspective has changed.

Individuals who had participated in the dialogue process had developed good relationships across conflict lines. They kept in contact often by phone or email, even with the resurgence of fighting.

INDICATOR TYPES IN THE RPP CASES

Given that agencies must identify indicators that are specific to their own programs in particular areas, at particular levels, at particular times, what can be learned from the RPP cases that is helpful?

This section will present examples of the most common types of indicators found in the RPP cases, and the strengths and weaknesses of each as they relate to the dilemmas of attribution, credibility, and significance for peace writ large.

1. Changed Attitudes, Communication, Relationships
Many of the RPP cases pointed to instances of changes in attitudes, beliefs, communication, and relationships as important indicators of impact. People reported feeling "personally transformed", "changing their minds" about the other side, 'forming friendships across conflict lines", and "sharing a common language" with former adversaries.

Such indicators mostly represented impacts on individuals, though not always. In one case, activities were reported to lead to "a moderation of the public discourse in the society at large, including increasing emphasis on the future versus obsession with the past". In another, "positive relationships among key leaders on both sides led to "a moderation of the tone of communication in the official peace talks".

The use of such intangibles as impact indicators is vulnerable to all three dilemmas.
Attribution and credibility are challenges in that one must rely on individuals' self-reporting, or subjective assessments of public attitudes. Opinion polling, or systematically monitoring the media are ways of knowing about public attitudes with more credibility. Visible actions demonstrating goodwill are even more credible, showing such changed attitudes and relationships in practice.

Attribution of these changes to a particular agency/program is also very difficult. The low visibility of a peace effort is one factor that may cause people to underestimate the role of a particular agency or effort in an outcome. Conversely people may overestimate the importance of a high visibility agency. As well, people who participate in an effort often view its significance differently than people outside the project.

In the area where local peace teams had been active for many months, members of both ethnic groups reported that "relationships were getting better" - especially with respect to commerce, sport, and community festivals. But when probed for the reasons for this, participants had clear answers that credited, among other things, the peace teams and their work. Non-participants, on the other hand, credited the passing of time and the recent change in government.

Finally, the significance of such changes in attitudes/relationships on the dynamics of the larger conflict is often unclear. Most often this connection is merely implicit in the agency's strategy/ approach. In the cases, some agencies worked with the understanding that changing the attitudes and relationships among key decision makers could help them reach agreements. Others saw that changing popular attitudes ("building a peace constituency" or "building a culture of peace") could put pressure on elites to make an agreement and /or anchor agreements already made.

2. Changes in Behavior
Observable changes in the way people behave are another often reported type of impact indicator in the RPP cases. These included instances of violence calming down, people responding differently (nonviolently) to new provocations, or powerful agencies changing their policies to address the conflict more directly.

In areas where the agency was working, provocative leaflets were circulated (many suspected the instigators were local political figures). This did not lead to fighting between the two ethnic communities, although five years earlier, similar leaflets had incited violent clashes.

An ethnically divided country's largest multinational enterprise (allied closely with the government) took part in the agency's workshops on labour relations. This agency used this opportunity to introduce concepts of dispute resolution and negotiation to senior staff. Some years later, a senior executive enabled crucial meetings between the country's main insurgent group and the government. The company also changed its internal policies regarding employment of people from the other ethnic group, and openly defied the government on such issues.

Other behavior indicators in the cases point to increasing levels of positive social contact among people from both sides (in such settings as markets, football games, bars, buses) and link this to peace activities conducted.

With such indicators, attribution of the change to the work of the agency is often difficult. Many other factors may drive changes. In the first example above, the past experience of violence may have been as responsible for resisting provocation as anything the peace agency had done. In the second, pressure from a trade boycott could have influenced the company's decisions as much as (or more than) its engagement with the conflict agency.

3. Peace Agreements/Covenants/Declarations
Peace agreements, ceasefires, peace covenants, or declarations were reported as indicators of impact in some of the RPP cases. In these cases, the efforts of peace agencies were said to have "played a role", "made an important contribution", or "had served a useful purpose in securing deeper confidence" in the reaching of agreements. Scenarios included:

• Women peace activists pressured military leaders to join negotiations


• Discussions at unofficial dialogue meetings shaped agreements made officially


• An agency forged direct agreements among 'the people' of both sides at massive peace meetings


• An agency hosted and funded secret peace talks in a neutral location

Attributing even some part of the agreement reached to a peace agency's work is extremely difficult since so many factors influence leaders' decisions to make a deal - not least geopolitical carrots and sticks and personal interests. And a great deal of time may have elapsed since the agency's work.

Several leaders who were part of secret negotiations that led to a historic peace accord had taken part in intensive conflict resolution trainings and workshops held by an international agency many years earlier.

But attribution aside, are agreements per se useful impact indicators of progress towards peace? Their significance clearly depends on other things: whether they are lasting, are implemented, address the key issues driving the war, etc. The assumption is often that agreements alone have important symbolic value, but as the earlier example showed, this should not be overestimated. There are cases in which "one more unfulfilled agreement" undermined mutual confidence.

4. Institutionalization
Many cases reported examples of the establishment, strengthening, or activation of institutions to manage conflicts nonviolently as impact indicators. Institutionalization as an indicator points to changes at the level of the larger society (versus the individual). Often this meant that previously informal peace activities of NGOs and activists became formalized. In several cases, local peace councils and committees were created, often with recognition and the cooperation of the relevant authorities. In one case, the ruling political party created an "Office for Reconciliation", giving pubic prominence to the ad hoc efforts of a local NGO.

However, the cases show that instances of institutionalization alone do not credibly indicate an impact on the conflict overall. Several cases showed such peace structures that, once set up, remained inactive, or died out once the agency played a lesser role. In the RPP cases, the more credible indicators of institutionalization showed people actively using these institutions.

Though the local council on land disputes did not solve a large number of the conflicts brought before it, the gathering became an accepted mechanism that was called on often by local people. It achieved national renown and gained influence with the authorities - when it appealed to a government institution, the government often "kicked into gear".

5. "Satisfaction" and "Demand"
In the RPP cases, participants very often reported "satisfaction", "appreciation" or "desire for more" trainings, dialogues, joint projects, or other peace-related activities. While this represented important feedback, in the cases these were often quite weak indicators of impacts on people or on the conflict. Politeness towards the interveners, or personal gain (in terms of status and prestige, ability to travel, etc.) are only two of the alternative explanations for such positive feedback.

However, more powerful 'deed' related indicators, that demonstrated such satisfaction/demand, were also reported in some cases. These showed people using the processes in ways which advance the overall cause of peace. For example:

An informal gathering of elites from both sides of the conflict had continued for many years, and the participants (some who were closely connected to the highest leadership) used these informal gatherings to discuss contentious issues in advance or to 'unstick' the formal peace talks. For example, a meeting of the group was held four days after a partial peace agreement was signed to discuss last minute amendments and concerns.

Another case reported;

People now view the local peace agency as a form of security. If attacks or rumours of impending attacks occur, community people immediately report this to the District Administration and to the agency.

6. Increased Local Capacity for Conflict Management
Another cluster of indicators in the cases pointed to specific instances of increased local capacity and skills for nonviolent conflict management. Such indicators included numbers of people trained (in skills for analyzing and dealing with conflict), and things like the creation of networks of community conciliators and peace mediators. But such indicators do not show if such conflict management resources are used, or what difference their use makes to the conflict overall.

However, related indicators in the cases (often narratives or anecdotes) describing the use of such skills and resources were more credible and useful. They often described how the activities conducted had led to new proactive efforts by local people to address conflicts.

High-level officials reported that they always used the negotiation training materials from the agency's workshops as preparation for the official peace talks with the other side.

A newly established local NGO, trained and supported by an outside peace agency, immediately undertook work on majority-minority tensions. Their project succeeded in getting unprecedented interest and participation from the 'hard to reach' minority group.

But even if such local capacities are utilized and valued, what contribution does this make to peace? One pattern found in the cases is that agencies often focus on building up non-governmental capacities for conflict management, but this alone rarely leads to substantive changes, even at the community level.

In one case, a local peace agency learned over many years that government must be involved in peace work from the beginning, along with civil society. The community realized that local "elders could mediate but could not act as a police or judicial system".

As well, one case example showed that the contribution to peace depends not only on "if" skills are used, but what they are used for.

A small community conciliation network had been trained and supported by an international agency. The network tried to facilitate dialogue between teenage robbers and their victim, and between a rape victim's husband and the rapist. The local justice systems were weak and not trusted in this area, but this raised concerns among agency staff about the implications of such approaches being used instead.

7. People Value the Activity
Many cases report a variety of ways of knowing that 'people value the activity' as indicators of impact. Such indicators range from personal testimonials by participants to letters from the top leadership to more tangible demonstrations, such as: people gave their time, the community or government committed resources, people replicated the process elsewhere, and people took personal risks to advance the activity. In general, self-reports vary in their credibility and each one needs to be judged in context.

A participant in an informal dialogue process risked conflict with colleagues and damage to her career by championing an unpopular pro-dialogue position within her government.

Many other cases show people risking (and sometimes losing) their lives because of their work to promote peace activities, or their refusal to stop participating in them.

Instances of many local people claiming ownership of a peace activity in the cases seemed a highly credible indicator, as it showed that many people saw the activity as useful and wanted to be associated with it.

In one area where an innovative peace process involved many sectors, agencies and large numbers of people from both sides, many local groups and actors each claimed they had been responsible for making it happen.

Even if it can be conclusively proven that people do value a peace activity, how significant is this for peace? One of the reasons that people value something may be that it is actually helping. However, as with satisfaction/demand indicators, there are many other possible reasons as well and such indicators are often equivocal.

8. Progress on Resettlement/Return of Refugees
Very often in cases where humanitarian agencies were involved in peace work, reconstruction and a return to 'normalcy' was reported as an indicator of an agency's impact on the prospects for peace overall. In several of the cases, progress on refugee issues seemed to have particular importance to agencies as an impact indicator.

In at least one case, speeding the resettlement of refugees back to their previous homes (in areas where they would be minorities), resulted in an easing of pressure on the government from the refugee community to regain the lands through a new war. In another area, the lack of resolution of the question of refugee return was a key issue that resulted in the disintegration of a long-standing peace processes that had seemed to be heading towards settlement.

How generalizable is this pattern? Because refugees are a flashpoint for renewed conflict is dealing with refugee issues a critical indicator of impact for work on conflict?

V. INDICATORS OF IMPACT AND CRITERIA FOR EFFECTIVENESS

The above discussion has sorted through types of indicators found repeatedly in the RPP cases in terms of their strengths, weaknesses, and ultimately, usefulness to agencies. The case examples presented illustrate how an individual impact indicator is highly context specific, and how its credibility and significance become clear only when a lot more information about the context is available. The examples presented are also suggestive of how one could identify indicators in each specific setting that do " pass" the tests of attribution, credibility, and significance.

A systematic review of all the case studies suggested also suggested criteria for effectiveness that apply not just to a specific context, but to the broad range of types of peace work being done to help agencies know if they are being effective. These are presented for review and comment in the RPP Issue Paper: Criteria for Effectiveness of Work on Conflict.

---