User:JAnstee (WMF)/Sandbox/Draft/Outcome Mapping

Outcome Mapping is an evaluation approach developed in 2001 for evaluation which places people as the center and looks at changes in behavior to define outcomes.

Importantly, program outcomes are defined as “changes in behavior, relationships, activities, or actions of the “individuals, groups, and organizations with whom the program interacts directly” as well as “those with whom the program anticipates opportunities for influence” are considered “boundary partners” and brought to the center of the outcomes mapping approach (Earl, Carden, & Smutylo, 2001, p.1). [1]

In this way, outcome mapping can help to get beyond the more direct program outcomes to deeper environmental outcomes to measure a program’s contribution to complex change processes.

In the Wikimedia world of programs, outcome mapping strategies can help us to gather data on the contributions that our programs make in terms of bringing about changes in our partners around the world that help to build toward greater engagement in open and free knowledge. Further, outcome mapping can also help us to evaluate both the intended as well as unintended results of our innovative Wikimedia programs.

WHAT Outcome Mapping: A Social Cartography of Wikimedia Programs
WHO Affiliates and program leaders of GLAM and Education,
WHY Because we need to get beyond topline dashboard metrics to mapping the qualitative outcomes of core programs as movement drivers (rather than content metrics) for hundreds of implementations of core programs.
SO WHAT Allows for qualitative outcomes and stories of our programs and helps to better surface our shared impact
HOW Help to tell local program stories
Honor the diversity of local programming by providing an evaluation framework that can engage across our landscape of programs
Enable the sharing and capturing of social and environmental changes targeted by our program leaders and their partners.
Emphasize qualitative processes and outcomes -- organically, holistically, and comprehensively

More details and some example tasks and potential timelines

edit

Importantly, there are three basic stages to outcome mapping which form a continuous feedback loop for holistic program thinking and development. While it typically follows a linear pattern at the beginning, it often cycles back through the various planning activities as learning happens.

Stage 1. Intentional Design

edit

Intentional Design is the initial planning stage of Outcome Mapping. During this stage, Wikimedia program and community leaders will work to reach consensus on two things:

  1. The broad environmental changes (macro level change) that a program would like to help bring about in the world
  2. The plans and strategies to provide support that is appropriate, based on resources and opportunities, to help create those changes

Much work has already been done to support our initial review by various program and community leaders who have shared their logic models to describe the theories of change behind their programs. The next step will be to look across these program models to focus on those changes in behavior, relationships, activities, or actions of the people, groups, and organizations are being targeted by our core programs which are not measured in direct program outputs or on-wiki metrics.

Key tasks

edit

Describe the vision and identify:

The mission Our partners in the work The ideal changes in behavior, relationships, and/
or actions
which we wish to influence in our partners
Examples:
Increase coverage of cultural content on Wikimedia projects

Increase access to cultural content available through open licensing of cultural content on Wikimedia Projects
Examples:
Community partners

GLAM institutions

Educational partners
Examples:
Partner promotion of Wikimedia values

Partner engagement in developing Wikimedia projects

Increased awareness of open knowledge projects and resources

After identifying mission, vision, partners, and desired changes, we will seek to map those expected changes collaboratively to identify a set of graduated progress markers which we can use to see if our partner-directed efforts are leading to the desired changes.

The outcome mapping framework prompts us to look at these desired changes in terms of:

The changes we expect to see The changes we hope to see The changes we would LOVE to see
Examples:
Participation in partnership and planning meetings

Development of shared language and skills for project/program

Successful implementation of collaborative events

Examples:
Partner articulation of movement values and strategies at the local level

Expanding the partnership to additional partnerships, opportunities, or resources

Partners changing their policies to better promote movement values

Examples:
Partners playing a leading role in managing resources

Partners sharing experiences and lessons learned

Partners influencing policy at a larger scale

After mapping our desired changes, the outcome mapping framework guides the creation of a clear strategy map for each shared outcome that we wish to document together. In this step, we will develop matrices that categorize the strategies that similar programs employ to influence partners. In this step, each strategy is classified along two axes:
Target: Whether the strategy is aimed a specific individual or group OR the environment of a specific individual or group, and
Type of Influence: Whether the strategy is causal, persuasive, or supportive.

Example: Outcome Area: Strategies for the desired outcome of increasing awareness of open knowledge projects and resources

The final step of stage one is articulating the key actions that would demonstrate these engagement practices within organizations and mapping those potential measures which would also evidence desired changes. This is based on how organizations are working to:

  • Seek new ideas and opportunities or feedback from key program targets and partners;
  • Obtain support from those at the next level of organization (partnership development with partners at the next highest power);
  • Assess the value of the activities, modify product/program design to increase value, and share wisdom about successful strategies; and
  • Engage in organizational reflection and experiment to remain innovative.

From this mapping we can then move into the second stage to prioritize what outcomes we want to map and design strategies for doing so.

Potential timeline and examples:

  • Collaborative identification of shared points of impact (Initiate at WMCON April 2016, push out to meta consultation as follow-up - May-June 2016)

Stage 2. Outcome & Performance Monitoring

edit

The Outcome & Performance Monitoring Stage is the second stage of Outcome Mapping. During this stage, program stakeholders work to define and map strategies for the ongoing monitoring of:

  1. The program's actions in support of the outcomes
  2. Partners' progress towards the achievement of outcomes.

Stage two planning is based largely on setting up shared systems for systematic monitoring and self-assessment. Key tasks include setting priorities for monitoring and developing an outcome journaling system for those target outcomes.

Potential timeline and examples:

  • Collaboratively design process (Follow-up online meetings with core stakeholder groups to respond to community consultation - bi-monthly following WMCon May 2016, July 2016)

Stage 3. Evaluation Planning

edit

The Evaluation Planning Stage Stage is the third stage of Outcome Mapping. During this stage, program stakeholders identify evaluation priorities and develop an evaluation plan.

Potential timeline and examples:

  • Map out our monitoring needs and options and agree upon a shared outcome and performance monitoring system and appropriate evaluation tools (e.g., tracking logs, reporting templates, survey templates, etc.) (September 2016, November 2016)
  • Report back to community on progress (Dec 2016)

Potential Implementation Timeline Beyond Planning

edit
  • Pursue an agreed upon voluntary cooperative evaluation plan in 2017 with program/community leader partners syncing on critical points of change we will monitor and report (beginning January 2017)
  • Monthly check-in meetings/Technical assistance office hours (with core volunteer working groups) (Jan-Mar 2017)
  • Regroup in 2017 at existing convening for working sessions in 2017 (April 2017)
  • Report on progress to community (June 2017)

By cooperatively identifying our shared impact story, we can better guide our reporting and storytelling toward shared practices and then better deliver the qualitative impact story that is not available through tracking behavior and metrics online. Together, we can begin to fill in the context and color of our shared strategic work through programs. This context will help us better understand and represent programs as they contribute to the larger forces of the open knowledge movement as a whole through a social cartography of our programs; a capturing the networks built and behavioral experiences on the front lines of our offline work.

References

edit
  1. Earl, Carden, & Smutylo (2001). Outcome Mapping: Building Learning and Reflection into Development Programs. International Development Research Centre (IDRC).

Resources

edit