Learning and Evaluation/Annual Plan

Theory of Change

With the goal of improving Wikimedia projects and the free knowledge movement, our team fosters a culture of learning and promotes the use of evaluation to improve planning and making decisions informed by data. To do this, we provide opportunities and resources for communities and movement organizers to learn from one another’s data as well as experiences.

Increase evaluation capacity to design more effective programs

Movement Learning & Evaluation

The Learning & Evaluation team has at its core the goal of supporting the infrastructures for learning about, and evaluating, the impact of community programs and initiatives. Infrastructures include regular responsive support to consult on high-potential projects in order to deliver quality evaluation for decision-making including maintaining Community Engagement Insights.

Among this work, Community Engagement Insights helps the foundation to measure various aspects of Wikimedia, such as demographics, behaviors, and opinions related to: (1) the movement as a whole, (2) Wikimedia projects, (3) Wikimedia affiliates, (4) Wikimedia programs, (5) Wikimedia software, and (6) Wikimedia Foundation programs, products, and services. The data gathered will be analyzed to support team decision-making and may be shared publicly upon privacy review.

Program Goal

Maintain core evaluation support: (1) To maintain evaluation guidance and provide regular responsive support to consult on high-potential projects in order to deliver quality evaluation for decision-making and (2) To maintain the Community Engagement Insights process to use surveys to increase the Foundation's understanding of communities' needs as well as gather data that measures the outcomes of the Foundation's work. (See also CE Strategy: “Movement Strategy Measurement”)

Outcome 1: Increase Foundation and community capabilities to conduct self-evaluation through consultations and infrastructural support.

• Maintain responsiveness for evaluation and survey consultations for high-potential projects and initiatives.

Outcome 2: Community Engagement Insights: Improve Foundation's ability to hear from communities in order to make decisions that align with community needs.

• Implement the Community Engagement Insights survey for a third time.

Milestones:

• CE Insights teams report their findings and next steps based on the results of FY2017-18 survey.
• Data collected from 4 Wikimedia contributor audiences by end of FY 2018-19.

• Conduct preparations and planning to improve a reporting interface for survey data in collaboration with Analytics, Design, and other survey stakeholder teams

Milestones:

• Report about Phase 1 (Discovery) completed and shared with stakeholders
• Roadmap completed for the report, contingent upon agreement with moving forward from Phase 1.

• Stabilize the annual survey for measuring long-term strategic direction outcomes by creating guidelines and policies that govern the survey purpose, scope and decisions.

Milestone:

• Draft policies, procedures and templates for team approval.

Outcome 3: Movement Strategy Measurement

See also Outcome 4 of Community Engagement “Strategy” program

Targets

Outcome 1	Target 1	Measurement method
Increase Foundation and community capabilities to conduct self-evaluation through consultations and infrastructural support.	Maintain responsiveness for evaluation and survey consultations: Support at least 2 priority evaluation learning projects and provide consultation for evaluation as needed on high-potential projects	Quarterly Check-In (QCI) reporting & other documentation
Outcome 2	Outcome 2 Target	Measurement method
Community Engagement Insights: Improve the Foundation's ability to hear from communities in order to make decisions that align with community needs.	1. Increase overall response rates by 5% by end of FY2018-19. 2. Increase completion rates to 3% by end of FY2018-19. 3. At least 10 teams participate in CE Insights for the 2019 survey. 4. 90% of teams report on the decisions they are making from the survey. 5. 90% of teams provide feedback for each of the three project outputs.	1. CE Insights sampling strategy document. 2. CE Insights sampling strategy document. 3. Project documentation. 4. Reporting templates for team learning. 5. Participation log for team meetings & focus group(s).

Strong healthy communities

AffCom & Affiliate Partnerships

Recognition as a Wikimedia movement affiliate — whether it be as a chapter, thematic organization, or user group — is a privilege that allows an independent group to officially, and legally, use the Wikimedia trademarks to further the Wikimedia mission.

The Affiliations Committee advises and makes recommendations regarding the recognition and existence of Wikimedia movement affiliates.

Staff support to the AffCom workflows and additional capacity has been engaged to enable the committee to focus on new goals to:

• oversee (i.e. observe and possibly comment on) movement affiliation and disaffiliation, including relevant compliance processes
• oversee affiliate development initiatives by WMF (and possibly other entities, if those would be interested), offering feedback and steering
• advise WMF staff and, as needed, the WMF Board, on perceived needs and issues of/from movement affiliates
• advocate for affiliation, encouraging unaffiliated groups to affiliate with WMF if/when/how it would be beneficial to do so, as well as supporting groups in the process.

In 2018/2019, we hope to maintain core support responsibilities as well as target two strategic improvement projects:

First, we have found that different affiliates post and store their annual reports in many different places which may change over time. This discrepant filing of reporting leads to very little knowledge sharing from that which is reported. Even answering the most simple query of how many affiliates engaged in a GLAM project last year is not easily achievable although this information exists in the majority of affiliate annual reports.

In addition, growing affiliate ecosystem has meant growing frequency of inter and intra-affiliate conflict. Conflicts that initiate within a growing, or stagnating, group can become the instigator of new user group applications that can only add to the conflict; still there are many reasons for conflict that are inherent to our collaborative style. Dealing with these conflicts takes focused investigative and intervention time, and support for such interventions is undersupported. While the AffCom care to oversee and understand how they can better adapt policy and decision-making to prevent conflict, they need additional professional support in order to adequately investigate and communicate around affiliate conflicts and mediate interventions.

WMF staff provide administrative support for user group recognitions and renewals, proactive compliance monitoring, and the affiliate derecognition process when necessary. In addition, staff lend specific administrative support for the committee’s monthly call hosting, annual strategic meeting facilitation support, and task management support to track and prompt continuous progress on strategic progress.

Program Goal

In order to enable basic queries about affiliate work and capacities it is important that we begin to capture more structured data, as well as store and monitor it regularly for movement learning about the development and impact of our affiliates ecosystem. One of our two strategic goals for this next fiscal year is to stabilize monitoring data and streamline data management for better learning over time in order to maximize the benefits of reporting through sustained data capture and knowledge curation. (Report Monitoring Systems)

A second strategic goal for this program is to bolster support for investigative response for conflict intervention support for key organized groups within the movement. In turn, this will support the AffCom’s conflict prevention policy setting and conflict mediation efforts as a developing focus of their work. (Affiliate Conflict Investigation)

Outcome 1: Report Monitoring Systems: Increased knowledge and awareness of affiliate data through more consistent reporting practices and data management.

• Development of affiliate reporting templates, protocols, and corresponding affiliate reporting data management system.
• More sustainable monitoring systems & KPIs: Decreased time spent tracking and monitoring and increased data gathering. We will be working to understand our efficiency and effectiveness through corresponding improvements to our KPIs as we go.

Outcome 2: Affiliate Conflict Investigation: To improve community health through affiliate conflict intervention. Learning & Evaluation will partner the AffCom Conflict Mediation Subcommittee with Support & Safety to more adequately support affiliate conflict investigation and intervention to enable the committee in their mediation efforts.

• Compiling the facts in their working language, structure material, consult with relevant WMF-internal teams like Legal and Support & Safety.
• Development of a framework for disagreements at the movement level not addressed by the standard legal frameworks AffCom operates with in order to prevent tension by refining legal definitions in user group agreements and other governing documentation.
• Exploration of additional tools which may better support interventions such as potential financial sanctions or recommendations for global ban investigations.

Outcome 0 (Core)	Target 0	Measurement method
1. Maintain response times for recognitions pipelines (Initial review within 5 days) 2. Maintain regular proactive reporting & compliance review and follow-ups 3. Conduct affiliate conflict investigation and de-recognition processes on time and with sensitivity. 4. AffCom leadership of their workflows and engagement in core and strategic projects.	1. 100% of new user group applications receive initial review within 5 days 2. Proactive reporting reminders are sent to affiliates 30 days before their due date. Reports are reviewed within 30 days of submission. 3. 100% of derecognition processes are handled according to protocols, on timeline, and in line with their communications plans. 4. AffCom leadership of monthly calls and increasingly equitable distribution of tasks across committee members	1. Tracking of the % of applications with on time initial review 2. Salesforce/Asana tracking 3. Asana tracking 4. Asana tracking
Outcome 1	Target 1	Measurement method
Increased knowledge and awareness of affiliate data through more consistent reporting practices and data management.	1. Increased affiliate awareness based on improved data consistency and data management system 2. Increased efficiency in time spent tracking and monitoring	1. Comparison of reporting and  report filing practices year over year. 2. Comparison of monitoring effort over time to be established as KPIs by Q1
Outcome 2	Target 2	Measurement method
To improve community health through affiliate conflict intervention. We will partner with Support & Safety to more adequately support affiliate conflict investigation and intervention to enable the AffCom in their mediation efforts.	1. Compiling the facts in their working language, structure material, consult with relevant WMF-internal teams like Legal and Support & Safety. 2. Development of a framework for disagreements at the movement level not addressed by the standard legal frameworks AffCom operates with in order to prevent tension by refining legal definitions in user group agreements and other governing documentation. 3. Exploration of additional tools which may better support interventions such as potential financial sanctions or recommendations for global ban investigations.	1. Count of cases and documentation of needed support 2. Framework and process for investigations and conflict interventions 3. Recommendations for additional tools for conflict interventions

Increased learning and knowledge sharing

Learning Days & Responsive Conference Events

Volunteers are integrally involved in every aspect of movement activity. Some individuals step up to take great responsibility, working to coordinate others and to lead activities engineered to help the movement succeed in its mission. Empowering and enabling program and affiliate leaders supports the success of group activities and contributes to the success of the movement. One of the goals of Community Engagement is to help these volunteer program and affiliate leaders to reach their aims as efficiently and effectively as possible by giving them access to critical informational resources and, importantly, building networks among them for peer-exchange.

The community continues to need professional development support in the area of program design and evaluation, and it would be much more costly for them to seek it out externally than for us to rally our internal and external assets alongside existing exchange opportunities.

Continued interest and signup uptake for Learning Days is steadily increasing with more than 150 community members participating in 2017 and on target to have more than 100 join us in Berlin April 2018. It is an important learning exchange space that helps to support movement collaborations and learning networks.

Program Goal

To support peer-to-peer connections and learning and to collectively develop and work to better enable more community leaders and trainers.

Outcome 1: Develop community leader evaluation and leadership skills through increased awareness of key tools & resources for program strategies, best practices, and available support.

• Learning Day events for 150 community leaders to engage in training and peer mentoring on critical learning and resource topics for core programs and community development strategies
• Provide responsive or proactive support for community development and evaluation learning support through sharing resources and supporting workshops at strategic community convenings we are invited to teach at.

Outcome 2: Facilitate engagement in our learning network by connecting Wikimedians from across communities.

• Learning products generated for documentation and exchange including: workshop presentations and facilitation guides, poster presentations, and peer connections for follow-up application or referencing.
• Experienced community members offer support to new community members.

Community members share resources on the WRC and identify themselves or others as leads in that topic.

Targets

Outcome 1	Target 1	Measurement method
Develop community leader evaluation and leadership skills through increased awareness of key tools & resources for program strategies, best practices, and available support.	1. Increased evaluation and leadership skills among Learning Day event participants 2. Increased use of tools and resources in grant proposals and non-grant program planning	1. Exit and follow-up survey and community level indicators 2. Targets mapping data, Logic Model use in proposals, self-report on Learning Days follow-up survey
Outcome 2	Target 2	Measurement method
Facilitate engagement in our learning network by connecting Wikimedians from across communities	1. Increasing connections between Wikimedians from across communities. 2. Increasing engagement in our learning networks.	1. m:Connect use 2. Community resources added to WRC 1&2. Self-report on Learning Days follow-up survey

L&E Communications, Outreach, & Collaboration

A large amount of L&E’s core work falls into our communications & outreach to repeatedly engage and expand our learning community and supporting the documentation and promotion of best practices, sharing development opportunities and tool updates, and enabling networking across foundation and community groups.

Program Goal

Learning infrastructure and support for program and community organizers to enable their continuous development and designing of effective initiatives. Maintenance of team collaborations and community learning infrastructures, portals and supports to coordinate and amplify guidance and learning across our growing network of community organizers.

Note: Learning infrastructures, portals, and supports include the Wikimedia Resource Center L&E Portal, Learning Pattern Library, Survey Support Desk, Learning Quarterly newsletter, Strategic CE communications coordination, and community storytelling consultations, and networking for learning stories.

Outcome 1: Elevate community stories to the movement at large to better serve the needs of community members looking for, and offering, resources.

• Basic maintenance of the Wikimedia Resource Center, and continual increase in community community engagement with the platform. This includes socializing features, improving current curation, and cleaning up categories for existing content.
• Up-to-date evaluation portal highlighting recent events, blogs and other resources.

Outcome 2: Improve strategic communications by CE teams and community storytellers.

• Support to Community Engagement teams in strategic planning of their communication workflows, to advance CE programs and projects.
• Support to community members, by demand and proactively, in communications, storytelling, and sharing learning.

Outcome 3: Maintain coordination on team projects, communications, and outreach efforts for carrying out core and strategic operations.

• Regular team meetings, check-ins, and shared task management systems
• Team strategic meetings and co-working time for updating on our longer term goals, progress, and planning.

Targets

Outcome 1	Target 1	Measurement method
Elevate community stories to the movement at large to better serve the needs of community members looking for, and offering, resources.	1. Increased participation of community members in learning infrastructure by 20%. 2. Increased pageviews of the WRC and relevant team portal KPIs by 20%	1. Number of resources shared on Learning and Evaluation portal and WRC. 2. Pageviews of WRC & team KPI reporting in Quarterly Check-In (QCI) reportings.
Outcome 2	Target 2	Measurement method
Improve strategic communications by CE teams and community storytellers.	1. Sustained engagement of community members in more CE projects, workflows, and/or events. 2. Improved communication rollout of community-led projects receiving consultation 3. Increased number of community generated resources for audiences beyond the original source	1. Number of participants contributing / adding to supported CE projects, workflows, and /or events. 2. Number of comms plans developed and used for community projects 3. Count of community-generated resources in the WRC marked for translation and/or adapted for use beyond the community that created them.
Outcome 3	Target 3	Measurement method
Coordinate our L&E project communications and outreach efforts as well as  team support for carrying out our core and strategic operations (Core L&E Admin)	1. Team member engagement in standups, weekly coworking, and manager one-on-ones. 2. Team consultation and collaboration on priority annual plan projects.	1. Observation/Calendar Record 2. Observation/Calendar Record