Community Capacity Development/Overall pilot year evaluation

Abstract

This page is an overall evaluation of the Community Capacity Development (CCD) program pilot, which was researched in 2015 and carried out in 2016.

The pilot was done with three communities, and description and evaluations of each of the three pilots are available separately (Ukrainian community, Brazilian community, Tamil community). This page evaluates the program as a whole, based on the pilot experience and observed outcomes.

Context

How it came about

The idea for a capacity-development program by WMF, supporting communities and affiliates, has been long proposed by Asaf Bartov. Indeed, it was a major component of the job he had applied for. By the time Asaf was hired, the emphasis (and name) of the position shifted, and he ended up focusing almost exclusively on grantmaking until late 2014, when he was able to hand over his grantmaking duties and re-focus on supporting emerging communities. In addition to ongoing and ad-hoc, reactive support to emerging communities (which had been part of his duties while a grantmaker too), Asaf initiated a proactive capacity-development program, as a pilot, which he called the "Community Capacity Development" program.

The program in wider WMF context

The program was a bold experiment to begin with, as it obviously involved high-investment work with specific communities to build specific capacities, in a time when the Foundation was very focused on "scalability" and product (i.e. technological) innovation, and was stepping away from high-investment involvement with specific communities (the "Catalyst" programs in India and Brazil). Anasuya Sengupta, Senior Director of Grantmaking at the time, agreed with Bartov that high-investment capacity building does make sense in certain contexts, and approved the pilot to test this approach.

Some capacity-building was already being offered by WMF at the time, notably by the Education team (for Education program leaders), and by the Learning and Evaluation / Program Evaluation and Design team, in the field of metrics and evaluation for program leaders. In designing the pilot, an effort was made not to intersect with any of the existing efforts. This included organizational development, which at the CCD pilot's inception was planned to be tackled by an effort called "Organizational Effectiveness", since suspended and cancelled. Organizational development remains a frequently-cited need by affiliate organizations; it is partially being addressed by recent peer-led capacity development work led by Tim Moritz Hector from Wikimedia Deutschland and Frans Grijzenhout from Wikimedia Nederland.

Process

Research

The pilot program began with a research phase, conducted by Asaf Bartov and Sati Houston, consisting of in-depth interviews with 16 emerging communities. The interviews were hand-coded and tagged to elicit repeated themes about community needs and perceived capacity gaps. The results suggested six capacities mentioned by multiple communities, and those six were written up in some detail in dedicated pages, as follows:

1. Capacity 1: Community Governance
2. Capacity 2: Conflict management
3. Capacity 3: On-wiki technical skills
4. Capacity 4: New contributor engagement and growth
5. Capacity 5: Partnerships
6. Capacity 6: Communications & Media relations

Planning

Based on the research results, the six identified capacities and their descriptions were shared with the Wikimedia movement, and comments were invited. Relatively few comments were received; the general tenor was supportive, curious, and several small-to-medium communities expressed interest in pursuing capacity-building activities. Bartov proceeded to have conversations with different communities, to gauge interest in and readiness for engaging in a capacity-building partnership with the Foundation. This took several months.

The original plan was to pilot with two communities, but Bartov decided to expand the pilot to three communities (and therefore three capacities), to increase the diversity (of both communities and capacities) explored in the pilot, and maximize the potential insight, especially considering a failed pilot would have likely spelled no further experimentation by the Foundation with this approach.

The three selected pilots were:

1. developing the Communications and Media relations capacity with the Brazilian Portuguese community
2. developing the On-wiki technical skills capacity with the Tamil community
3. developing the Conflict management capacity with the Ukrainian community

Some of the factors contributing to the selection of these three communities were size of active community, size of population served, level of off-wiki activity/organization of the community, and above all, community interest in participating in this experiment.

Execution

Once the pilot communities and capacities were determined, the three pilots proceeded independently of each other, and in parallel. Broadly, all three followed these steps:

1. conversation with the community regarding timing and method for delivering in-person capacity-building training
2. curriculum development for in-person training
3. identifying co-trainers (Asaf Bartov participated as a trainer in all three pilots)
4. Logistical preparation (international and domestic travel; venue arrangements; accommodations; catering; etc.)
5. In-person training event (with pre- and post-training surveys of attendees)
6. Follow-up survey and qualitative interviews

These steps, as well as links to the curriculum, survey results, etc., are all documented in the pages dedicated to the three pilots. See the following for those details:

• Brazil (Communications)
• Tamil (On-wiki technical skills)
• Ukraine (Conflict management)

Evaluation

Fitness to purpose

Did the pilot programs achieve capacity building? Two of the three (Brazil, Tamil) definitely did (see Evaluation section in each individual report, linked just above); it's too soon, or too hard, to tell regarding the third (Ukraine).

In all three pilots, the community appreciated the effort and interest by WMF. (Anecdotally, during the Research phase, more than one community told us things like "This is the first time WMF took an interest in how this aspect works in my community.")

All three communities expressed interest in future opportunities to build other capacities, perceiving value in the general approach of a limited-time, focused partnership with WMF to build a specific capacity or overcome a particular obstacle. (drawn from the follow-up surveys of the individual pilots, linked above)

So, does it scale?

Well, yes, it does! Not in the same way technology (such as the Visual Editor) scales across the projects, but in an important way nonetheless: effective capacity-building scales across time. In other words, effective capacity building means the capacity "stays built".

For example, whereas the Tamil community had not been contributing to Wikidata at all beyond the necessary minimum (interwiki links), not only are Tamil contributors now active in Wikidata, but the Tamil community will never again need an introduction to Wikidata. That knowledge is -- now -- owned by the Tamil community, and will continue to disseminate within that community, in Tamil, in perpetuity. It was the initial obstacle that the capacity-building effort helped overcome, and now further growth will happen organically, without further investment (in that specific direction) by the Foundation.

Capacity-building is inherently a high-investment (though limited in time and specific in scope) activity. But as demonstrated above, it is efficient when effective.

Crucially, it is fulfilling a need not met by any other process. Technological innovation cannot be the only tool deployed by WMF in support of the communities it serves, as it does not address the variety of needs and obstacles those communities face in their natural growth and activity.

Conclusions and recommended next steps

1. Scale up (additional communities; additional capacities)

2. Develop a Community Capacity Map (CCM) to invite communities and affiliates to self-assess their capacity (according to guiding criteria). The map will be used to track capacity development, and to identify opportunities for impactful capacity-building projects.

3. Identify already-effective trainers and knowledge diffusers; enhance their opportunities to share knowledge; train prospective trainers and empower successful trainee trainers.

The above implies increased resourcing (budget and staff time), as well as participation from additional teams.  This coming fiscal year (starting July 2017) will not see increased resourcing, but some other teams have agreed to contribute trainers occasionally.