Grants:Wikimania scholars/Proposed 2015 Process

Thank you for your feedback. The final 2015 Scholarships Program can be found on the 2015 Wikimania Scholarships page.

After Wikimania 2014, we collected feedback from Scholarship applicants, recipients, Wikimania organizers and WMF staff, across multiple aspects of the Wikimania Scholarships process, including from:

• The Wikimania 2014 Scholarship Survey
• The 2014 Wikimania Scholar reports
• Joint sessions between WMF staff responsible for various functions
• Joint sessions between WMF staff and some members of the 2014 Scholarship Committee

This data has been analyzed by staff members of WMF Grantmaking and the Wikimania conference coordinator, and has been used to create the following recommendations, to be discussed by the Scholarship Committee and the wider community.

We invite your feedback on this document before the 2015 plan is finalized! Please add your comments/questions to the talk page. Please note: All underlined text indicates areas where we particularly invite your feedback.

Overview

The Wikimania Scholarship Program is funded and administered by the Wikimedia Foundation in partnership with the Wikimania Scholarship Committee. In 2014, the Wikimedia Foundation brought operations of the program under the umbrella of the Wikimedia Travel and Participation Support program, to streamline administration efforts.

The goal of the program is to offer a limited number of scholarships to selected individuals, to offset the cost of attending Wikimania. Started in 2005, the Scholarships program was initially handled by the Wikimania host country, but operation was transferred over to WMF in 2012. Since 2012, WMF has funded a total of 317 individuals. For London's Wikimania 2014, 118 scholars were selected from 1170 applicants, and 97 ultimately attended the event.

As of 2014, Scholarship recipients were determined by a two-phase selection process, with a Scholarship Committee of 10 volunteers responsible for vetting applications and providing a recommended list of recipients. Final decisions were made through joint discussion between the Scholarship Committee and WMF coordinators.

Outcomes reported by 2014 Scholars

In order to gain insight into the outcomes driven by Wikimania attendance, 2014 scholarship recipients were asked submit a post-conference report, detailing either their shared experience or new creations. Of the 82 reports submitted, the majority had sufficient detail and highlighted six major outcome themes:

1. Deepening relationships & understanding through face-to-face interactions
2. Formal & informal experience sharing
   1. Focused on Outreach & events, e.g. best practices for running home Education programs
   2. Focused on Online projects, e.g. Language team mentoring on page translation
   3. Specific to shared challenges & experiences of smaller wiki communities and Wikimedia organizations, e.g. struggles involved in establishing a wiki community
3. On-wiki improvements or contributions, e.g. new IdeaLab submissions, improving interoperability between Wikipedia and WikiData
4. Communication back to their local wiki community about their Wikimania experience, learning, or presentations (e.g. on-wiki reports, Chapter meetings)
5. Communications targeting a broader audience, through off-wiki channels (e.g. radio talk shows, local newspapers, personal blog posts)
6. Creation of new groups & communication mediums, such as a new mailing list, user group, facebook group, or regional conference

Ordered by the number of mentions (where themes could be mentioned uniquely multiple times in a single report), the top outcome themes across all scholars were: Face-to-face interactions (#1), Experiences sharing for online projects (#2-2), and On-wiki improvements (#3).

Wikimania Scholarship Program - Outcomes from 2014 scholars - Chart of results by total mentions

These results could be segmented by the size of the home Wikipedia each scholar reported, where "size" was determined by the # of Wikipedia articles, using article counts as of October 2014:

• "Large" Wikipedia - Article count over 1 million
• "Medium" Wikipedia - Article count between 1 million and 100 thousand
• "Small" Wikipedia - Article count below 100 thousand

This segmentation revealed that the #1 outcome theme differed by scholars belonging to different sized of Wikipedias:

• "Large" Wikipedia - Deepening relationships through face-to-face interactions
• "Medium" Wikipedia - Communication back to home wiki community about Wikimania experience
• "Small" Wikipedia - Experience sharing focused on Outreach and Events

Moreover, while the top 3 outcome themes between "Large" and "Medium" Wikipedias were largely similar, "Experience sharing for Outreach and Events" was the clear leader for the "Small" Wikipedia scholars at 37% of total mentions.

The full presentation summarizing these results includes examples excerpts from scholar reports.

Feedback on the current program

Feedback from 2014 applicants and recipients

After the 2014 Wikimania, WMF conducted a "Wikimania 2014 Scholarship Survey" of 262 scholarship applicants (all applicants who entered Phase 2 of evaluation), of which 42% received scholarships. Results from the survey show that recipients were largely satisfied with the process overall (for process steps they engaged in).

However, 46% of rejected applicants perceived the result as unfair. Additionally this perception of unfairness correlated with higher incidences of decreased motivation and contribution, as a large number of those rejected applicants agreed with the statement "I was demotivated and my contributions started to decline".

There was also strong agreement (around 80% agreement) across applicants that the Scholarship Program should:

• Fund a diverse set of contributors
• Fund Wikimania presenters
• Should not grant scholarships randomly

Moreover, there was not strong agreement (less than 80% agreement) that the program should only fund scholars who had not previously had a scholarship or previously attended Wikimania.

Summary of Issues

Despite the strong outcome reported in 2014, the program has experienced a number of issues across multiple dimensions, expressed by Scholarship applicants, recipients, Wikimania organizers and WMF staff. To help focus the redesign efforts, the following issues were collected across all groups; each issue was assigned a priority level, with any "Priority 1" issue to be addressed explicitly in the re-design.

Priority 1 issues

Area	Issue
Strategic direction	Missing target outcomes for program
Roles and responsibilities	Unclear responsibilities between Committee and WMF on final selection; WMF with ultimate override ability WMF internal - Unclear owner of Scholarship process within WMF
Process	Long timeline between submission and final decision, exacerbated by soft deadlines Significant amount of committee time spent/wasted on simple eligibility checks (Phase 1) Unclear eligibility process, including lack of comprehensive eligibility criteria Selection criteria: Extensive selection criteria with multiple discretionary exceptions Regional allocation system - Unclear “why” and “how” No consideration for accepted Wikimania presenter, speaker No explicit rules for handling repeat vs. new recipients Minimal coordination between WMF and Chapter-awarded scholarships leads to inefficiency and redundancy WMF internal - Post-selection process: Lack of upfront deadlines for post-selection process steps (e.g. travel and visa assistance, reimbursement) Unclear handling for countries with anticipated visa difficulties Absence of clearly defined tasks between scholars, Grantmaking, and Travel leads to long and involved reimbursement process; exacerbated by accommodation of late requests
Incentives	Committee participation has a significant time commitment (40+ hours) and perceived as “thankless job” Varied commitment - 20% dropout rate for Scholarship Committee members
Culture	Lack of clarity in community about who we fund and why (criteria, etc)

Priority 2 issues

Area	Issue
Roles and responsibilities	Committee with significant # of new members; high drop-out rate, hard to maintain geographic diversity Absence of clear communication between Committee, WMF Eng. & staff around the scoring software
Process	Highly varied applicant quality Regional allocation system does not account for Chapter-awarded scholarships Lack of standardized scoring, including uneven # readers per application Minimal engagement with scholars at Wikimania WMF internal - Post-selection process: Numerous soft deadlines, leading to long timelines and overloaded capacity WMF staff overwhelmed handling arrangements for Scholars on top of WMF staff and Wikimania speakers in very limited timeline Credibility of visa application might be undermined by lack of official invitation package Late handoff to distribute reimbursements (visa + incidentals) via TPS process
Incentives	Committee aren’t eligible for Scholarship due to participation in process
Culture	Perception that going to Wikimania is a “reward” for large contribution / activity Measurement of program impact around “diversity”, rather than other movement measurements of value via “contributions”

Proposed 2015 Scholarship Program

Target Outcomes

Based on an understanding from the 2014 Scholar reports and associated results, the Scholarship Program should continue to target outcomes that support contributors from wiki-communities of all sizes. And given that Wikimania already facilitates relationship-building for all attendees by the very nature of its formal & informal face-to-face interactions, this will not be a specific target outcome for the program.

As such, the 2015 Scholarship Program should aim to target the following three key outcomes for scholars:

1. At Wikimania: Experience sharing about both offline topics & online projects
2. After Wikimania: Enrichment of home community via sharing of experiences and dissemination of learning
3. New collaborations:
   1. On-wiki improvements begun at Wikimania (e.g. hackathon creations, knowledge acquired from workshops)
   2. New groups & communication forums for future collaboration

Application and selection process

Recruitment

As in previous years, any active contributor to a Wikimedia project, or Wikimedia volunteer in any other capacity, from anywhere in the world, is encouraged to apply. The Wikimania Coordinator and Scholarship Committee will reach out via multiple mailing lists and Wikimedia Chapters to encourage applications, as well as put up a banner on Wikipedia to encourage a broader pool of applicants. However, results of the Wikimania 2014 Scholarship Survey showed that a large number of unsuccessful applicant come from this banner. As such, to improve applicant quality, these banners would now target logged in users with over 50 edits.

Selection process

The proposed selection process will still follow three phases of review, with notable changes in each phase:

1. Phase 1: Eligibility filtering
   • Initial eligibility evaluation completed by WMF staff, rather than by the Scholarship Committee, to make better use of Committee members' time and focus their efforts on in-depth evaluation
   • Refined rubric against eligibility criteria, with clear definition for the "Wikimedia contributions and activities" criteria so that the process can be better automated
   • All applicants will be notified by e-mail on whether their application has passed or failed once this phase of review is complete

1. Phase 2: Evaluation
   • The Scholarship Committee will evaluate all eligible applicants (passed from Phase 1) via new evaluation criteria. These proposed criteria are designed around target outcomes and focused primarily on activities specific to Wikimedia movement (involvement in other free knowledge, free software initiatives a plus but no longer a requirement).
   • Clear rubric against these criteria will be provided to the Scholarship committee, with scoring against each criteria returning to the 0-4 point scale, to simplify evaluation.
   • Explicit handling of previous scholars: Applicants who have previously received a Wikimania scholarship will have to answer an additional question around demonstrating the impact of attending Wikimania previously, and articulating clear goals for going again.
   • Funding will be set aside for Wikimania presenters who did not receive a scholarship but are deemed by the program committee to be essential. Given the selection of presenters typically trails the selection of Wikimania scholars, presenters who have applied for a scholarship but are not selected will still be able to attend by receiving as-needed financial assistance.
   • Scholarship Committee's recommendation will go to WMF for final approval
2. Phase 3: Final approval
   • Applicants will be separated via a new diversity system, designed around creating a equal mix of scholars from different sized wiki communities, to address concerns about a diversity system based purely on geography.
   • Based on estimated travel costs for each individual (based on home airport), top scoring applicants from each group will be offered a scholarship.

Eligibility Criteria for Phase 1

Eligibility criteria have remained largely the same, with the addition of automatic disqualification of 2014 recipients who did not complete a post-conference report.

To assist in better filtering of applicants, a clearer definition of "Wikimedia contribution and activities" is proposed below:

• Involvement in Wikimedia organization (Chapters, Thematic Organizations, User Groups)
• Community Check Users, Admins, Bureaucrats, Stewards and OTRS (both current and former)
• Wikimedia Foundation Grantee
• Participant in a Wikimedia program (e.g. Education program students, professors, and online ambassadors)
• Participant in Wikimedia organized events (e.g. photographer contributing to Wiki Loves Monuments, workshop attendee)
• Developers active in Tool Labs
• Contributor to a Wikimedia project, with over 50 contributions (i.e. edits, to minimize reader-only applications)

Evaluation Criteria for Phase 2

As of 2014, applicants who passed Phase 1 were evaluated along a set of "Phase 2" evaluation criteria structured around: activity within the Wikimedia movement, activity within other free knowledge / software initiatives, and interest in the Wikimedia movement. Applicants were scored (0-10) on their level of participation, impact and interest in the movement. While strong candidates were eventually selected using these criteria, both the Scholarship Committee and WMF staff felt the criteria lacked strong rationale and a clear and consistent rubric (especially for non-Wikimedia related activities).

With target outcomes defined for 2015, the new Phase 2 evaluation criteria will tie closely to the "Experience Sharing" and "Enrichment" outcomes. The hope is to highlight applicants who have compelling Wikimedia-related experiences and demonstrate a particular ability to use their experience/learning to enrich their home community. As such, the following are the proposed criteria to evaluate activity within the Wikimedia projects, affiliate organizations or initiatives:

1. Relevant experience in the Wikimedia movement, where the following dimensions will inform the ultimate score:
   • Collaboration - Degree of collaboration with other individuals or organizations in order to execute activities
   • Impact - Level of online or offline impact, e.g. content created or participants reached
   • Community leadership - Role(s) played and scope of activities within the Wikimedia movement, e.g. members who serve on committees, project or program leaders
2. Demonstrated ability to share experiences and learnings with a wider community, thereby enriching their home community or increasing awareness in broader audiences in their home country. This builds off of behaviors seen in the 2014 Scholars, demonstrated through their post conference reports about creating on-wiki reports, blogging about what they learned on their personal blogs, or writing about what they learned in their local newspapers.

Such close ties between target outcomes and evaluation criteria will hopefully promote even better outcomes in 2015.

Diversity system for Phase 3

Previously, all Phase 2 applicants were grouped by their home country into a geographic region, where the number of scholarships awarded by region was subject to the Phase 3 regional percentages. Many concerns about the structure, rationale and implementation of this system were raised by past Scholarship Committees, with a primary concern around a diversity system based purely on geography.

To ensure the new diversity system addressed these concerns, discussions on the new system included ensuring the new system would:

1. Define diversity by a broader set of factors than just geographical region
2. Create a nuanced approach to diversity, while balancing complexity vs. ease of execution
3. Ensure comparison between similar candidates in the same group, to the best extent possible
4. Have a clearly articulated rationale for the quota structure

In response to the above, the proposed Phase 3 diversity system retains the quota-based structure but replaces the regional allocation system with a wiki-based system:

• Inclusion of a "Global North" or "Global South" characterization of home country, where the number of scholarships is distributed between the Global North and Global South at 25% and 75%, respectively
• For both Global North and Global South groups, applicants are separated by the size of their home Wikipedia into three sub-groups - Large, Medium, and Small as defined above - with equal allocation of scholarships across all sub-groups
• Reallocation of un-used scholarships:
  • A minimum score will be determined for each sub-group (different for each sub-group), below which applicants will not be considered for final selection. In the event of insufficient applicants with a high-enough score in a particular sub-group, un-used scholarships will be equally reallocated to the other sub-groups.
  • In the event where there are insufficient applicants with a high-enough score across all sub-groups from either the Global North or Global South, un-used scholarships will be reallocated to other group (and distributed equally among sub-groups).
• For the applicants within 10% of the cutoff, preference will be given to the non-male applicant to promote gender diversity
  • The cutoff will be the score associated with the last scholarship awarded in each sub-group

The rationale for the proposed diversity system is as follows:

• Attributes about the existing wiki community and the broader geographical context are both necessary inputs when grouping applicants, as different sized communities see different value in attending Wikimania. While the previous regional system did account for geographic location, it did not take into account the many wiki communities being grouped together, and lacked an ability to distinguish between communities in different stages of development.
• While home country is the most nuanced approach to geographical context, country-by-country evaluation is not realistically feasible in the given timeframe of evaluation
• WMF has made a clear commitment to decreasing the gender gap and increasing reach and participation from the Global South as a part of the five-year strategic plan developed for 2010-2015.
• Looking at the breakdown of scholars from 2012 and 2013 (including those funded by Chapters), the percentage of Global South scholars has been increasing, suggesting that the target of 75% of scholars from of the Global South is achievable:

	2012: Awarded	2012: Accepted	2013: Awarded	2013: Accepted
Global South: Partial Scholars	19	6	12	7
Global South: Full Scholars	62	43	58	51
Global South: Total Scholars	81	49	70	58
Global South: % of Total Scholars	61%	57%	69%	68%

Coordination with Wikimedia organizations

While WMF is responsible for the primary Wikimania Scholarship Program, individual Wikimedia organizations (Chapters, Thematic Organizations, User Groups) may select to award additional scholarships to members, volunteers, or contributors in their organization or community. In previous years, Wikimedia Chapters have been the main organization offering scholarships, but there has been minimal coordination in executing these different scholarship programs. On one hand, there are some Chapters who declare upfront their intention to offer their own scholarships, asking WMF to simply hand over all applicants for their country. On the other hand, there are other Chapters who have decided to offer scholarships late in the WMF selection process, causing duplication of work.

In order to improve coordination between WMF and Wikimedia organizations who would like to coordinate with the WMF Scholarship Program, the proposed program will publicize two hard deadlines for engagement. These would not be relevant to Wikimedia organizations who do not wish to coordinate with the WMF Scholarship program.

• Before the Scholarship Committee begins scoring eligible applicants. The requesting organization will be sent all eligible applicants for their group / country and will responsible for evaluation and consideration.
• After the Scholarship Committee has scored all applicants. Organizations who want to fund applicants from their group / country will be given the list of applicants and their associated scores. The requesting organization will be responsible to funding those applicants.

Once an organization has received all relevant applicants for their group / country, those applicants will be taken out of consideration for WMF-awarded scholarships.

Roles, Responsibilities and Incentives 

The Wikimania Coordinator will still own the overall process for the Scholarship program, and Grantmaking will still own the Scholarship budget. However, as stated above in changes to the Selection Process, the following summarizes the proposed roles and responsibilities in the Selection process:

• Eligibility filtering and provision of materials and tools to Scholarship Committee - Owned and executed by WMF Staff
• Evaluation of eligible applicants and delivering ranked list of potential recipients - Owned and executed by Scholarship Committee
• Final approval - Approval done by WMF Staff

Moreover, the Scholarship Committee will remain a group of 8-10 individuals who have volunteered to serve on the committee. However to increase interest and appreciation for their commitment, the following changes will be implemented:

• Offering of financial support to attend Wikimania (if needed), upon completion of duties
• More efficient use of committee member time:
  • Committee members will only review applicants after initial eligibility has been assessed by WMF
  • The number of applications to review will be stabilized around 50 per person (to the extent possible)
• Committee members will serve a more public role as ambassadors for the Scholarship Program, meeting the final recipients at Wikimania (at a mixer organized by WMF) and encouraging more individuals to apply next year

Internal process changes

The proposed timeline and responsibilities of associated WMF staff are described on Proposed 2015 Internal process.

Discussion with Scholarship Committee

The 2015 Scholarship Committee will meet to discuss this proposal. The agenda and discussion topics can be found on Discussion with Scholarship Committee.