Wikimedia Meta-Wiki

Research:Spring 2015 Inspire campaign

Created
19:36, 5 May 2015 (UTC)
Duration:  2015-March – 2015-April
This page documents a completed research project.


The Inspire campaign was an experimental initiative undertaken by the WMF Community Resources team (formerly Grantmaking) in March 2015. The campaign had two primary goals:

Pilot a proactive, challenge-based model for finding and funding impactful, community-driven projects, and
Make a substantial impact on Wikimedia's gender gap, the strategic issue selected to be the focus of the campaign, in terms of content, contributors, and project leaders.

Campaign overview

edit

The Inspire campaign was advertised through Central Notice banners on all Wikimedia projects in 15 languages, through a post on the Wikimedia Blog, and through posts to relevant mailing lists and social media. The campaign was focused on eliciting novel approaches to increasing gender diversity (in terms of both content and contributors) in Wikimedia projects.

A total of 676 registered users participated in the Inspire campaign in the IdeaLab between March 3rd and April 1st 2015. Participants created 266 ideas. On May 1, 2015 WMF funded 16 projects that stemmed from Inspire campaign proposals.

Evaluation goals

edit

This report represents an attempt at a comprehensive, near-term evaluation of the campaign itself. It focuses primarily on the initial, idea elicitation stage of the campaign, which took place within the IdeaLab between March 3rd and April 1st. It contains results, analysis, and recommendations based on a survey of campaign participants and from information about the characteristics and dynamics of the content contributed during the campaign and the editors who participated.

Our goal in collecting and sharing these data is to provide a window into who participated in the campaign, what they did, and how they felt about that experience. This information will inform WMF's grantmaking priorities; help us address pain points in the tools, documentation, and workflows we provide to community participants; and identify opportunities for future proactive funding campaigns.

Key findings

edit
Read more: additional data and findings are available in the survey report.

Achievements

edit
Massive scaling of ideation and grant-making process
The campaign exceeded initial goals for ideas (100) and participants (500) within the first two weeks. The number of participants in the campaign, and the number of ideas submitted, exceeded the totals for the previous two years: between February 2013 and February 2014, there were only 148 ideas created in the IdeaLab and 2103 participants. Within the month of March, the monthly idea count was 4000% higher, and the number of participants 1800% higher, than historical monthly averages.
Gender diversity
Non-male gendered individuals were well represented among survey respondents (34%), committee members (57%), and grant-funded project leadership (88%).
Geographic diversity
Editors from 89 countries participated in the campaign. The top three countries were the United States, Germany, and India.
Wikimedia project diversity
Survey responses indicate that participants came from a variety of projects, not just English Wikipedia. Projects include Wiktionary, Wikisource, Wikivoyage, MediaWiki, and multiple language editions of Wikipedia.
Process innovations
Tools such as the FormWizard (easily create ideas), AddMe (easily endorse and join ideas), IdeaLab profile pages, idea recommendations, and a friendly space policy were generally well received by participants, particularly idea creators, and helped support a large-scale community-driven open innovation process.

Challenges

edit
Language barriers
Although the content of the main IdeaLab pages and the interface text for gadgets and infoboxes were translated into many languages, ideas submitted in languages other than English could not be easily translated.
Social barriers
Some participants were discouraged from contributing because of the tone of the criticism they received on their ideas, or the general tone of the discussion they witnessed on the main campaign talk page. Several female-identified survey respondents stated that they experienced misogynistic or anti-woman sentiments from other campaign participants.
Technical barriers
Several survey respondents reported difficulty using the FormWizard to create their idea, or were confused when their idea did not show up in the main idea list immediately after publication.
Ideological opposition
Several survey respondents indicated their opposition to the design of the idea template, which places endorsements on the content page but opposition (and general commentary) on the idea talkpage. Other respondents indicated that they were ideologically opposed to the idea of the Wikimedia Foundation organizing a gender gap-themed grant campaign.
Process confusion and miscommunication about expectations
Some participants were unsure how to progress from idea creation to proposal submission, as well as deadlines, budgets, and the significance of endorsements. Idea creators expressed disappointment when they were unable to find guidance on next steps, or when their ideas did not receive any feedback from Foundation staff or other volunteers. Other participants thought that successful ideas would be taken up as independent projects by the Wikimedia Foundation itself, and were dismayed that no one claimed their idea.

Recommendations

edit
Increase on-wiki community support
Many participants expressed frustration at the lack of feedback on their ideas. Others were unsure of how to refine or expand their proposals. And still others found the tone of the discussion around individual ideas and the campaign in general to be hostile and alienating. Finally, there were a (thankfully, small) number of blatantly offensive submissions and comments made during the campaign. Future campaigns should include more dedicated community organizers who perform a variety of idea creator support, content curation, and moderation/monitoring roles.
Improve campaign messaging
The purpose of the campaign (fund community-driven projects that address the gender gap) wasn't always communicated clearly, nor were other crucial pieces of information about the idea submission and feedback process, eligibility criteria, selection process, idea creator responsibilities, etc. The information that was available was scattered across multiple pages. In addition to providing more personal support (see above), any future campaign should provide all relevant information in a single, central location (similar to Grants:IEG).
Make it easier to find relevant ideas and people
The massive influx of new ideas and participants during the Inspire campaign resulted in a degree of information overload. Even the campaign organizers found it challenging to keep up with the stream of activity. Features such as the Idea Leaderboard, user profiles, automated idea recommendations (available to people who created profiles), and bot-updated activity feeds and idea profile cards offered a substantial improvement over the standard MediaWiki interface, especially for users who are not accustomed to tracking activity through power user tools such as watchlists, RelatedChanges feeds. However, it was still difficult to find ideas that are relevant to your interest, seek out potential collaborators, and even keep track of the development of projects you joined and conversations you are involved in, across so many different pages.
Continue to prioritize friendly spaces, constructive criticism, and lightweight ways to get involved
While opinions were divided on the usefulness of process innovations deployed in the IdeaLab, such as friendly space expectations, and the separation of endorsements from general discussion (including threads devoted to listing ‘opponents’ of an idea). We saw no indication that separating endorsements from other forms of feedback, including categorical opposition, hampered the feedback process. In fact, several idea creators called out constructive criticism as a positive development. A gadget that allowed users to endorse and join ideas easily was widely used, and counting endorsements provided a useful (though not definitive) way of identifying ideas that resonated with members of the community. And the friendly space expectations document seemed to provide some measure of assurance for participants who would otherwise have been hesitant to participate publicly in discussions around a polarizing and controversial subject like the gender gap.
Debug, browser test, and internationalize all tools and documentation
Gadgets and templates should be thoroughly browser-tested; fonts and colors should be tested against current standards for web accessibility; and all instructional and procedural documentation should be translated into as many languages as possible before the beginning of the campaign, and updated as the source content changes.
Improve multilingual support for idea creators
Several dozen ideas were created in a language other than English. This represents a huge missed opportunity for the campaign, since little was done during the campaign to make these ideas accessible to speakers of other languages. Although the translation extension on Metawiki is configured to make English the base page language, workarounds should be developed to allow ideas submitted in non-English languages to be translated into English as well as other languages, to make the ideation process more inclusive.

Campaign activity

edit
return to top
return to top
Cumulative ideas created per day during the 2015 Wikimedia Inspire campaign
Monthly edits to IdeaLab pages from July 2013 through April 2015. Edits by bots and WMF staff are not included.

Summary statistics

edit
Ideas - all 266
Ideas - eligible proposals 43
Ideas - selected proposals 16
Participants - all 809
Participants - registered 676
Participants - unregistered 133
Pageviews - all 23463
Pageviews - mobile 1298
Pageviews - desktop 22165
Endorsements 350

Historical comparison

edit

Historically, the IdeaLab has received 6 ideas per month and hosted 35 registered editors, on average, with a maximum of 19 ideas (in March 2014) and 146 participants (in January 2015*) Totals for the Inspire campaign of 266 ideas and 676 editors represent a 4333% and 1831% increase, respectively, over these historical averages. The number of edits also increased exponentially over the historical average (190/month) to 5293 in March 2015, a 2686% increase. All counts exclude edits by bots and Wikimedia staff.

*The majority of the January 2015 participants were discussing a single idea: WikiProject Women, which was actually created in anticipation of the upcoming Inspire campaign. The next highest monthly editor total was 58 during March 2014, in which month the IdeaLab was being piloted as an incubator for proposals for the upcoming round of Individual Engagement Grants.

edit
return to top
return to top
  • edits
    edits
  • ideas submitted
    ideas submitted
  • views of Grants:Inspire
    views of Grants:Inspire
  • comments on ideas
    comments on ideas
  • endorsements of ideas
    endorsements of ideas
  • joins of ideas
    joins of ideas

Ideas by endorsements

edit
return to top
return to top

The endorsements are counted by a bot that reads the edit comments on the page, so that endorsements may in some cases be over or under-counted. To see the exact number of endorsements an idea has received, please see the "Endorsements" section on the idea page itself.

Editors by country

edit
return to top
return to top

'Unique registered and unregistered editors of campaign-related wiki pages during the month of the campaign, by country.

Pageviews by country

edit
return to top
return to top

Views of Grants:Inspire main page during the month of the campaign, by country.

Survey results

edit
return to top
return to top

For the complete survey results, see: Research:Spring_2015_Inspire_campaign/Survey


A survey was conducted in April 2015 to evaluate the experience of participating in the Inspire Campaign, which focused on the gender gap and was hosted on the IdeaLab. The survey was conducted by the Wikimedia Foundation Community Resources team.

The survey was delivered to editors' talk pages on Metawiki. It was open between April 13th and April 20th. Editors were selected to receive the survey if they had edited any pages within the IdeaLab from a dedicated Wikimedia account between March 1st and April 1st. Every attempt was made to exclude Wikimedia staff accounts and bot accounts.

In all, the survey was delivered to 698 editors, and 145 valid responses were recorded, for a response rate of 21%.

Overview of findings

edit
  • 34% of respondents were women
  • 78% of respondents were veteran Wikipedians (1+ years editing)
  • 44% of respondents were recruited through Central Notice banners
  • At least 47% of respondents listed English Wikipedia as their home project
  • Commenting on ideas was the most popular way of participating (70% of respondents), followed by endorsing ideas (50%)
  • 38% of respondents submitted at least 1 idea during the campaign
    • 37% of these idea-creators turned their idea into a grant proposal
    • Only 21% of respondents joined an idea created by someone else
  • Satisfaction ratings averaged between 3.2 and 3.6 (out of 5) for all measures
  • No significant differences existed between the satisfaction ratings of:
    • idea-creators vs. other respondents
    • men vs. women
    • veteran editors vs. those with less than 1 year of editing experience

Key outcomes

edit

Caveat: the survey was delivered to each participating editor's Metawiki talk page, rather than the talk page on the editor's home wiki. As such, veteran Wikimedians, and especially Meta-pedians, are likely over-represented in the response set.

Women were well-represented
Women participated at roughly double the rate of (estimated) female participation in English and other major Wikipedias.
Most respondents were experienced Wikipedians
The vast majority of respondents (78%) had edited Wikimedia projects for more than a year.
Around half of respondents were not English Wikipedians
Although English Wikipedia, the largest Wikimedia project by far, was well represented, approximately half of the respondents listed another project as their home wiki. Coupled with the high percentage of respondents recruited through Central Notice banners (44%), this suggests that Central Notice is an effective strategy for encouraging participation by editors from a diverse set of Wikimedia projects.
Idea creators generally appreciated feedback
Endorsements were motivating and criticism of the idea was valued, as long as it was constructive.
  • "My core Idea was challenged for being essentialist, which had the effect of having to better understand what I was trying to do."
  • "yeah very legitimate queries I received / endorsements were encouraging."
  • "I liked that discouragement and opposition was moved away to talk page. That really helped others to analyze the ideas without bias."
However, lack of feedback was powerfully demotivating for idea creators.
  • "No feedback, no help, from anyone. no instructions on how to turn it into a grant proposal... no guidelines."
  • "one person responded in terms of participation and i got no feedback on my request for information about how to apply, how much money to request and how to implement my idea... I am new here but at least i was trying..."
And incivility and was a major issue.
  • " I was deeply offput by the commentary around the project from Wikimedians opposed to this idea and preferred not to get involved."
  • "The opposition posts were many and nasty."
  • "The amount of anti-woman sentiment was very distressing."
Some commenters objected strenuously to the concept of endorsements
Although there was almost no negative feedback on the endorsement process from respondents who also created ideas
  • "Facebook syndrome: You can state your enthusiasm, but not your dislike."
  • "The problem with this campaign, is that there's not really a metric of disagreement."
  • "Discontent on ideas is shifted out of view."
Many idea creators were unsure about expectations, next steps
Respondents expressed frustration about a lack of clear instructions, and confusion about what was expected from them after they initially submitted their idea.
  • "I was confused that no-one followed up with me after I submitted an idea."
  • "It was a difficult process to navigate in terms of understanding what the steps were, what exactly were the phases, when to add what additional templates."
  • "Information was scattered across many pages. I did not realize that there were special granting abilities related to Inspire. Wikimedia grant stuff in general was confusing for me, and I was unsure of what scale of project to propose for any sort of grant."
  • "I didn't understand what it would have implied to join an idea."
Others felt that the grantmaking process lacked transparency.
  • "No indication on how Wikimedia would respond to the ideas. Would a response be offered to every idea? I find WM's lack of transparency frustrating."
  • "I wasn't convinced by the process that this was a worthwhile endeavor. The selection criteria was absent, the reviewers invisible, and the credibility of the existence of funding sources wasn't established."
Profiles and matching were relatively unseen and under-utilized
very few people created profiles. One participant that did create a profile appreciated the ability to see other users' faces. However, other profile-creators were confused by the recommendations, and by the fact that their profile was not visible in other parts of the IdeaLab instantaneously.
Participants appreciated the gadgets... when they worked
several respondents indicated that they appreciated the ease of use of gadgets such as the FormWizard and AddMe gadgets, which facilitated idea creation, expansion, endorsement, and joining. However, other participants found these gadgets to be buggy and inconsistent.
Some commenters objected to the campaign as a whole
some respondents considered the Gender Gap theme of the campaign to be illegitimate, and/or the Wikimedia Foundation's role in soliciting ideas to fund gender-gap focused projects to be inappropriate.
  • "I found the campaign insulting to all genders of editors."
  • "I'd be pleased if IdeaLab ceased to exist."
Editors have lots of intriguing ideas for future campaigns
respondents suggested a diverse set of topics for future Inspire campaigns.
  • "I'd like to see more multimedia projects."
  • "Combating systematic bias with regards to low editing habits of Global south Wikimedians."
  • "Paid editing, adminship reform, civility"
  • "intellectual property/copyright issues, human resource-type processes and infrastructure for open communities, small language communities, open education, better coverage of marginalized forms of knowledge, gender gap again!"
Most respondents would participate in a future Inspire campaign
of the 93 respondents who answered the question "would you participate in another campaign?", 82 responded yes or maybe, only two responded negatively or expressed reservations. Among those who responded positively, some of the primary considerations were the topic chosen for the campaign, and having enough time to participate.


Grantmaking reflections

edit

This campaign combined both an idea-generation phase and a funding phase, and outcomes of funded projects will be assessed as projects are completed and grantees report back on what they’ve learned and accomplished. We did, however, also want to capture here some reflections of what the grantmakers learned from the funding phase of the Inspire campaign.

What worked well

edit
  • Inspire brought new participants to our grantmaking process. From idea creators to endorsers, proposers and committee members, we saw lots of people actively participating in the grants space on meta-wiki for the first time.
  • Inspire broadened the base of grantees we support, including bringing in more women as project leaders than ever before. 88% of the funded projects are led by women (compared with less than 1/3 of IEG and PEG grantees in the first half of 2014). And 81% of Inspire grantees are receiving a grant from the Wikimedia Foundation for the first time.
  • Inspire brought more gender-focused grants into our funding portfolio. We’ve nearly doubled, from 9% of IEG and PEG funds in the first half of 2014 to now 18% in the first half of 2015.
  • Inspire gave us the opportunity to rethink our old processes in new ways. Bringing together committee members and program officers from 2 different WMF grantmaking programs into 1 joint Inspire funding process got us thinking differently about routine systems. We’re increasingly likely to cross-pollinate review tools and strategies across programs in the future thanks to this experience.
"I was inspired by WMF staff and volunteers, their commitment to the campaign, and their competence in handling issues (e.g. abuse from within the community). I also felt like it was more productive to create, refine, evaluate ideas from the same campaign and trying to address the same issue" - Inspire funding committee member

What needs improvement

edit
  • Launching this pilot took many hours from many staffers, contractors, and volunteers! As noted in the recommendations section above, processes and tools will need further refinement to go beyond a pilot and create a sustainable campaign system.
  • Grant proposals take time to develop and review, and the campaign’s tight timeline didn’t allow all proposals enough room to mature. We saw a lot of proposals that had potential, but still needed more time and discussion to get from an idea to a fundable project plan. Perhaps not surprisingly, proposals from the Global North and native English speakers seemed to fare better in this quick proposal cycle. Any future campaigns should build in more time between idea creation and the funding cycle, perhaps considering running an idea campaign in the quarter before a funding open call.
  • Responsive grantmaking also needs adequate time. Our PEG program officer had a backlog of non-gender focused grants during this period that also needed attention. Future campaigns should be planned around other grant cycles, with time adequately resourced for both proactive and responsive grantmaking. Any changes to existing grant programs require more advance communications.
"Provide training (grant-writing workshops) and information sessions (IRC sessions); provide more time between promoting campaign and accepting grant proposals; promote outside of wiki and online communities" - Inspire funding committee member

Next steps

edit
  • We’ll be evaluating outcomes and impact on gender diversity from the 16 funded projects over the coming year and sharing this with the community.
  • Based on the outcomes of this pilot, we are considering running other IdeaLab campaigns over the coming year, focused on different strategic topics, as a complement to our existing grantmaking programs. As WMF’s annual planning process moves forward towards a board-approved plan for 2015/2016, we’ll be looking at our available resources before making any final decisions.

References

edit
Retrieved from "https://meta.wikimedia.org/w/index.php?title=Research:Spring_2015_Inspire_campaign&oldid=24695673"