Community Insights/2018 Report/Community Engagement Department

Community Engagement Insights 2018 Report: Support & Safety

• Read summary
• Learn the survey design
• Browse team results
  • Anti-Harassment Tools
  • Communications Department
  • Community Engagement Department
  • Community Programs
  • Community Resources
  • Contributors
  • Partnerships & Global Reach
  • Legal Department
  • Learning & Evaluation
  • Support & Safety
  • Technical Collaboration

The Community Engagement department at the Wikimedia Foundation is focused on the goal of increasing the quantity, quality, diversity, and reach of free knowledge by supporting people and organizations aligned with the Wikimedia Foundation mission. The department supports contributors by representing them in the development of new technical tools, supporting community governance and innovations, implementing trust & safety, scaling strategic programs, and by making grants to individuals, groups and organizations working on building community and growing content on Wikimedia projects and sites as well as related open knowledge projects. They are committed to supporting under-represented contributors and emerging regions, languages, and communities.

In the survey, the Community Engagement Department would like to answer the following questions:

1. What are the long-term changes in the Wikimedia movement in relation to attitudes towards health of Wikimedia communities?
2. What is the current state of attitudes towards diversity & inclusion in the Wikimedia movement?

1. What are the long-term changes in the Wikimedia movement's attitudes towards community health? (What are the year over year results in community health?)

To learn whether there have been long-term changes in community health we ask survey questions to Wikimedians who share their opinions about the community around them. These statements have been gathered from existing research literature. Important aspects of community health that we seek to hear about include collaborative intention, awareness of self and others, problem solving & negotiation, engagement, feedback and recognition, and Foundation leadership. All survey questions, except CE20, are on a scale from one to five, where one is "Strongly Disagree", and five is "Strongly Agree".

From 2017, the wording for all the measures except "Engagement" changed. Differences from one year to the next, or lack of differences, should be interpreted with caution.

Collaborative Intention includes statements about how contributors support or don't support one other (e.g. "Most people are sincerely interested in supporting each other.").

Awareness of Self and Others includes statements about how contributors perceive others emotional awareness. (e.g. "Most Wikimedia contributors are aware of their biases and patterns of behavior.").

Problem Solving & Negotiating includes statements about how contributors are able to work through conflict or problems with other people (e.g. "Most people are committed to seeking fair solutions that are responsive to the interests of all parties.").

Engagement includes statements about contributor's motivation to contribute to the Wikimedia projects (e.g. "I often think about quitting as a Wikimedia contributor.").

Feedback & Recognition includes statements about how contributors learn from others and feel achievement in being part of the community. (e.g. "Good contributors are recognized for their efforts by other Wikimedians.)

Foundation Leadership includes statement about how contributors perceive the Foundation's role in the movement (e.g. "I am confident in the Wikimedia Foundation's ability to support the global movement.").

A complete list of the statements can be found at the bottom of this section.

• 
• 

In examining the year over year results for everyone who took the survey, we found the following:

• Collaborative Intention (CE06) among Wikimedia communities has not changed from 2017.

• For all audiences between 2017 and 2018, Foundation Leadership decreased the most. There was a 5.5% statistically significant decrease in the average response between 2017 and 2018. The wording for one of the statements in this measure changed slightly, which may have contributed to this change.

• For all audiences between 2017 and 2018, "Awareness of Self and Others" (CE07) among Wikimedia communities continues to be the lowest score. However, aspects of self-awareness have increased since 2017. The question wording continues to require improvement, even though changes were implemented in 2017.

• For all audiences between 2017 and 2018, perceptions of "Problem Solving and Negotiating" (CE08) may have increased, but it is unclear whether the measurement is accurate. Caution should be drawn since the question wording has changed from 2017.

• For all audiences between 2017 and 2018, Engagement may have decreased slightly, but unclear whether the measurement is accurate. Caution should be drawn since the question wording has changed from 2017.

In examining the year over year results by community audience we found the following:

• For Wikimedia Affiliates between 2017 and 2018, perceptions toward Wikimedia Foundation's leadership decreased 18%. For the other measures of engagement and health, no significant differences between years was found for affiliates. There might be two reasons for this change. First, The survey closed on Sunday April 22nd, the last day of the Wikimedia Conference. Second, the wording from one question changed slightly, from "I am confident in the Wikimedia Foundation to support the global movement" to "I am confident in the Wikimedia Foundation's ability to support the global movement".

• Among Program Organizers, there was a 45% increase from 2.1 to 3.0 on a scale of 5 in aspects of self-awareness. This very high change should be investigated further. For the other constructs, no significant differences were found.
• Among Developers, there was a 20% average increase from 2017 results, with a 20% increase across all the constructs. This changed needs to be investigated further to understand the context around why this increase occurred.

While we have values that provide an average for each of the six measures for collaboration and engagement, for each measure we are able to identify additional significant differences between various cross-sections.

Click here to learn about additional differences observed

Collaborative intent:

• In comparing across Wikimedia projects, Dutch Wikipedia is 13.6% lower in Collaborative Intent than French Wikipedia.

Awareness of self and others

• In comparing across Wikimedia projects, Dutch Wikipedia is 28% lower in self awareness than Italian Wikipedia and 17.9% lower than German Wikipedia.

Problem solving & negotiating

• Comparing gender overall across everyone who responded to the survey, men reported 7.1% higher in perceptions towards a community's problem solving & negotiation abilities than women.
• In examining differences between genders within the audiences, women had a 15.7% lower attitudes towards the community's problem solving & negotiation abilities than men.

Engagement

• In examining across audiences, volunteer developers have a 8.7% lower engagement than program organizers.
• In examining across regions, contributors living in the South Asia region, women had a 50.4% lower engagement than men.
• In examining differences across projects, Japanese Wikipedia was statistically significantly less engaged than Spanish (+14%), Portuguese(+17%), and Western Europe (+11%).

Feedback & recognition

• In examining across audiences, volunteer developers were 9% more positive about Foundation leadership than low-activity contributors.
• In examining differences across Wikimedia projects, Dutch Wikipedia was statistically significantly less positive about Foundation Leadership that Wikimedians from several other projects. Dutch Wikipedia was 8% lower than Central & Eastern European Language Wikipedias, 4% lower than Western European Wikipedias, and 4% lower than Non-Wikipedia Wikimedia projects group. This group excludes Wikipedia, Commons and Wikidata.) .

Foundation leadership

• In examining across audiences, among developers, men had a 8% lower perceptions toward Foundation leadership than women.
• In examining across regions, from contributors who were women living in Oceania region had a 60.4% lower perceptions toward Foundation leadership than men from the same region.

2. What is the current state of attitudes towards diversity & inclusion in the Wikimedia movement?

A major goal of the Wikimedia Foundation, and the movement, is to create a project where every person can contribute. Measures for diversity and inclusion help us understand whether we are creating an environment where anyone can participate and feel welcome. Described in the paragraph below, most of the constructs measured use a 5-point scale from "Strongly Disagree" to "Strongly Agree". The construct "Valuing Diversity" is designed a bit differently. Participants will indicate whether aspects of diversity are present or not in their community. Discrimination is on a 5-point scale, but uses a frequency scale rather than Agree/Disagree.

Valuing Diversity includes statements to learn how contributors reflections on their community encouraging or valuing diversity. (e.g. "In my Wikimedia project or community, there is leadership that encourages various cultural perspectives and contributors."). This is the only measure that is not on a traditional 5-point scale, like the others.

Discrimination includes statements about how often contributors feel they are being treated unfairly based on personal characteristics (e.g. "How often have you personally witnessed someone be treated unfairly because they are part of a specific culture or social group (e.g. ethnicity, age, class, sex)?"). This question is the only question on a 5-point frequency scale.

Inclusive Interactions includes statements that measure whether communities have social space where people can communicate freely (e.g. "My Wikimedia project or community provides an environment for the free and open expression of ideas, opinions and beliefs.").

Inclusive Culture includes statements about whether individuals in the community take actions to improve diversity (e.g. "Wikimedia is making progress with diversity initiatives (e.g., Wiki Women, LGBT, Gender Gap, etc.)".

Individual Commitment to Diversity includes statements about contributor's self-perceptions towards diversity (e.g. "I believe that multicultural collaboration produces positive outcomes.").

Belonging includes statements about included people feel among the people around them. (e.g. "I feel respected in the Wikimedia movement.").

A complete list of the statements can be found at the bottom of this section.

• 
• 

Key findings from Diversity & Inclusion measures

• Across all the constructs, "Individual commitment to diversity" and "frequency of discrimination" have favorable outcomes. Individual commitment to diversity is highest among the measures while the frequency of discrimination is fairly low.

• 

• Across all the constructs, sentiments towards "belonging" and "inclusive culture" in the Wikimedia community were two of the lowest scores among the diversity and inclusion measures. A more ideal score we would want to achieve is an average of 4 on a scale of 5.
• There is room for improvement in communities "valuing diversity" of content and people. The average response to "attitudes towards the importance of diversity" is 1.5 out of a scale from zero to four. Compared with editors, program and affiliate organizers perceive that their communities place more value in diversity.

• 

• For various sub-audiences, women reported statistically significantly lower perceptions of an inclusive culture.
• Editors on the Wikimedia projects reported lower frequency of discrimination than developers, affiliates organizers and program organizers.
• Chinese Wikipedia contributors reported poorer measures than at least one other project in two areas of diversity and inclusion: frequency of discrimination and inclusive interactions
• Dutch Wikipedia contributors reported lower than at least one other project in two areas of diversity and inclusion: individual commitment to diversity and inclusive interactions.

Taking a closer look at Diversity & Inclusion measures from the 2018 survey

This is the first year that we have asked questions to communities about Diversity & Inclusion. This allows us to take a first look into differences between the various subgroups within the audiences and projects.

The average response to attitudes towards the importance of diversity is 1.5 out of a scale of 4, indicating there is room for improvement in communities valuing diversity of content and people. Compared with editors, program and affiliate organizers perceive that their communities place more value in diversity. This measure for attitudes towards the importance of diversity is designed differently from the other questions. The measurement should be read differently from the others and is not comparable. This question asks participants to indicate whether they perceive community members show the importance of diversity of content and people. The average person checked off 1.5 out of 4 options.

• Comparing across audiences, high-activity editors valued diversity slightly more than low-activity editors.
• Comparing across audiences, affiliate organizers and program organizers reported valuing diversity more than low-activity editors, high-activity editors, and volunteer developers.
• Comparing across audiences, program organizers reported valuing diversity more than low-activity editors, high-activity editors, and volunteer developers.

Among Wikimedia communities, observing or experiencing Discrimination seems to be low. Editors on the Wikimedia projects reported less frequent discrimination than developers, affiliates organizers and program organizers.

• Comparing across audiences, low-activity editors less frequently experienced discrimination than developers (+17%), program organizers (+12%) and affiliate organizers (+10%).
• Comparing across audiences, high-activity editors less frequently experienced discrimination than developers (+14%) and program organizers (+9%).
• Comparing across Wikimedia projects, Chinese Wikipedia editors more frequently experienced discrimination than editors on Commons (+10%), French Wikipedia (+11%) and Western European Wikipedias (+9%).
• Comparing across the types of communities people were thinking about when they answered this question, those thinking about their Wikipedia language experienced discrimination less often than those thinking about their affiliate group (-9%) and another group type (-9%).
• Comparing across the types of communities people were thinking about when they answered this question, those thinking about their Non-Wikipedia Wikimedia project experienced discrimination less often than those thinking about their developer community (-8%), affiliate group (-12%) and another group type (-11%).

Among Wikimedia communities, perceptions of Inclusive Interactions with community members seems to be a bit lower than what would be ideal.

• Comparing across the types of communities people were thinking about when they answered this question, those thinking about their Wikipedia language reported statistically significantly lower perceptions towards inclusive interactions than those thinking about their non-Wikipedia Wikimedia project (+9%), affiliate group (+10%) and another group type (+11%).
• Comparing across Wikimedia projects, contributors from other Wikimedia projects (projects that are not Wikipedia, Wikidata or Commons) reported statistically significantly higher perceptions towards inclusive interactions than Dutch Wikipedia (-15.7%) and Chinese Wikipedia (-13.1%).

Among Wikimedia communities, a climate of Inclusive Culture seems to be lower than what would be ideal, and is the lowest among the measures of diversity and inclusion. For various sub-audience, women reported statistically significantly lower perceptions of an inclusive culture. The average response across audiences was 3.57, which was the lowest score across the diversity and inclusion measures.

• Looking at gender, men from various cross sections reported statistically significantly higher than women overall (-8%) as well as women from Oceania (-64%), from low-activity contributors (-12%) from developer communities (-54%).

Individual Commitment to Diversity seems to be high across all the audiences, but there are particular differences that should be noted from various sub-audiences. Affiliate organizers reported higher commitment than online contributors to diversity. Dutch Wikipedia is statistically significantly lower from French Wikipedia.

• Comparing across audiences, Wikimedia Affiliate reported higher self-perceptions towards individual commitment towards diversity than low-activity contributors (-9%) and high-activity contributors (-7%).
• Comparing across the types of communities people were thinking about when they answered this question, those thinking about their Wikipedia language reported lower self-perceptions towards inclusive commitment than those thinking about their non-Wikipedia Wikimedia project (+4%), affiliate group (+8%) and another group type (+8%).
• Comparing across projects, Dutch Wikipedia reported lower self-perceptions towards inclusive commitment than French Wikipedia (+11%).

Sentiments towards Belonging in the Wikimedia community is one of the lowest among the diversity and inclusion measures. For a couple sub-audiences, men reported higher measures of belonging than women. Across all respondents, the average response for this question was 5.59, the second lowest measure.

• Looking at gender, men from various cross sections reported statistically significantly higher than women in Oceania (-47%) and from developers (-9%).

Report Summary

For CE Insights 2018, the Community Engagement department is most interested in learning about changes to community health of Wikimedia Communities as well as the climate for diversity and inclusion. To learn about this information, the survey asks two sets of questions which measure community health in terms of "Collaboration & Engagement" and "Diversity & Inclusion". In total, there are 65 statements that are categorized into 13 constructs. For example, collaboration & engagement measures include six sets of questions, including one about Foundation leadership and one about Engagement. Survey participants "Agree" or "Disagree" with these statements to provide a score from 1 to 5.

Collaboration & Engagement

• Changes across the measures for collaboration & engagement seem to be minimal from 2017. Even though the wording of various constructs changed, the change seems to be small between 3 and 5%. Due to the wording changes to some of the constructs, it is difficult to know whether the changes are real.
• Awareness of Self and Others continues to have a lower average by far. The average for this question is about 3.05, which is considerably lower than the next lowest construct at 3.46 out of 5. The ideal place for constructs is typically an average of 4 of 5. While this question continues to need improvement in its design, the concept it measures seems to remain an area for improvement. Among Program Organizers, there was a 45% increase from 2.1 to 3.0 on a scale of 5 in Awareness of Self and Others. This very high change should be investigated further. For the other constructs, no significant differences were found.
• For all audiences between 2017 and 2018, Foundation Leadership decreased the most, which is something to watch. There was a 5.5% statistically significant decrease in the average response between 2017 and 2018. The wording for Foundation Leadership changed slightly. Among affiliate organizers between 2017 and 2018, perceptions toward Wikimedia Foundation's leadership decreased 18%. For the other measures of engagement and health, no significant differences between years was found for affiliates. Some Wikimedia affiliate organizers may have attended the Wikimedia Conference, which co-occurred with the survey and may have influenced results.
• Among Developers, there was a 20% average increase from 2017 results, with a 20% increase across all the constructs. This change needs to be investigated further. While staff are included in this survey, in other questions, staff have made very little changes to responses. However, we need to investigate this.
• Dutch Wikipedia contributors surfaced several times in both Collaboration & Engagement and Diversity & Inclusion measures as being statistically significantly lower from their Wikimedia project peers. Dutch Wikipedia was statistically significantly lower in Collaborative Intent, Awareness of Self and Others, Feedback & Recognition, Individual Commitment to Diversity, and Inclusive Interactions.

Diversity & Inclusion

• Individual Commitment to Diversity and Frequency of Discrimination have favorable outcomes. Individual commitment to diversity is highest among the measures while the frequency of discrimination is fairly low.
• There is room for improvement in communities Valuing Diversity. The average response to "attitudes towards the importance of diversity" is 1.5 out of a scale from 0 to 4 (this question is formatted differently from the others). Compared with editors, program and affiliate organizers perceive that their communities place more value in diversity, about 2.5 out on a 0 to 4 scale.
• Sentiments towards Belonging and Inclusive Culture in the Wikimedia community were two of the lower scores among the diversity and inclusion measures. The scores for these constructs were 3.59 and 3.57, respectively. For Inclusive Culture, The average response for women in the survey was 8% lower than men, which was statistically significant. Similar differences in gender were found among developers and low-activity editors.
• Editors on the Wikimedia projects reported statistically significantly lower Frequency of Discrimination than developers, affiliates organizers and program organizers.

Most useful findings*

• While the construct “Foundation Leadership” continues to be the second strongest asset for healthy community collaboration the construct demonstrated a significant decrease from 2017 to 2018 (p<.001)
• Similarly, the strongest asset from last year’s survey remains in the lead, but showed a similar decrease. This change was significant (p<.05), however the score continues to reflect strength in the climate of “Engagement“ (Mean= 4.08)
  • This means that most participants agreed strongly that they “would recommend Wikimedia as a great place to contribute,” are inspired to contribute by “the vision to freely share in the sum of all human knowledge,” “take pride in contributing,” don’t often think about quitting,” and “expect to be contributing two years from now”.  

• On “Awareness of Self and Others" (CE07), the construct remains the weakest asset for a healthy collaboration climate, however there was demonstrated improvement which was significant in 2018 (p<.01).  
  • Still, Dutch Wikipedia was statistically lower from the strongest scoring community, Italian Wikipedia, and was consistently lower than all other Wikimedia projects (e.g. Wikipedia, Wiktionary, Wikibooks, etc.).
  • Similarly, Dutch Wikipedia (-15.7%) and Chinese Wikipedia (-13.1%) less often reported perceptions of Inclusive Interactions than other Wikimedia projects, this was also significant (p<.05).

Next steps

• Create and present data summary to CE Department
• Connect data to incorporate movement organizers
• Consider further refinement of scale items which could be improved
• Evaluate the significance of changes surfaced in subsets of "Awareness of Self and Others" (CE07).
• Evaluate the persistence of lower ratings of Foundation Leadership subcomponents related to communicating (CE15.2) and leading the global movement (CE15.3) across CE Insights 2017 and 2018.

Construct Question Design

Click here to see the items making up the different community health constructs

Constructs for Collaboration & Engagement

CE06 Assess Climate for Collaborative Intention

To what extent do you agree or disagree with each of the following related to the Wikimedia movement, including all Wikimedia projects, organizations or offline activities:

• Most people are sincerely interested in supporting each other.
• Most people are committed to building long-term cooperative relationships.
• Many contributors see others as adversaries, seeing conflict as a battle to win at any cost.
• Most Wikimedians with collaborative projects work together well.
• Most Wikimedia contributors are good at reaching out when they need to consult with others.
• Most contributors do not face social barriers to working effectively online.

CE07 Assess Climate for Awareness of Self and Others

To what extent do you agree or disagree with each of the following related to the Wikimedia movement:

• Most Wikimedia contributors lack self-awareness.
• Most Wikimedia contributors know how to motivate others.
• Most Wikimedia contributors are aware of their biases and patterns of behavior.
• Most Wikimedia contributors are not aware of their own defensiveness to the situation at hand.
• Most Wikimedia contributors seek out and value feedback.
• Most Wikimedia contributors have a difficult time understanding and empathizing with others.

CE08 Assess Climate for Problem Solving & Negotiating

To what extent do you agree or disagree with each of the following related to the Wikimedia movement:

• Most people are dedicated to working together to solve problems.
• Most people attempt to coerce others.
• Most people are committed to seeking fair solutions that are responsive to the interests of all parties.
• Most people are sincerely willing to talk and negotiate and are able to separate “people issues” from the problems.

CE09 Assess Climate of Engagement (Key to Inclusion)

To what extent do you agree or disagree with each of the following:

• I would recommend Wikimedia as a great place to contribute.
• The Wikimedia vision is to freely share in the sum of all human knowledge.
• This vision inspires me to contribute more to Wikimedia than I would somewhere else.
• I take pride in contributing to the Wikimedia projects.
• I often think about quitting as a Wikimedia contributor.
• I expect to be contributing to Wikimedia two years from now.

CE10 Assess Climate of Feedback & Recognition

To what extent do you agree or disagree with each of the following related to the Wikimedia movement:

• My contributions to Wikimedia are often challenged or removed inappropriately.
• Good contributors are recognized for their efforts by other Wikmedians.
• Wikimedians are free to experiment and try new ideas.

CE15 Assess Foundation Leadership

To what extent do you agree or disagree with each of the following:

• The Wikimedia Foundation makes it clear that volunteers and partners are key to the success of the movement.
• The Wikimedia Foundation communicates well about their projects and initiatives.
• I am confident in the Wikimedia Foundation's ability to support the global movement.

Constructs for Diversity & Inclusion

CE20 Wikimedia community attitudes toward value of diversity

Thinking about your home project or community which of the following apply? (choose all that apply)

• In my Wikimedia project or community, there is leadership that encourages various cultural perspectives and contributors.
• In my Wikimedia project or community, administrators demonstrate through their policies and actions that including content and contributors from different backgrounds is important.
• In my Wikimedia project or community, contributors are committed to including content from different cultures and perspectives.
• In my Wikimedia project or community, contributors respect individuals and value their differences.
• None of these options apply in my Wikimedia project or community

CE21 Community recall filter

When you answered the previous questions, what type of Wikimedia community were you thinking about? (select all that apply)

• The Wikipedia language I edit the most (e.g Spanish Wikipedia)
• The Non-Wikipedia Wikimedia project I edit the most (e.g. Wikisource, Wikidata)
• My Wikimedia developer community (e.g. Phabricator, Wikitech)
• My affiliate group (e.g. Wikimedians of Wikiland User Group)
• Another type of group

CE22 Discrimination

Thinking about that same home project or community, in the past 12 months, how often have you:

• personally witnessed someone be treated unfairly because they are part of a specific culture or social group (e.g. ethnicity, age, class, sex)?
• been treated unfairly based on your association with a particular cultural group or social category?

CE23 Inclusive interactions

To what extent do you agree or disagree with each of the following:

• Contributors of different backgrounds interact well in my Wikimedia project or community.
• Contributors of different backgrounds are valued equally in my Wikimedia project or community.
• Racial, ethnic, and gender-based jokes are not tolerated in my Wikimedia project or community.
• My Wikimedia project or community provides an environment for the free and open expression of ideas, opinions and beliefs.

CE24 Inclusive Culture

To what extent do you agree or disagree with each of the following:

• Wikimedia is making progress with diversity initiatives (e.g., Wiki Women, LGBT, Gender Gap, etc.).
• Different project and contributor groups are treated fairly in the Wikimedia movement.
• In the Wikimedia movement, contributors appreciate others whose context and cultural background is different from their own.

CE25 Individual Commitment to Diversity

To what extent do you agree or disagree with each of the following:

• I am comfortable working with different kinds of people (e.g. ages, genders, ethnicities, religions, etc.)
• I am confident when voicing my opinion, even when it’s different from those around me.
• I believe that a diverse community positively impacts the quality of Wikimedia content.
• I believe there is community policy and systems in place to take appropriate action to respond to incidents of harassment.
• I believe that multicultural collaboration produces positive outcomes.

CE26 Feelings of Belonging

To what extent do you agree or disagree with each of the following:

• I feel like I belong in the Wikimedia movement.
• I feel respected in the Wikimedia movement.
• There is open and honest communication between different organizations and groups in the Wikimedia movement.
• I am satisfied with the Wikimedia movement decision-making process.

Notes