Community Health Metrics/Drop-off Framework

This framework has been published at the Wiki Workshop 2021, April 14, 2021, Lubiana, Slovenia (virtual event) (PDF).

Paper: Miquel-Ribé, M., Consonni, C., & Laniado, D. "Wikipedia Editor Drop-Off A Framework to Characterize Editors' Inactivity". published at the Wiki Workshop 2021, April 14, 2021, Lubiana, Slovenia (virtual event) (PDF).

In this page, we present an approach to characterize Wikipedia’s editor drop-off as the transitional states from activity to inactivity. Our approach is based on the data that can be collected or inferred about editors’ activity within the project, namely their contributions to encyclopedic articles, discussions with other editors, and overall participation. Along with the characterization, we want to advance three main hypotheses.

This framework is both as a preliminary stage of our research to understand editor drop-off and as a flexible frame to look at the phenomenon that we believe can be useful in the future. Furthermore, by characterizing drop-off and identifying interaction patterns that may be associated with it, it may be possible to assess the general health of a community, and ultimately propose changes to improve it.

Introduction

Even though the project is undeniably successful along many dimensions - from worldwide popularity and geographical spread to its adoption even among professionals and academia - it has issues in being able to renew or grow its communities.

In many Wikipedia language communities, the decline or stabilization in the number of editors was detected more than a decade ago. Despite dedicated interventions by the Wikimedia Foundation and other organizations, and groups of editors (English Wikipedia editors' efforts are coordinated in the Editor Retention WikiProject), it has not been possible to reverse this trend (visible in stats.wikimedia.org). While one can speculate whether the project is not as attractive to newcomers as in the early days, we can observe that there are both language editions and topics with very different levels of quality. Hill and Shaw hint that a possible factor that is restraining Wikipedia’s growth is the need to create, monitor, and maintain quality – including the management of tools and policies to counter vandalism and attacks caused by Wikipedia’s growing visibility. Thus, if newcomers are crucial to renew the communities, experienced editors are essential to maintain the project.

Any Wikipedian that is active in a particular moment can become inactive, and, eventually, withdraw from the project. In this context, the risks associated with a declining community are especially relevant. Thus, understanding why and how editors leave Wikipedia has become an important question that has so far received limited attention, not only to try and mitigate its causes, but also to foster the future of the encyclopedia.

Previous research has described the relationship between participation and editor retention and, to a lesser extent, using techniques from survival analysis, has depicted the contributor lifecycle. While work on new editor retention focuses on editors leaving in an initial period, often called the “onboarding phase”, reasons and dynamics may typically be different for users leaving once having settled in the community and having undergone the learning process that becoming a community member entails.

In the rest of this page, we first present in Section 2 the background on editor drop-off that we collected from community essays on the subject. In Section 3, we present a framework with metrics to describe different states of transition from activity to inactivity; rather than asking Wikipedians to provide reasons for drop-off, we propose a computational approach to characterize their activity and interaction patterns. In Section 4, we outline the main hypotheses that have been advanced in the state of the art to explain the phenomenon of drop-off. Finally, in Section 5, we discuss the next steps and the implications of understanding editor drop-off.

We think of this work both as a preliminary stage of our research to understand editor drop-off and as a flexible framework to look at the phenomenon that we believe can be useful in Wikipedia or any other collaborative project where contributions are tracked.

States of Drop-off

The process by which an editor leaves Wikipedia presents a great variability in its motives and can lead to a partial or complete stop in editing. This process is also known on Wikipedia as retiring or withdrawing from contribution. Even though a variety of external causes - even death - can influence the process by which editors stop contribution, we limit the scope of this study to what we can observe from the ensemble of events that happen on-wiki.

We define drop-off as a user-driven voluntary choice to become inactive, thus excluding blocks or bans where a user loses their ability to contribute following a community decision. Being blocked or even banned for a long period of time from a Wikimedia project implies the impossibility of participating, but it is not a drop-off. Instead, deciding not to return to editing after block or ban, or even forcing a ban into what is known as “retirement by admin,9 ” are different ways in which an editor can decide to withdraw from Wikipedia.

The three states of editor drop-off in their most common forms are “wikibreaks,” “semi-retiring,” and “retiring.”

Wikibreak (temporary drop-off).

Figure 1: The different states of drop-off related to activity and their possible transitions.

A wikibreak or wiki-absence is a period during which an editor decides not to edit, although only temporarily. The community essay explaining the concept exists in 47 Wikipedia language editions, but the template a user adds to their userpage to signal a wikibreak exists in 74 editions. The template can be personalized with reasons such as “being busy,” “pursuing an education,” “health reasons,” “vacation,” “technical reasons,” or “grieving,” among others. Wikibreaks or reductions in activity could reflect significant issues, either independent of or related to Wikipedia. We may hypothesize that the same reasons that can encourage a Wikipedia editor to take a wikibreak could also be the cause for drop-off, and even though the absence is meant to be temporary, some Wikipedians may not return.

Retiring (permanent drop-off).

Retiring means leaving Wikipedia. Even though a user may eventually return, it is generally a permanent status rather than a temporary absence. The difference between retirement and a wikibreak is therefore the intention, and, as such, the community suggests avoiding unnecessary drama. When an editor has the intention of leaving, they can use a template to signal their retirement. Usually, users abandon their account and simply stop all editing activity, but in some cases, an editor can ask for a procedure called “courtesy vanishing,” which means having their username renamed and their userpages blanked or deleted, while the contributions made to encyclopedic articles as well as user talk pages remain. This differentiates this procedure from what happens when a user leaves a social network and decides to “purge” and delete all the submitted content.

Semi-retiring (partial drop-off).

A less radical option is semiretirement, which means becoming less active. Highly active Wikipedians commonly report that it is hard for them to take wikibreaks and find it challenging to stop editing at once. A semiretired editor can pass from making thousands of edits per month to making just a few. Similar to wikibreaks and retiring, editors can signal their status by adding a “semi-retired” template to their userpage along with a message, so that other editors know that they can communicate with them, but they may not get a quick reply. With respect to the other forms of editor drop-off, semi-retiring is the one that implies a less drastic decision.

Proposed Metrics

“I like you when you do not edit, because it’s like you are absent”

Following Neruda’s famous verse, we can assume an editor is absent when they are not editing, but it is harder to establish if they have left. Contrary to retention – where we can clearly define when an editor has engaged after a given period of time and number of contributions – in our case it is always uncertain whether an editor has made a final decision to leave and will not return later. In fact, we can only look at templates added by a user to disclose their status, or measure editor activity to infer it, but we cannot be entirely certain. For defining active editors, a 5-edit per month threshold is commonly considered, but being absent is more complex, and cannot be captured by a one-size-fits-all metric. In the following subsection, we will present different metrics to assess both inactivity and decrease in activity

Inactivity Metrics

Inactivity, or the period of time elapsed without editing, is an indicator of temporary or permanent drop-off. The longer the inactivity, the more likely it will become permanent, but how long is long enough?

Fixed Thresholds.

The Wikipedia communities across several languages usually establish thresholds that are helpful for considering whether an editor has gone through one of the phases defined in Section 2. For example, for editor retention, it is assumed that an editor has survived when they have edited after a period of 60 days since their first edit . This threshold is also reinforced by the finding that the median lifetime of newcomers is found to be around 53 days . For defining drop-off for experienced editors, we propose different thresholds, in a range from 90 to 730 days, that are used by the community in order to determine whether an editor has left.

• 90 days (3 months).

In English, German and Chinese Wikipedias, a period of at least 3 months stands as the recommended time to wait for other editors before listing a user as missing from Wikipedia. For Italian or Romanian, this period extends to six months or one year. This definition considers that the editor could be on wikibreak. Inactive editors with administrator permissions are also reported on a page when they have not edited for at least 3 months.

• 180 days (6 months).

For stewards – a group of users with broad permissions on Wikipedia, even superior to the ones of the administrators that is gained after a community election – a period of inactivity of six months, including no “steward actions,” or less than 10 actions within a year will lead to a flag removal, i.e. a removal of the special permissions and of the steward role.19 Stewardship can be recovered through a new community vote after a year. In a community-driven survey of former administrators, it was considered that administrators had left if they used a template to signal their departure and had not edited after, or if they had not edited in the previous six months.

• 365 days (1 year).

In a study conducted by the Wikimedia Foundation, former editors were contacted via e-mail and they were asked to return contributing to Wikipedia. While for 3 months of inactivity the number of editors who returned after receiving the e-mail was 5-6%, if they had been inactive for one year or more the return rate was nil. Thus, this threshold may represent a relevant reference for permanent inactivity.

• 730 days (2 years).

According to a global policy, for any holder of advanced administrative rights (administrators, bureaucrats, oversighters, checkusers, and stewards), the maximum period of inactivity - understood as zero edits and zero administrative actions - that prompts a community review is of 2 years. This is a global policy, although in some communities local policies, the period is one year, or even six months.

Thresholds for groups and individuals.

Fixed thresholds are useful to gauge the number of editors that have left, especially in the case of users with additional permissions and duties, such as administrators. However, these thresholds do not adapt to the specific context of every community or to every contributor. Sarasua and collaborators found that the longest gap was 16 months – editors got up to that period without editing and still came back. In the same study, they showed that there is no clear relationship between the editor’s lifespan and the volume of edits. While power editors tend to show a constant contribution rate over months, average editors show instead increasing and decreasing patterns of activity.

For this reason, we suggest grouping editors according to activity levels, assuming that the behavior within homogeneous groups may follow similar patterns, and then calculating group thresholds based on the distribution of the longest past period without editing and then still returning across editors in a given group. Measuring the number of editors whose inactivity is over the group threshold would allow estimating the potential of inactivity to turn permanent for each group.

Another possibility is establishing individual thresholds based on each editor’s longest period of inactivity in the past. The advantage of this approach is that it would allow for measuring the probability of drop-off for each editor according to their previous activity, without equating their characteristics to a group. The number of months of inactivity over this threshold may indicate a more reliable measure of the possibility of a permanent withdrawal.

Activity Decrease Metrics

The previous metrics were based on the patterns of inactivity. However, the process of retiring may not be sudden, and - as described in Section 2 - the community recognizes that a user can semi-retire, i.e. lower their contribution, which does not necessarily lead to a total stop in contributing. Thus, we need some metrics to capture the cases in which an editor decreases their activity.

In order to understand when there is a drop in the number of edits, we need to define a way to compute the baseline activity level of a user, so that we can compare their number of monthly edits over time. Similar to the approach taken by Jian and MacKieMason , one can tink of the baseline as the median number of edits per month. Then, work intensity at any particular month is the number of edits made in that month divided by the baseline that considers the edits in the previous months.

This comparison - expressed as a percentage - would allow knowing when there is a sudden change in the level of activity. For example, suppose for a specific month that the expected baseline activity level for a given editor is 200 edits, and the user makes 150 edits. In this case, we can say the activity level is 75% of the baseline or a decrease of 25%. Similar metrics can be computed for different proxies of activity, such as the number of editing days per month (days in which the editor has made at least one edit).

Main Hypotheses (Causes)

As said earlier, editor drop-off may happen with different degrees of awareness and with the intent of making it partial, temporary, or permanent. While the previous metrics may allow us to understand which kind of drop-off is happening for an editor, they do not answer the question of why they are making such a decision.

Motivation studies have explored the reasons and factors that drive Wikipedians to contribute . However, one cannot assume that it is simply the absence of these factors what triggers any of the three states of drop-off.

The Wikimedia Foundation regularly administers surveys about multiple aspects of the editing experience, but only in one case in 2010 they conducted a survey of former editors asking for the reasons why they left. Furthermore, this study was mostly aimed at newcomers.

Thus, rather than asking Wikipedians about why they abandon the project, we suggest studying the different patterns of interactions that precede them leaving. Once having identified the different types of drop-off, by analyzing the interactions, it is possible to validate hypotheses on which are the patterns more often associated with each one of them. In addition, this approach can also be helpful to draw conclusions from multiple language editions and different cultural contexts.

In the following subsections, we present three main hypotheses on why editors drop-off. Based on the literature, they are: 1) abrupt interactions or conflict with other editors; 2) excess in the number of interactions; 3) a lack of interactions with editors with similar characteristics. Each hypothesis translates into specific patterns of interaction. We present them along with the prior research that supported them and the community conversations and documents in which they were mentioned.

Abrupt Interactions or Conflict with Others

As reported by different community consultations, leaving Wikipedia or losing interest in contributing is often due to negative interactions with other editors. Handling and preventing conflict has been identified as a priority by communities in strategy consultations. In particular, risk of drop-off has been associated with the following interaction patterns:

• Received reverts.

Previous research has shown that having one’s edits reverted, especially without proper communication, lowers editor retention for newcomers; this was found to affect editors’ irrespectively of their gender . While having made some reverts in the last year was found to relate to higher probability of survival, as it is an indicator of a higher level of activity and involvement , it can be hypothesized that having one’s edits reverted, and participating in edit wars may discourage experienced editors.

• Discussions.

By means of a survey, Konieczny found that conflict with other editors, an unpleasant atmosphere or a perceived lack of respect were important reasons for retirement and semi-retirement. Brandes and collaborators showed that participating in discussions increases the likelihood to retire. This seems to impact especially women editors according to previous studies that showed that women tend to engage less in discussions having a negative tone, and that they are more likely to stop contributing because of conflict with other editors.

• Controversial pages.

Similarly to discussions, editing controversial pages may increase the risk of withdrawal, whereas editing more general content has a weakening effect.

• Role and activity restrictions.

Through a qualitative study with editors, Jiang et al. developed the core contributor withdrawal theory to explain drop-off as a process from psychological to behavioral withdrawal that is triggered by a contributors’ unmet expectations. For example, a user could candidate themself for the administrator role through the “Requests for Adminship” page. The process is in general very selective and it is common that such requests are denied; in some cases the candidate may also receive harsh reviews from other user. Having one’s administrator privileges stripped - also known as having the “flag27” removed in Wikipedia parlance - or being temporarily blocked are usually negative experiences that may impact the contributions of a user.

Excess in the Number and Spread of Interactions

In a quantitative study, Wang and collaborators found that increased productivity, in terms of the number of edits on articles, also increased the likelihood of withdrawal, possibly out of burnout or a sense of having accomplished one’s “mission.”

Some editors recommend to others, and especially to those holding an admin flag, to take wikibreaks. In fact, tackling burnout has become a strategic priority, while the movement is lacking strategies to prevent it.

Nonetheless, the problem of excess in the number of interactions can also apply to the number of different projects. The same study by Wang and collaborators also found that contributing to multiple projects was associated with a lower likelihood of leaving any of them, but at the same time, with a higher overall likelihood to withdraw from Wikipedia.

No Interactions with Editors with Similar Characteristics

As a last hypothesis, we posit that editors may drop-off more easily when they cannot interact or collaborate with other editors with whom they share some characteristics, such as the time they first started editing, their origin, their communication style, or their gender. Indeed, previous research shows evidence of homophily according to gender and to all the markers of language style and emotional expression considered ; preference for interaction with editors sharing political affiliation in personal pages was also observed . In this sense, Ren and authors studied the impact and evolution of group diversity in Wikipedia and found that when editors collaborate with peers with different participation seniority they tend to contribute more, but they are also more likely to leave Wikipedia. These results hint at the importance of having groups that are both diverse and consolidated to ensure both effectiveness and sustainability.

Future Developments and Impact

The study of editor drop-off in Wikipedia is as complex as necessary, especially considering that the project relies on volunteers for almost all aspects of its management. In the following section, we describe the next steps that we envision for tackling the problem of understanding editor drop-off in Wikipedia and how this research could be applied for improving community health.

Next Steps

Besides reviewing the state of the art, in this page we laid the groundwork for researching editor drop-off in Wikipedia. Once implemented, the metrics proposed in Section 2 will provide a full characterization of the communities that will clarify the dynamics of editor drop-off. Equipped with this new picture, we will be able to explore in-depth the hypotheses about the root causes of drop-off proposed in Section 3 and additional ones. We ultimately aim at understanding the health status of a community as a whole and assessing the general risk of drop-off, in its different forms, of its members will be a way to describe it. Furthermore, by highlighting the spaces where the risk of drop-off is higher we can identify interaction patterns that can be considered harmful. We also acknowledge that a parallel qualitative research, for example by conducting surveys, could complement our approach, particularly about the cases in which external causes may trigger or fuel editor drop-off.

Likewise, it can also be valuable to understand the possible transitions between the different states of drop-off (Figure 1) and how using a template to communicate one’s status can have an effect on the decision of leaving Wikipedia permanently. Specifically, we would like to answer the following research questions:

• How widespread are wikibreaks, semi-retirement and retirement?

• How many editors are signaling their activity state using a template? Is this self-report coherent with their subsequent activity (or lack thereof)?

• Are editors who use the activity templates more likely to become active again than those who do not use the templates?

• Are editors taking wikibreaks less likely to retire? If an editor goes on a wikibreak for a longer period of time are they more likely to retire?

• Is semi-retirement sustainable or does it always evolve into retirement?

• Are editors able to return to their former activity level when they semi-retire? In fact, we argue that having a method to estimate the different states of drop-off, along with its causes, and monitoring them on a regular basis would be a prerequisite to designing an effective community health management system.

Impact

We believe that the characterization proposed in this framework can be transformative to community practices and culture. Building on the research that has been carried out on the quality of content in Wikipedia, over the years, Wikipedians have embraced a wide array of tools that allow them to monitor content quality and compare different projects. We think that, in an analogous way, editors could be incentivized to improve communications and address the causes of drop-off. In the following subsections, we suggest several applications of our study.

Supporting editors.

Some of the applications that derive from understanding editor drop-off can be directed towards palliating and mitigating its causes. For example, signaling content or topics that may require some special facilitation in discussions, preventing burnout and raising awareness on the need for wikibreaks, or designing strategies to re-engage retired Wikipedians considering the potential causes of the retirement.

Diversity and gender gap.

Systemic bias and gender gap are among the main concerns for the Wikipedia community. A recent study on gender gap in the Spanish Wikipedia reports that women editors’ life cycle tends to be shorter than for their men counterparts, suggesting that women editor drop-off may be a determinant factor for the gender gap in the community. Therefore, studying drop-off dynamics for women, as well as for underrepresented groups of users according to other socio-demographic aspects (e.g., country or native language) may be especially important for understanding how and why they leave, and to design strategies to help retain and attract them.

Community health.

Other applications may lie in the capacity to anticipate and act on mid-term and long-term concerning scenarios. In this sense, studying editor drop-off is necessary to support the communities in understanding and monitoring different aspects of role renewal and composition in terms of diversity. The study of drop-off should also help in raising awareness on the diversity among the profiles that exist in Wikipedia with respect to the tasks they perform (e.g., substantive experts, technical editors, vandal fighters, and social networkers).

Understanding drop-off and the diversity of profiles seems a necessary step towards the foundation of effective community health practices. One that prevents burnout and manages conflict, as pointed out in different initiatives proposed in the Wikimedia Strategy 2030 recommendations such as the Universal Code of Conduct.

We may be transitioning towards a culture less focused on participation (i.e. editcountitis) and more willing to suggest signaling wikibreaks when needed or deemed convenient. But as we accept drop-off, we should always differentiate one goodbye from another, as it may be more desirable to leave with a gradual and gentle process, and possibly with a feeling of accomplishment - or simply to pursue other goals outside the project or because of other external reasons.

Looking at Wikipedia from the lens of drop-off will uncover opportunities to improve community health and has the potential to inform many decisions at different community spaces, at organization levels, and at strategic venues. Having insights on editor drop-off is useful in any case, as the Wikimedia movement should aim at attracting newcomers but at the same time also at consolidating the current value in the communities that made Wikipedia successful.

Discussion

This page provides a framework for characterizing the different states of editor drop-off and reviewed the leading hypotheses for its causes as described by community documentation and previous research.

In addressing this issue, this framework contributes to state of the art in two ways. First, our review demonstrates that no single framework has been proposed to date to describe drop-off holistically. We reviewed the research on participation in Wikipedia, covering various aspects of the roles, the interactions, and the multilingual nature of Wikipedia. Second, this work considers both the scholarly research and the Wikipedia community’s experience on the matter; to the best of our knowledge, the latter was overlooked by previous research. We developed our framework taking into account the customs and peculiarities of different Wikipedia language communities to address the issue in its entirety.

We believe this can help the characterization to respond to the current and future needs of the Wikimedia movement. This framework may facilitate effective inquiries on Wikipedia participation patterns, and pave the way for future research combining quantitative and qualitative methods to investigate the dynamics and possible causes of drop-off.