IP Editing: Privacy Enhancement and Abuse Mitigation/IP Editing Restriction Study/Portuguese Wikipedia

Slide deck for the metrics shared below

Background

In fall 2020, the Portuguese Wikipedia community decided to turn off IP editing by a community vote. The community implemented the change on October 4th 2020. The Wikimedia Foundation's Product Analytics team built a dashboard to track the impact on Portuguese Wikipedia on a weekly basis, to learn more about the impact of IP editing on a Wikipedia project. The quantitative measurement mainly focuses on the impacts on editors, edits, and administrator actions. The metrics include:

  • Number of new accounts
  • Number of active user editors
  • Retention rate
  • Number of edits
  • Number of reverts
  • Number of net non-reverted edits
  • Number of net non-reverted content edits
  • Number of blocks
  • Number of protected pages
  • Checkuser requests

In this report, we will go through the trends of above metrics from October 2020 to June 2021, i.e., Q2, Q3, and Q4 of the Foundation's 2020/2021 fiscal year. We will also share the qualitative data of the community feedback about this change.

Takeaways

We evaluated the effects of turning off IP editing on the health of Portuguese Wikipedia and we found no significant negative impact in the analysis conducted thus far. Key takeaways are captured below:

  • The number of user account registrations as well as active editors on Portuguese Wikipedia have gone up since the IP editing block was introduced. We do not know if it is a long-term trend.
  • The user retention metric has not seen a significant increase or decrease. We do not know the long-term impact on it.
  • The number of reverts, page protections and blocks have declined considerably, indicating a decrease in the amount of vandalism on the project.
  • The total number of edits made to the project show a decline when bot edits and reverts are included. Without bot edits and reverted edits, the number of edits was down in Q2 and Q3 compared to last year, and was up in Q4. Impacts on edits are unclear.

We also sought community feedback on Portuguese Wikipedia regarding their experience and feedback over the IP editing ban. We found that a majority of the ptwiki editors remain in favor of the ban and report seeing less conflict and hostility on the platform.

Recommendations

Based on what we learned, the Wikimedia Foundation's Product department made the following recommendations.

  • The WMF shouldn't try to interfere with Portuguese Wikipedia's decision to continue with their current practice, assuming the volunteers want to continue.
  • Validate these findings by running a similar experiment on 2 more Wikipedia projects (languages) where the volunteers are interested in trying this out. The experience of a single wiki should not be used as decisive evidence.
  • Publish these findings, to help inform continued discussions across the movement about the role of IP editing.

Impacts on editors

Number of new accounts

 
New Accounts

After IP editing was turned off on October 4th, 2020, newly created accounts increased by 20% YoY in Q2, from 12487 monthly average to 14960. In Q3 the growth continued at a 26% YoY rate, from 12873 monthly average to 16234. In Q4, the average number of monthly new accounts is 16059, same as last year. It indicates some anonymous editors registered accounts after turning off IP editing. It’s hard to connect an IP address to an anonymous editor, therefore we cannot tell the percentage of anonymous editors who became registered editors and the percentage of anonymous editors who left Portuguese Wikipedia. But we confirmed growth of new accounts in the following six months.

Number of active user editors

 
Active User Editors

The number of active registered editors has increased from October 2020 to March 2021.(FY20/21 Q2 & Q3 ). We observed a 76% YoY increase in Q2 monthly average , from 5362 to 9438, and a 98% YoY increase in Q3 monthly average, from 5270 to 10409. In Q4, the number of active editors increased by 66%, from 5926 to 9836.

Retention rate

 
Retention Rate

Retention rate is defined as out of the non-bot users who registered in the month before the previous and made at least one edit in their first 30 days, the proportion who also edited during their second 30 days. Due to the nature of the definition, this metric surfaces 2 months after user account creation. In Q2, average retention rate increased from 5.6% in FY19/20 to 6.7% in FY20/21. In Q3, it increased from 3.71% to 5.46%. We believe some features launched by growth team would increase the retention rate, which is estimated to be 0.3%. Therefore, the impact of turning off IP editing is estimated to be an increase of 0.8% in Q2 and an increase of 1.4% in Q3.

Takeaways

When Portuguese Wikipedia turned off IP editing, some anonymous editors were willing to register new accounts and continue actively editing on Portuguese Wikipedia, resulting in a 26% growth of new accounts. The 98% growth of active editors is partially from these newly created accounts. Secondly, some editors who had accounts not used that often now began to login and edit. This group of editors also contributed to the growth of active editors.  

Impacts on content

Number of edits

 
Total Edits

Total edits decreased since October, 2020. Monthly average edits decreased by 9% YoY in Q2 , from 238069 to 216890. In Q3, it decreased by 22% YoY in Q3, from 282520 to 219327. In Q4, it decreased by -0.48%, from 239037 to 237883.

Edits can be created by bots, registered users, and anonymous users. After looking into the edits by each user category, we confirmed that  in Q2 the decrease is not from edits by bot and registered users.  The edits we lose in Q2 are edits by anonymous editors. In Q3 the decrease is from both anonymous editors and bot edits.

Number of reverts

 
Reverts Within 48 hours

Number of reverts is defined as the number of reverted edits within 48 hours of editing. It dropped by 46.8% YoY in Q2, from 24454 to 13006. In Q3, it dropped by 43.9%, from 23421 to 13144. In Q4, it dropped by 47.2%, from 25434 to 13429. It indicated that turning off IP editing prevented vandalism on a certain level.

Net non-reverted edits

 
Net Edits Excluding Bot Edits and Reverts

As we discussed, bot edits and reverts are changing in different directions. Therefore we defined a metric to exclude the changes from bot edits and reverts and to only measure ‘meaningful’ edits by human users. Net non-reverted edits is defined as the number of edits which were not reverted within 48 hrs of editing, and not a revert edit, and not an edit by bots. Monthly average decreased by 0.9% in Q2, from 180066 to 178419, and increased by 2% in Q3, from 179613 to 183289. In Q4, it increased by 6%, from 185227 to 196557.

We estimate that features from the growth team increase edits by 0.3%. Therefore the impact of turning off IP editing on net edits is estimated to be a decrease of 1.2% in Q2, an increase of 1.7% in Q3, and an increase of 5.7% in Q4.

 
Monthly Edits Forecast

In Q2, the monthly average of net non-reverted edits across all wikipedias projects increased by 13.5% YoY. It makes us wonder what the trend would be on Portuguese Wikipedia if the intervention does not happen. We built a forecast model to estimate the edits on Portuguese Wikipedia if IP editing was not turned off, and compared with the actual edits. We do not have statistical evidence that turning off IP editing has negatively impacted editing activeness level. Given that the trend is not clear. We will keep monitoring this metric.

Net non-reverted content edits

Net non-reverted content edits decreased by 2.5% in Q2, from 132066 monthly average to 128690, and also decreased by 7.2% in Q3, from 133310 monthly average to 123779. It increased by 0.9% YoY, from 131338 to 132459.

 
Net Content Edits Excluding Bot Edits and Reverts

We estimate that features from growth team increase content edits by 0.5%. Therefore, the impact of turning off IP editing on net content edits is estimated to be -3% in Q2, -7.7% in Q3, +0.4% in Q4. The trend is not clear considering the fluctuation in history. We will keep monitoring this metric.

Takeaways

The reverts decreased after turning off IP editing. It indicated that vandalism was suppressed at some level as reverting is often used to fight vandalism. The impact on edits is not clear given the fluctuations in last year and this year. The number of edits on the non-content page has grown compared to last year. The growth on non-content pages is mainly from talk pages, pages of policies, guidelines, pages of users.

Impacts on administration

 
Blocks

Number of blocks

Number of blocks has decreased significantly since October, 2020. The YoY change of block monthly average is -86% in Q2, from 9278 to 1274. In Q3, it decreased by 79%, from 4361 to 931. In Q4, it decreased by 76%, from 4207 to 992.

Protected pages

 
Protected Pages

Number of protected pages decreased by 78% in Q2 (from 825 to 185), decreased by 68% in Q3 (from 468 to 148), and decreased by 40% in Q4 (from 384 and 232).

Checkuser checks

 
Checkuser Requests

Requests by check users did not show a clear trend.

Takeaways

Most administration actions are taken to handle vandalism and anti-harassment.  We saw a significant reduction in administration actions. It seems turning off IP editing directly reduces administration workload.

Community Feedback

A survey was conducted among a group of Portuguese Wikipedians. 19 editors responded, 14 supported the ban, 4 didn't support the ban, and 1 person had a neutral view. Here are some selected feedback quotes.

Opinions of some editors in support of the ban 

[pt]
Depois de anos votando contra, no final do ano passado finalmente me rendi ao óbvio, o trabalho que o vandalismo dá não compensa alguma pouca melhoria, sem eles nossa vida está muito melhor, muitos artigos que viviam bloqueados puderam ser liberados, hoje podemos deixar uma mensagem para um novato que a chance de ser lida é maior que para um IP (aqui no Brasil IPs são atribuídos aleatoriamente cada vez que se conecta, a chance de receber o mesmo IP e portanto ler um aviso é quase zero). Os editores tem mais tempo para trabalhar nos artigos, incluindo fontes. E não é verdade que somos a primeira wiki da WMF a proibir IPs, o Meta, o Commons e o Phabricator nunca permitiram. Claro que ainda existem LTAs e um ou outro chato, mas o grosso do vandalismo sumiu. Foi a mudança mais positiva em meus tantos anos aqui.

 — Editor 1, IP Editing Ban Sentiment Survey


After years of voting against it, at the end of last year I finally surrendered to the obvious, the work that vandalism does does not compensate for any little improvement, without them our lives are much better, many articles that used to be protected could be freed up, today we can leave a message for a newbie which has a greater chance of being read than for an IP (here in Brazil IPs are assigned randomly each time you connect, the chance of receiving the same IP and therefore reading a warning is almost zero). Editors have more time to work on articles, including sources. And it is not true that we are the first WMF wiki to ban IPs, Meta, Commons and Phabricator never allowed it. Of course, there are still LTAs and this or that minor annoyance but the bulk of the vandalism is gone. It was the most positive change in my many years here.

 — Editor 1, IP Editing Ban Sentiment Survey

[pt]
Certamente não foi uma decisão que tomamos com prazer, mas minha lista de páginas vigiadas nunca andou tão pacífica, e olha que ultimamente eu nem tenho conseguido olhá-la o tempo todo como uns meses atrás. Os IPs agora fazem certa farra nas páginas de contato, mas é um efeito colateral do qual não vejo escapatória.

 — Editor 2, IP Editing Ban Sentiment Survey


It certainly wasn't a decision we were happy to make, but my watchlist has never been so quiet, and lately I haven't even been able to look at it all the time like a few months ago. IPs horse around a bit on the contact pages, but it's a side effect that I see no escape from.

 — Editor 2, IP Editing Ban Sentiment Survey

[en]
My impression about the ip block, or I rather call it "obligation for account registration". It has been very positive for the community. The sheer drop in blocks and vandalism is very impressive, dropped by around 80%. Blocking and reverting vandalism by registered accounts is much easier, we have a lot more context for their edits. We have even started unprotecting several long-term protected articles with little vandalism. Resulting in more articles that newcomers can edit. Another very positive impact is the interaction, registered users see the messages and respond at a much higher rate than unregistered users. This means less friction and more understanding from newcomers and patrollers.

 — Editor 3, IP Editing Ban Sentiment Survey

[pt]
A edição se tornou muitíssimo mais agradável, desapareceu muita da pressão causada pelo vandalismo. O nível de hostilidade e conflito dentro da comunidade baixou também, e agora tenho mais tempo para ajudar os novatos, e fazer aquilo que realmente gosto no projecto (editar, manutenção focada, etc).

 — Editor 4, IP Editing Ban Sentiment Survey


The editing became much more pleasant, much of the pressure caused by vandalism disappeared. The level of hostility and conflict within the community has also dropped, and now I have more time to help newbies, and do what I really like about the project (editing, focused maintenance, etc.).

 — Editor 4, IP Editing Ban Sentiment Survey

Opinions of some editors who opposed the ban  

[pt]
Não concordo com o banimento de IPs. Na minha experiência como editora, observo que grande parte do vandalismo na wiki acontece a olhos vistos, mts vezes por parte de usuários altos na hierarquia, adm especialmente. O bullying de usuários, o viés de gênero, a intolerância cultural e política devem ser considerados vandalismo na wiki porque tem repercussão nas páginas como constantes reversões, sugestão de eliminação etc. Além disso, por outro lado, a utilização do IP pode servir como ferramenta para alguém que queira contribuir, mas, por conta dessas perseguições, prefira preservar sua privacidade. Por fim, não vejo de maneira positiva uma mudança que afasta tanto a wiki pt do modelo das outras wiki ao redor do mundo.

 — Editor 5, IP Editing Ban Sentiment Survey


I do not agree with the ban on IPs. In my experience as a publisher, I notice that much of the vandalism on the wiki happens in plain sight, often by high users in the hierarchy, especially admin. User bullying, gender bias, cultural and political intolerance should be considered vandalism on the wiki because it has repercussions on the pages as constant reversals, suggested elimination, etc. In addition, on the other hand, the use of IP can serve as a tool for someone who wants to contribute, but, because of these persecutions, prefer to preserve their privacy. Finally, I do not see in a positive way a change that distances the wiki pt from the model of other wiki around the world.

 — Editor 5, IP Editing Ban Sentiment Survey

[pt]
Acho um erro. Um retrocesso. Sabemos que a Wikipédia cresceu justamente por ser totalmente aberta. Existem de sobra usuários com poderes e tempo para controlar vandalismos na Wikipédia em português. Além de outras ferramentas. Fechar assim a edição por IP é mais perder do que ganhar.

 — Editor 6, IP Editing Ban Sentiment Survey


I think it's a mistake. A setback. We know that Wikipedia has grown precisely because it is completely open. There are plenty of users with the power and time to control vandalism on Wikipedia in Portuguese. In addition to other tools. Thus closing the edition by IP is more to lose than to win.

 — Editor 6, IP Editing Ban Sentiment Survey

Opinion of an editor who held a neutral view  

[pt]
Foi importante a restrição de IPs no domínio principal, pois reduziu-se o número de vandalismo nestas páginas e no controle das mudanças recentes e na lista de minhas vigiadas. Por outro lado, não podemos esquecer que também caiu a possibilidade de anônimos desenvolverem conteúdo nos verbetes, o que não impediu que o número de contas criadas tenha aumentado e muitos usuários entenderam que vale mais a pena editar como registrado.

 — Editor 7, IP Editing Ban Sentiment Survey


The restriction of IPs in the main domain was important, as the number of vandalism on these pages was reduced and in the control of recent changes and in the list of my watchmen. On the other hand, we must not forget that the possibility of anonymous content developing in the entries has also fallen, which has not prevented the number of accounts created from increasing, and many users have understood that it is more worth editing as registered.

 — Editor 7, IP Editing Ban Sentiment Survey

Some Frequently Asked Questions

Some frequently asked questions about this study are answered and available here.

Other research

Above quantitative and qualitative data only reflect the impact of turning off IP editing on Portuguese Wikipedia. The experience of a single wiki should not be used as decisive evidence to predict what would happen if the same change was enacted on other projects. We have more documented studies on the value of IP editing by other researchers.

References

  1. Wikipedia:Votes/Registration required to edit Portuguese-speaking Wikipedia
  2. ptwiki Dashboard
  3. Research:Value of IP Editing