Research:Open Collaboration Systems Workshop/Theory and systems

This page documents an outline for a 10 minute talk that Halfak will give at Open Collaboration Systems Workshop @ CSCW'15.

Introduction & call for theory

Why do we need Theory?

Wikipedia works in practice, not in theory. Which is great until you want to create another Wikipedia -- or Wikipedia stops working.

I'm guessing that a lot of people in this room would love to have simple and robust answers to the following questions:

• What makes an OCS work and not work?
• How do OCSs mature with time and how do the needs of an OCS change at scale?
• How can we best support OCSs with technology?

I work for the Wikimedia Foundation. Right now, some of our primary concerns are related to Wikipedia's community health. But what the heck is "community health"? Right now, I don't think that we have a very good way of thinking about it. We tend use crude metrics like the number of "active users" and some intuitive canaries like the survival rate of good-faith newcomers -- both of which have been tanking. So, is Wikipedia unhealthy. Could the trends we see be benign -- or maybe even desirable. Either way, they may be inevitable. We need to know in order to do our jobs.

System theory, the Paramecium and Wikipedia

Systems theory can be helpful here. By thinking about Wikipedia as a living system within an ecology, I think that we can start thinking about our questions better.

I don't have enough time to give an introduction to systems theory and a reflection on recent work. Instead, I'll make a comparison that systems theory affords in an attempt to get us thinking about how to apply such strategies to the systems we study and maintain.

Comparison

• Concepts: Permeable membranes, ecological competition and subsystems.

To give you a sense for what I mean, let's examine a living system that may be more intuitive. See "Bacterium vs. Paramecium"

A biological example

Single cellular organisms have permeable membranes. By this I mean that water and many other chemicals can flow in an out of the cell relatively freely. This is critical for a lot of reasons, but can also cause problems. One problem is osmosis in freshwater environments. When the PPM of salt inside of a cell is greater than outside, water rushes in. If this continues without remedy, a cell will literally burst.

Enter the "Contractile vacuole" which is quite simply a water pump. This specialized sub-component allows paramecia to thrive in freshwater environments -- occupying an ecological niche that would have otherwise been impossible.

Knowledge of contractile vacuoles can be powerful. Assuming that we were looking at a particular paramecium with as much scrutiny as we tend to bring to the systems that we study, we might notice that ours is having issues maintain the right amount of water pressure. Because we know something about contractile vacuoles, we're likely to add that to our list of potential causes.

A socio-technical example

Like other OCS, Wikipedia also has a permeable membrane. This is critical for a lot of reasons, but it can also be a problem. One problem is bad-faith users who much cause vandalism and damage. We might imagine that, if the amount of damage got too high, Wikipedia would cease to be relevant and the system would die.

Enter the socio-technical quality control subsystem in Wikipedia. By combining a specialized set of editor roles (CVU, NPP, TeaHouse Hosts, etc.) and intelligent software tools (ClueBot NG, Huggle, Snuggle, HostBot, etc.), English Wikipedia editors are spending less and less time filtering damage from the recent changes stream.

This subsystem allows Wikipedia to keep is permeable membrane and also occupy an interesting ecological niche among the attention of internet users. I'd argue that this position allowed Wikipedia to become so dominant.

But things can go wrong. For example, this counter-vandalism subsystem relies on technology. Predictive algorithms make mistakes and user-interfaces can strongly direct human behavior through defaults and limited affordances.

If we were to examine Wikipedia and find that a lot of good newcomers were getting reverted for "vandalism" -- and that this was causing a decline in the rate which they stick around, we might consider what was going wrong with Wikipedia's socio-technical quality control system.

System thinking in Wikipedia

In this example we see how conceptualizing a paramecium as a system composed of specialized sub-systems helps us think critically about the organism's health. Just as the paramecium has many sub-systems that contribute to its overall health, Wikipedia seems to have many as well. Personally, I like to think of Wikipedia's subsystems as fitting into 5 major categories.

• Work allocation -- Directs people towards work
  • Healthy: The people with the right skills and interest write the most important articles first. Editors fill specialized roles at near-optimal rates.
  • Unhealthy: The people with the wrong skills or interest are often directed towards work on less important content. Many roles are understaffed -- others overstaffed.
• Regulation of behavior -- Ensure consistent application of rules and process
  • Health: Application of rules is consistent. Power is distributed widely. When rules don't make sense under local conditions, they are ignored.
  • Unhealthy: Rules are often unknown by editors. When they are known, they are applied inconsistently and or under conditions where they do not make sense.
• Quality control -- Filters new content & contributors for bad-faith
  • Health: Most vandalism is removed very quickly. Newcomers who make mistakes (not bad-faith) are welcomed and trained. Bad-faith actors are quickly identified and blocked.
  • Unhealthy: Much vandalism persists for a long time. Newcomers who make mistakes (not bad-faith) are often reverted, warned and banned. Bad-faith actors persist and continue causing harm.
• Community management -- Maintains a healthy, active, & motivated population of volunteers
  • Healthy: Newcomers take advantage of training resources. Social spaces exist to help work out technical and interpersonal troubles without sanctions. Long-term editors have many options for reducing wiki-stress without leaving.
  • Unhealthy: Newcomers are adrift in complex unknowns and they fail because of it. The primary recourse for disagreements is sanctioning. Long-term editors regularly leave due to stress.
• Reflection / adaptation -- Discovers problems/opportunities and proposes changes in process/technology
  • Healthy: Analysis, discussion and experimentation are common. Experiments lead to learning if not actual improvements. Improvements are adopted broadly.
  • Unhealthy: Analysis and experimentation rarely happen. Many discussions about problems/opportunities lead nowhere. What improvements are demonstrated are not adopted.

I suspect that these subsystems are probably relatively common across other OCS as well. If I knew how all of these systems worked, I could tell you how to build the next open knowledge project so that it works and is sustainable. I think that this work is a critical part of Reflection / adaptation and it is my goal that you'll leave here agreeing with me.