Research talk:The motivational arc of massive virtual collaboration

Initial notes

Before I get the rest of RCom involved, this proposal will need a few more bits of information:

• How many subjects would you need completed surveys from?
  • How many Wikipedians will you suspect you'll need to contact?
• What is your preferred contact mechanism and why?
  • We (RCom) usually recommend a structured post to User_talk pages.
• I have been contacted by Gabriel, yet he is not listed as a contact for the project. Why is that?
• How will this research be different than other surveys examining the motivation of Wikipedians?
  • A good starting place for the literature is documented on the Wikipapers keyword Motivation.

Further, we (RCom) usually request a local IRB approval before giving our own. If it would be helpful for your IRB, we could produce a conditional approval along the way assuring that we have reviewed the study and are ready to sign off given the IRB's OK. ...if, of course, we get to that point. --EpochFail (talk) 13:01, 29 March 2012 (UTC)

Response to initial notes

• We would need 180 total, so 60 from each of the 3 categories (initial, sustained, meta) described in the paper by Crowston and Fagnot (2008)
  • Assuming a 5-10% response rate, we will need to contact 1,200 for each category for a total of 3,600.
• Our preferred contact method would be leaving a message on a user talk page
• Gabriel should now be listed as a contact on the proposal (gmugar@syr.edu)
• The research is different from other research on motivation in that it proposes motivations to contribute are unique across the arc of contributor types listed in Crowston and Fagnot (2008). In other words, the motivations to contribute for an initial contributor are different than the motivations that drive the contributions of a sustained contributor.
• A conditional approval would be helpful for our local IRB. Gabrielm199 (talk) 16:03, 1 April 2012 (UTC)

Thanks for the updates. A recent study that used a similar survey instrument acquired a response rate of 53% (which is kind of amazing actually). Is there a reason you expect to have such a low response rate? If not, I'd recommend assuming ~50% response and messaging 360 users. You could then message more if you didn't get enough responses. Would that be acceptable? Either way, I'd recommend performing a pilot study over a small set of requests. That should give us insight into the response rates of the three groups you hope to compare.

Also, I think it will be easier for us to sign off on your study if you demonstrated the merit of your (yet another) motivation related survey of Wikipedians. I imagine an abbreviated lit. review of the related motivation surveys the form below would be sufficient:

* <authors> in <study1> looked at <analysis> and found <conclusions>

* <authors> in <study2> looked at <analysis> and found <conclusions>

I assume you're familiar enough with the literature that this would be trivial. --EpochFail (talk) 21:11, 3 April 2012 (UTC)

• I'd echo Epochfail's suggestion about response rates and running this in tranches. If you are only targetting currently active editors then response rates can be very high. One interesting test might be to do a tranche of editors with high CentiJimbo counts. My expectation is that the response rate to userpage requests on those of us with the highest centiJimbo counts will exceed 100%. It would be interesting to put a confirmation question in the survey to see what proportion of the survey responses are from the targeted editors. Another thing you might consider would be to run all or part of your survey on a less surveyed part of the project like commons, the English language wikipedia gets rather heavily surveyed, Simple Meta, and Commons I believe are much quieter and therefore should be less subject to survey fatigue. WereSpielChequers (talk) 11:15, 4 April 2012 (UTC)

• Yes, now that you mention it, surveying incrementally is a great way to proceed. Thanks for pointing this out!
• We certainly do plan to do a pilot to check the response rates, get a sense of the effect sizes and to get feedback on the questions.
• Regarding the uniqueness of this motivation study, most of the motivations we want to study have in fact been studied before. Our main innovation is to identify users at different stages of contribution and to examine motivations for these groups separately. It's not possible to do that cleanly retrospectively from past data, not that we have access to the past data. We do have a literature review to answer the specific question, but it may have to be updated…
• Regarding the note about confirmation questions, we have included them in the survey. Gabrielm199 (talk) 23:11, 16 April 2012 (UTC)

Would you add the above details to the project description? --EpochFail (talk) 14:50, 15 May 2012 (UTC)

I'd also like a copy of your consent form (first page of survey) on the wiki for reviewers/other readers to see. Would that be alright? I'd recommend making a page for it beneath the project and linking to it. For example:

1. Edit Research:The_motivational_arc_of_massive_virtual_collaboration/Consent_form (verbatim copy-paste would be fine)
2. Link to [[Research:The_motivational_arc_of_massive_virtual_collaboration/Consent_form|consent form]] from your project page.

I'd just do this for you, but copyright and all that. --EpochFail (talk) 14:58, 15 May 2012 (UTC)

Question Set

Hi, please can we have a preview of the questions that you would like to ask. WereSpielChequers (talk) 11:18, 4 April 2012 (UTC)

• Survey software is causing me some trouble at the moment. Will have it to you as soon as tech issues are resolved. Gabrielm199 (talk) 23:12, 16 April 2012 (UTC)
  • problem resolved here is the link to the survey Please note that this is still in pre-pilot phase. If you or anyone else is interested, I would greatly appreciate getting some live feedback over Skype from someone taking the survey. Gabrielm199 (talk) 14:13, 18 April 2012 (UTC)

"Placing relevant article improvement wiki project tags in articles" needs amendment as the wikiproject tags go on talkpages. I suggest "adding maintenance templates to problematic articles, or wikiproject tags to article talkpages". But more broadly the whole section could be collapsed. What I think you want to convey is that you want anyone who has been active in the project in the last month, even those who haven't actually edited Wikipedia but have been active in research, organisational aspects such as chapters or support activities such as IRC and OTRS.

Your notification doesn't say whether you are holding any data that would identify the Wikipedian, and if so for how long you will hold that data.

I suggest you add retired, student and homemaker to your employment options. You might choose to fold the results together, but I think you'll find that unemployed is a pejorative term that pensioners won't tick.

When I reached the end I had to close the screen as it wouldn't let me back to this page. It is important that when someone completes a research survey they are returned to the wikipedia page.

WereSpielChequers (talk) 08:34, 19 April 2012 (UTC)

Thank you for this great feedback. I am in the process of collecting more feedback from Wikipedians and will implement their suggestions as well. It would be great to hear from others on Rcom regarding the current draft of the survey. Gabrielm199 (talk) 16:15, 20 April 2012 (UTC)

• More points:

1. Indentifying with the goals and ideals of Wikipedia should be Identifying with the goals and ideals of Wikipedia
2. The section "Visibility of Wikipedia's needs for contribution" leaves me as a bulk standard Wikipedian somewhat confused. "These question refer your perception of how well Wikipedia makes its need for contribution known to you. Contribution is defined as anything ranging from minor edits to participating in policy related discussions." and "Wikipedia’s description of needs for contribution are clearly stated." and "Wikipedia’s needs are self-explanatory." and "Wikipedia’s needs are difficult to understand." and "There are understandable instruction on how I can contribute to Wikipedia." and "The explanations on the needs of Wikipedia are not comprehensible." and "Wikipedia’s needs are easily comprehensible". They could be sightly clarified as "These questions refer to your perception ofas to how well Wikipedia makes its need for contribution known to you. Contribution is defined as anything ranging from minor edits to participating in policy related discussions." and "Wikipedia’s description of needs for contribution are clearly stated." and "Wikipedia’s needs are self-explanatory." and "Wikipedia’s needs are difficult to understand." and "There are understandable instruction onas to how I can contribute to Wikipedia." and "The explanations of the needs of Wikipedia are not comprehensible." and "Wikipedia’s needs are easily comprehensible". But more broadly, is this asking about how we communicate to editors what needs to be done on the site, i.e. the backlogs. Or is it about how we communicate what people need to do when they are editing i.e. rules, guidelines and the Manual Of Style?
3. Another area is repetition. I can appreciate that to identify confused and respondees and those who answer each question randomly you want to repeat questions or ask the reverse condition. But if you do it in the same screen, especially in near adjacent questions then those doing random responses can mask that by giving logically consistent answers to such question pairs, and good faith respondees like myself are left rereading the questions to try and work out the logical difference between them. WereSpielChequers (talk) 16:18, 22 April 2012 (UTC)

Thank you WereSpielChequers for the valuable feedback. I am in the process of gathering feedback from other experienced Wikipedians and will have a revised draft of the survey next week. I will announce the availability of the revised version here. Gabrielm199 (talk) 13:10, 27 April 2012 (UTC)

• Latest draft of the survey https://survey.ischool.syr.edu/TakeSurvey.aspx?SurveyID=92M39p7 Gabrielm199 (talk) 03:07, 15 May 2012 (UTC)

Still unclear on the need for another motivation study

Gabriel, I'm still unclear on how your survey will expand upon the current understanding of Wikipedian motivation. I had previously asked for an abridged lit. review to describe what has been done and where this research fits in. I have not seen your response to this request. Also, WereSpeilChequers asked if you would be able to survey non-English Wikipedia communities to answer your research questions, but you did not respond. You may not be aware of this, but English Wikipedians are a heavily surveyed community. One of the burdens that we (RCom et al.) bear in this review process is the responsibility to ensure that additional surveys (distractions from editing activities) are not pushed upon the community unnecessarily. It is important that you both convince us, your reviewers, and provide documentation that would convince future readers that this survey will answer questions that cannot just as easily be answered using previous work or by surveying a less fatigued community. --EpochFail (talk) 15:14, 17 May 2012 (UTC)

I would like to second this concern. In addition to the work already linked to, I can find large number of ACM papers [1] which are substantially about motivations of Wikipedia editors. There is also data from the annual survey of editors conducted by the Wikimedia Foundation, which is quite broad and covers a lot of motivation-related material. Steven Walling (WMF) • talk 23:07, 25 May 2012 (UTC)

I am working on an annotated bibliography to answer your concerns about another motivation study. While this is about motivation, it addresses motivation not as a singular phenomenon, but from the perspective of different types of motivation being unique to different types of contribution to Wikipedia. Pertaining to WereSpeilChequers question about conducting the survey on other Wiki projects, we would like to conduct our study on the English Wikipedia because it has the largest number of contributors that the article upon which this survey is based classifies as meta contributors. Meta contributors are those that contribute to the social infrastructure of Wikipedia rather than contributing to the maintenance and development of articles. The hypothesis of the article and what we are hoping to test with this survey is that motivations for meta contributors are unique and distinct from the motivations of contributors who work on articles. Gabrielm199 (talk) 19:35, 1 June 2012 (UTC)

Is there any reason why you could not simply use the data dump from the 2011 editor survey? Steven Walling (WMF) • talk 17:30, 13 June 2012 (UTC)

Hi Steven, I am not familiar with the 2011 Editor survey. Do you have a list of the questions asked? Here is an overview of six articles I selected to demonstrate how the proposed hypothesis of this survey is unique. While I am not annotating the entire literature on motivation studies, the six I selected should represent the current trend in motivation research. As you will see, motivation is not treated in the way the hypothesis for this survey treats it. Where other literature looks at motivation from a very specific or broad over arching perspectives, this survey seeks to understand motivation from a contributor typology and temporal perspective.

Author and Date	Title	Research Question	Finding
Byrant, Forte, Bruckman(2005)	Becoming Wikipedia: Transformation of Participation in Collaborative Online Encyclopedia	"how users’ motivations and their perceptions of their roles in Wikipedia change as they become more engaged in the community."	Perceptions of Wikipedia change as they move from peripheral members to more active members. View Wikipedia not as a collection of articles but as a community. Motivated by a concern for overall quality of WP rather than just one article's quality. Increase in complexity of motivations.
Fershtman and Gandal (2007)	Open Source Software: Motivation and Restrictive Licensing	Motivation is connected to visibility level of a contributions. Less restrictive licensing=more exposure, Does license type influencing motivaiton to contribute?	Projects with less restrictive licenses have higher output per contributor. Projects with more restrictive licenses see contributions per contributor reaching a threshold equivelent to maximum visibility offered by project (e.g. a contributor list ranking).
Forte and Bruckman (2005)	Why Do People Write for Wikipedia? Incentives to Contribute to Open-Content Publishing.	Is the motivaitonal structure of Wikipedia similar to that of the scientific community in that it is motivated by reputation? How does the design of the community impact the motivational structure as it relates to reputation?	Authors find many similarities between motivaitons of scientific publishing commmunity and Wikipedia. They make recommendations regarding the design of online communities as they pertain to sustaining involvement. Such recommendations include rewarding regular contributors and avoiding stratification of participants.
Freeman (2007)	The Material and Social Dynamics of Motivation: Contributions to Open Source Language Technology Development.	Motivation in FLOSS should not be viewed as a singular abstract phenomenon nor from the intrinsic and extrinsic dichotomy, but as one that evolves over the career of a contributors involvement with the project and as unique to the activity.	Rather than categories of motivations, author describes "patterns of motivations" tied to different tasks, objectives, and personal histories.
Johnson (2010)	Should I Stay or Should I Go? Continued Participation Intentions in Online Communities.	What are the individual and community level antecedants for continued participation in online communities?Authors look at the role of interaction with leadership, community size, and psychological safety as antecedants for continued participation.	Results show that interaction with leadership and community size are positively associated with continued participation. Psychological safety was also found to be positively associated with continued participation. The authors thus conclude that continued support is multi-dimensional.
Kuznetsov (2006)	Motivations of Contributors to Wikipedia.	what are the motivations for contribution, what are the values behind the motivations, and how does the technology support and facilitate such values.	Of the motivations identified, the values of altruism, reciprocity, community, reputation, and autonomy where defined. Of these values, community, autonomy, and reputation where identified to be deeply embedded and fostered by the Wiki technology.

Gabrielm199 (talk) 15:56, 30 June 2012 (UTC)

In addition to the description in the full data dump, there is a great deal of detail about the Editor Survey 2011 here on Meta. Anyway, I understand your theory on motivation is different, but I remain unconvinced it would be worth the annoyance and further survey fatigue the population of English Wikipedians already are suffering from. There are certainly ways you can take the raw answer data and use it to support a particular framework for studying motivations. Steven Walling (WMF) • talk 18:45, 2 July 2012 (UTC)

Thank you for the link, Steven. I will take a look at the survey to see if it has relevant data for the model proposed in the paper. Gabrielm199 (talk) 13:43, 12 July 2012 (UTC)