Research talk:Effects of Feedback on Participation in Wikipedia

Methodology

• How many total messages will you be posting?
• How will you randomize the postings?

--EpochFail 18:24, 31 July 2011 (UTC)

+1 to the above questions, as well as a timeline of when exactly you want to start and end. Additionally, I'd encourage you to state explicitly (and link to) all accounts that will be posting messages, for the sake of transparency. Doing that, as well as making an announcement about the study on the Village Pump or another appropriate place prior to performing it will increase community trust in the experiment (RCom and myself can help you do that, if you feel it would lend weight to the notice). Steven Walling (WMF) • talk 21:43, 31 July 2011 (UTC)

I have added some details about total messages and randomization in the description. In general, we are going to post 850 messages. We will target Wikipedia editors who have recently created a new Wikipedia article. After analyzing the content of the article for elements qualified for feedback (e.g., noteworthiness of the topic, amount of content, number of references, neutrality of viewpoint, quality of writing), we will randomly assign the creator of the page to receive a message from one of sixteen conditions or to receive no message (control condition).Haiyizhu 05:09, 2 August 2011 (UTC)

Steven, thanks for pointing us to Village Pump. Which place in Village Pump is appropriate for us to post a study like this, proposals or idea labs? Thanks! Haiyizhu 05:09, 2 August 2011 (UTC)

Misc is probably best for this sort of thing, though proposals might also work. This sounds like an interesting study, by the way, I'll be interested to see the results. Staeiou 18:28, 3 August 2011 (UTC)

Messages that will be posted

The Constructive criticism with social components: message reads a little awkwardly IMO. Are these the final versions of the messages or are you still working on them? --EpochFail 18:24, 31 July 2011 (UTC)

With my Wikipedian hat on, I have some small issues with the accuracy of the templates. Such as that it's not called "New Article Patrol" (rather, New Page Patrol) and that it's an informal group which doesn't have a monopoly on dealing with new pages. We don't want to give new Wikipedians the impression that you have to join NPP or some other bureaucracy to deal with new pages. Another small example is that the guideline on inline citations is a content guideline, not a style guide. There is a distinct cultural difference in Wikipedia, even if it seems subtle. If your research group is comfortable, I'm happy to create alternate versions of templates or just dive in and edit here (where you'll be able to track changes). Best regards, Steven Walling (WMF) • talk 21:43, 31 July 2011 (UTC)

Thanks for the feedback. I've made some changes to the structuring and punctuation of the messages, and so hopefully they read less awkwardly and less template-like now. I've also made changes to the errors that you pointed out, though let me know if they're still inaccurate. Actually it would be great if you'd like to create a few alternate versions of templates, especially for base message, since I think we've had the most challenge in creating those and making them sound comfortable enough for both experienced and new users. Jipinghe 21:42, 1 August 2011 (UTC)

Rcom reaction

• I am not sure how we proceed with the proposal (I mean, do we vote? How do we determine that the proposal gets the green light?), just wanted to point out that I am satisfied with the modified version and, as far as I am concerned, there are no obstacles in starting the research.--Ymblanter 11:12, 6 August 2011 (UTC)
  • I agree. I think a quick straw poll might be helpful in confirming consensus. --EpochFail 17:34, 9 August 2011 (UTC)
    • Just to clarify: as we discussed before, RCom is not in a position to grant or deny approval on behalf of the community or the Foundation regarding a research proposal (unless a project requires access to resources from WMF (private data, technical support, endorsement, hosting etc.), in which case WMF approval is explicitly needed). The role of RCom is to review a proposal and address any issues with the proposed methods that we have reasons to consider problematic. If the proposal is cleared (as in: it doesn't raise any major concerns) RCom can provide the appropriate type of support to the research team (e.g. in this case to facilitate recruitment or mediating communication with the editor community). We should also make it clear to researchers what are the implications of seeking support from the Foundation. The actual role of RCom and the review procedure are not formally defined on Meta (we only have instructions on how to submit requests, not what happens to these requests after they are submitted): that's a potential source of confusion for researchers and for community members. As part of the proposed OTRS-based system to triage these requests I suggest we draft a document describing how they are handled by RCom. The feedback on this talk page is a great example of a useful review process and what I think our role should be. Thanks everybody for helping out with this. --DarTar 22:20, 10 August 2011 (UTC)

Poll

Is this research project ready for RCom's approval?

• Yes This study should cause no harm to newcomers and the results of their experimentation will valuable to the community. --EpochFail 17:34, 9 August 2011 (UTC)
• Yes. To confirm my opinion stated above.--Ymblanter 17:48, 9 August 2011 (UTC)
• Yes. There doesn't seem to be any downside, and Kraut's group has a history of doing useful studies of Wikipedia.--Ragesoss 19:14, 9 August 2011 (UTC)
• Yes. The project is clearly aligned with Wikimedia's strategic goals on editor retention and the fact that it relies on existing community members from NPP to do field work in a non-disruptive way (i.e. without distracting editors from their regular activity) is definitely a plus. I consider this request to fall under Research:Subject_recruitment unless we have reasons to believe that these messages are technically not a form of recruitment. Thoughts? --DarTar 22:20, 10 August 2011 (UTC)

Given the slowing conversation here, the unanimous agreement of active discussants and lack of concerns raised at the Village Pump, I'm closing this discussion with a clear consensus that this project is ready to proceed. Thanks for your patience. Let us know if you need our help interfacing with the community. --EpochFail 23:36, 11 August 2011 (UTC)