Research talk:Committee/Areas of interest/Subject recruitment

Information

Meetings

• Talk:Research_Committee/Areas_of_interest/Subject_recruitment_processes/Meeting_2011-01-18

What now?

I'd like to re-try finding consensus for WP:Research and WP:SRAG in some form. There was quite a lot of conflict over whether they were acceptable in their current state, but I think with a re-focus and re-write, we'll be successful in finding consensus in the community. Focusing on enwp first will allow us to take advantage of a common language between all RCom members and address the recruitment need in the most active of community. --EpochFail 20:35, 27 September 2010 (UTC)

Proposal for research review

There was a thread on the Wiki-research-l mailing list wherein user:DarTar suggested that I might draft a proposal for research review here. This was part of a broader conversation about research review.

I propose that researchers who want to recruit Wikipedia users for research list their research on a public board for review. When they list their research, they should complete a form which includes the following information:

1. investigator name and Wikipedia username
2. names of co-investigators and their Wikipedia usernames
3. organizational affiliation
4. contact information, including email, phone number, and physical address
5. description of purpose of research
6. a description of what the researchers will ask of Wikipedia users
7. the amount of time expected from Wikipedia users participating in the research
8. the number of Wikipedia users the project intends to recruit
9. a description of the advertising plan
10. a list of all advertising or participant landing pages which are used in the conduct of the study
11. a description of the risks of participating in the study
12. a description of the benefits of participating in the study
13. an outline of the plan to protect the privacy of research participants
14. a description of the process whereby a participant may withdraw consent to participate and leave the study
15. an explanation of what happens to Wikipedia users who choose not to participate
16. an explanation of what happens when a participant engages the researcher, then later wishes for the researcher to remove them from the study
17. a statement of review for research ethics by their organization's ethical review board, or otherwise a statement their their ethical review board decided that the research needed no review
18. the desired starting date for the study
19. the expected end date for the study
20. an expected date for the results of the research to be published
21. a commitment to open access as described in the table here Research:Subject_recruitment
22. the route by means of which the publishing of the study results will be announced
23. a contact for asking questions about the study

In the process of completing the form the researcher will create a page on Meta which publicly posts all this information. At this point the research project is open for public perusal and comment on the talk page. Any registered Wikipedia user may review the proposal; the only criteria for review at this point is whether the researcher completed the form. If the reviewer determines that the researcher has completed the form then that reviewer may flag the form as "reviewed".

A research committee, as yet undefined, must review all reviewed proposals before the researcher may begin the research on any Wikimedia project. The review process, as yet undefined, will have criteria which include protection of research participant rights and consideration of comments on the talk page of the research project.

Once the research committee has approved the research then the researcher may recruit participants. In recruiting participants the researcher must either link to a landing page which gives information about the study and which links to the reviewed and approved research proposal page or to the research proposal page itself.

Before participating, all research participants must acknowledge having access to the reviewed and approved research proposal page.

How is this for a start? Thanks for your attention. Blue Rasberry (talk) 03:38, 20 March 2012 (UTC)

My two cents:

1. research approval is a must in cases where the researcher asks for Wikimedia help in getting volunteers. However, I don't see any reason to strictly demand going through the process in cases of users who e.g. do qualitative research and occasionally ask for interviews Wikipedians, whom they already know anyway. Any research approval procedure should be required only in case of projects which either require some sort of propagation (through Wikimedia, distribution lists, etc.), or are in some other way massive.
2. On a similar note, the committee should review only the projects, which really need to be reviewed, and also only ask for descriptions which really need to be explicit. Also, the requirements for the project description should take into account the nature of Wikipedia. Instead of requiring hundreds of researchers to perfunctorily describe that all subjects can leave the study any time they want, we can just make a requirement that subjects have to be informed of this right, and also prepare a short "code of conduct" for research on Wikipedia. This way, instead of making all researchers write more paperwork up, and also try to guess the proper procedures (important in case of beginners), we just can offer the minimal requirements scenario, to which all projects have to adhere.
3. "Human Subjects Research Committees" are typical for American institutions and have originated in the medical sciences, after quite a number of scientists did unethical and controversial studies. While I believe that such committees and rules do have important use also in the social sciences (and often serve well to protect the minors, the underprivileged, the handicapped, and other groups prone to damage), I think they should be introduced to Wikipedia in a way, which is more culturally neutral, that is balanced between other countries' research traditions, including e.g. European ones (clearly, difficult to be considered inferior, and yet different), as well as of other research-intensive regions (which I'm not listing simply because I have my own experience only with the US and Europe). Btw, what if a university does not have a "research ethical review board"? What, practically, can a researcher put under "an explanation of what happens to Wikipedia users who choose not to participate?" question (I'm guessing "They go to hell for not advancing the science", but seriously, do we need to emphasize the right to not participate, and the right to quit the research project at any time once again, and in a form?).
4. Before introducing an all-inclusive procedure for all research on Wikipedia, let's decide what needs to be protected by it and what categories of research projects need to be banned. This way it may be easier to start something a bit more tailored and customized for the real problem. Otherwise, we just generate a lot of additional work both on the side of the researchers and on the side of the committee.
5. Btw, making revealing one's name and Wikipedic nickname mandatory violates the right to remain anonymous on Wikipedia. While I understand that it is not a big deal in case of researchers who set up accounts for the purpose of research, I can imagine a situation where well established editors are also scholars and would like not to reveal the link between their Wikipedic persona and their true name. Naturally, to conduct research they should identify themselves (to the committee? the foundation? perhaps a "researcher" flag would be useful?...), but I can't see a reason why this always needs to be public. Pundit (talk) 12:59, 20 March 2012 (UTC)

My background is in United States research ethics and all recommendations I will make are with the bias of what is considered ethical by those standards. I edited your bullet points to be numbers so that I could respond to them.

1. I disagree. All human subject research should be reviewed by a third party. This is the standard in United States academic research.
2. I have no problem with researchers using an informed consent template, but all research subjects should have access to all of the information they need to understand their participation. It is never right to expect that participants should seek out information about the study external to the study.
3. Many researchers know that they have access to an ethics board and if asked, they will use it. If they have access and choose not to use it then they can just say so. If a researcher has no such access, then they should declare that they have no such access. The form I am proposing is not about making a judgment call but rather about presenting information to research participants which could help them make a decision about joining the study. The "explanation of what happens to (potential research subjects) who choose not to participate" is a concept in United States research ethics which has a lot of implications. I could give examples, but there are study designs where bad things can happen to people who are asked to participate and then refuse or who agree and then later change their minds.
4. The United States research standard requests some general information and protections for all research studies. I am more interested in setting minimum standards than I am for protecting all participants in all cases. I am not asking anything more or less than what is common in the United States. If another culture has a different system which is different yet equally protects research participant rights then I would welcome alternate proposals. Right now I am trying to find something which is better than nothing at all, not something which is the best of all conceivable systems.
5. Wikipedians have a right to anonymity. Researchers never do. If they recruit on Wikipedia using a Wikipedia user account then when they declare their name they should declare themselves and their organizational affiliation. Regardless of whether they want to reveal their names it is totally unethical in the United States for a scholarly researcher to conduct research without self-identifying. This is analogous to the established practice of self-identifying among paid editors who are editing on behalf of an institutional interest. Almost all researchers have an organizational affiliation.

If you have questions or want background on case history about the rules I suggested then it might be quicker to talk by phone or Skype or Google Hangout. I am not saying that what I am suggesting is best, but I can say that it comes with a lot of precedent and it is what practically all United States scholarly researchers would call "normal". I am trying to create a quick catch-all proposal which will mostly work in most cases. I acknowledge a lot of bias toward United States customs in this proposal, and this bias does put extra burden on the researcher and less burden on the research participant as compared to ethical systems in most other countries and cultures. I would love feedback on the extent to which I addressed your concerns and especially any ideas you have for a compromise which you could approve. One compromise which I would offer is that an acceptable answer for any question on the research proposal form could be "Researcher chooses not to answer". Blue Rasberry (talk) 17:48, 20 March 2012 (UTC)

Well, I don't exactly see how you addressed my concerns. You simply reiterated that you believe all this is necessary, and emphasized it as standard for US research, which again was something I've already pointed out (honestly, rather as a cultural flaw than anything else). We can dispute whether American predilection for declarative research ethics is related to the sad history of medical studies in the US, the contemporary protection of universities (which want to avoid going to court), or true protection of the informants. Quite likely, it is a mix of all. However, I don't think it is reasonable to impose one country's procedural standards on everybody else, especially when other research-intensive countries do not share the same drive to such procedures. This is particularly true in the case of non-invasive social studies on privileged knowledgeable adults, which is probably the majority of all Wikipedia studies. We need to protect Wikipedians - fine, but let's try to do so in a culturally neutral way, and also by adopting standards relevant for social studies research, and not mechanically copying those generalized (and coming from the medical environment where the protection of informant's rights has a totally different dimension). You clearly do not take into account even the simplest fact that some researchers may speak very poor English, and yet want to conduct research on some Wikimedia project. Or are you suggesting that all major languages should have their own research boards? Why not own research rules then?

The fundamental question that needs to be asked, before this procedure is introduced, is what exactly is it going to fix (on real, past examples). Instead of saying that something is a standard, let's try to identify problems which actually happened that this procedure may help to address in the future. My concern is that this easily can become an empty exercise, which has nothing to do with the protection of human subjects.

According to your (quite strong) statements, I'm a totally unethical researcher, because I identify myself by both my nickname and true identity only to people on Wikipedia I interview (while in all research institutions I only use my real name, and not the Wikipedia persona; and on Wikipedia I only use my nickname). Perhaps indeed I lack basic moral skills, because I still totally don't see why in your view it is important to reveal both the researcher's name AND their Wikipedic nickname in public even in theory. In practice though, because of my work on Wikipedia, I have been a target of several documented threats (including a martial arts bully) and I can see that in case of Wikipedia activists in particular, demanding that they reveal their Wikipedic nickname serves no purpose (after all, their research is under their real name, what on Earth is revealing the nickname for?).

All in all, I'm not saying that guidelines for research are not important (but sometimes our declaration of mandatory minima may be better than requesting and approving research proposals). I'm only saying that they should be compatible with the international research standards. Also, all procedures should be tailored to Wikipedia actual needs. cheers Pundit (talk) 18:47, 20 March 2012 (UTC)

Forgive me. I mean well. I really am doing my best to address you. If I say foolish things it is only because I am at the limits of my competence and not because I am trying to put you down.

I am open to hearing about non-United States non-English research standards, but I am very closed minded about postponing discussion until such time as those standards are developed if they do not already exist. In cases where there is no counter proposal, I say default to United States standard. If you or anyone else has counterprosals, I would love to hear them. So far as I know there is no English-language text describing the research ethics history as it developed in any country not heavily influenced by the United States standard. You say that it is not right to impose one country's cultural standards - I agree, so please propose any other culture standards you like. I appreciate alternative ideas and would rather present them all rather than to stop the discussion by saying that something is not right without offering an alternative. It would also be helpful if you say that you do not have an alternative if that is the case.

There is a saying on English Wikipedia that competence is required in any case. In the end when there are standards, regardless of anyone's language ability, the standards need to be respected above anyone's ability to meet the standards. They should not be lowered to accommodate researchers of low ability.

I have not conducted a review of actual Wikipedia needs. I work in medical research and clinical ethics and I know institutional away-from-keyboard needs, and what I proposed meets problems which happen in routine health intervention research. I created one of the articles to which you linked and am active on related ethics articles, and I am very biased to apply this system to Wikipedia because I see it applied to every other kind of academic research. I am trying to take into account what you are saying and I hope you believe my sincerity. Please forgive me if it seems I do not understand. I support your idea of anyone reviewing problems which have happened in Wikipedia research and creating an ethics code in response to that, but until someone does that work, and unless someone has a better idea, the United States system exists and could be the default until or unless someone can propose alternative models. I am thinking of medical research ethics, and I do think those ethics standards apply well to social research. That is another bias which I am pushing because I believe this to be normal in the United States.

I think you are joking when you seem to interpret what I am saying as a personal assessment of your moral system. I never raised the topic of morality; the concepts of ethical and unethical, right and wrong, good and bad, legal and illegal, just and injust are all separate topics and not to be conflated. The United States ethical code at times encourages behavior which is wrong, bad, illegal, and injust, but at almost all times the code itself can allow people to make a distinction between behavior which the code defines as ethical or unethical and that is completely separate from the other value judgments. If you say that you want to conduct research anonymously, then you are saying that you want to do something which a certain ethics code defines as unethical. If you choose to evaluate yourself in the context of an ethics code which defines a certain act as unethical, and understanding that you perform the act defined as unethical, then still the ethics code makes no judgement about whether you are ethical or unethical because the code only evaluates actions.

About the revealing of identity - the United States ethics code says that the researcher is relatively savvy and has a higher degree of agency, and the research subject is relatively naive and vulnerable due to having a lesser degree of agency. Researchers reveal their identity because in almost all cases they have more potential to harm the research participants than participants do to harm the researcher. This applies to social surveys also - Wikipedia users need their time protected and they need to have protection from subjection to advertising and commercial interests. Marketing preference social surveys intended to determine how to sell products to Wikipedians, for example, should be identified as such and perhaps should be treated differently from not-for-profit social surveys about improving the Wikimedia user experience.

The reason a person's real name needs to be revealed is so that Wikipedia users may contact the person's organizational affiliation if they have a complaint with the research. Yes, that makes them vulnerable to stalking. Yes, there are stalkers on Wikipedia and I have had them also. Yes, this is a compromising and vulnerable position for any researcher, but this is the United States research standard because historically researchers have had more power than research subjects and revealing a researcher's identity gives power to the participants.

The reason a person's name needs to be associated with a person's Wikipedia account is because Wikipedia has restrictions on paid editing. Almost all researchers are doing paid or personally-enriching editing with a conflict of interest to benefit their governing organization.

If you have more comments and are enjoying this conversation then perhaps we could break this into topics. I feel encouraged by your feedback and if you feel the same way about my responses I want to continue this, but I do think the length of this may be making our talk inaccessible to casual users. Thoughts? I am available by phone, Skype, or Google Hangout if you want to talk this through, or we can continue here. Let's get organized and slow the pace, though. Blue Rasberry (talk) 05:02, 21 March 2012 (UTC)

• Hello, I think it goes without saying that we both mean well :) I am certain that all of your suggestions are intended to increase fairness/ethical standards of Wikipedia research. I only disagree with the means and the anticipated results. If I say something foolish or incompatible with your background, please forgive me.
• Now, I'm glad you agree to accept more international standards. They usually involve relying on researcher's ethics, as well as at the discretion of their own institution's control systems, rather than on creating one-size-fits-all bureaucratic procedure. Therefore there are at least two clear and concrete counter-proposals: (a) instead of requiring submitting every single project for review, we can publish the research requirements (policies, guidelines...) that need to be followed (so, act in line with Wikipedia philosophy: tell what we expect and act only if there is a breach - per analogy, we don't require "editor's ethics report" to be filled in before people start to edit, do we? ;), (b) we can require e.g. just declaring the name of the researcher, the name of the project and a confirmation that their project is externally controlled (could be by H-S-R committee, but could be also the university, the NSF or other scientific foundation supporting the project, etc.). This way we not only lower the amount of bureaucratic creed (which is increasing significantly anyway), but also are more compatible with the international, and not local standards. Btw, I don't think you replied to the problem I pointed out, with different language studies (what should people who do not speak English do? abandon their studies? hire translators?...).
• I do not think that medical research standards (developed to protect the underprivileged, also in life-death situations) apply well to the online studies of well-educated, digitally privileged adults, and this is the vast majority of studies of Wikipedia. Again, this may well be just my fancy belief, but I think it is pretty intuitive. More importantly, I also believe that any additional bureaucratic procedure (in your proposal, quite major, time and effort consuming) should address some real problems that actually happened and need to be avoided in the future. If we cannot point to actual problems that happened, let's not introduce a huge procedure just because it is considered to be a standard in a largely different context in one country.
• Per anonymous research: I think the misunderstanding roots perhaps from my poor wording in the first place. I have never objected to identifying the researcher by their name and affiliation. I do, however, strongly object to requiring the researcher to publicly disclose the link between their name and Wikipedic persona. This is endangering all Wikipedia activists, who also are researchers (and the falsity of such approach I tried to show you on my own example). I hope you realize also that this is against the privacy policy of Wikipedia. In my own view, the bottom line could be revealing the name+nickname to the foundation and/or the research committee, but not requiring them to be public.

The reason why I don't want to move the discussion off-wiki is simple - I think we are not the only two people who need to agree :) Discussing here allows for other people to comment, too. take care Pundit (talk) 13:47, 21 March 2012 (UTC)

I want to break this into sections. I am not addressing all of the points you raised at this time. Blue Rasberry (talk) 15:59, 21 March 2012 (UTC)

Minimal process

Suppose that a researcher wants to conduct research on a Wikimedia project which will include collecting responses from Wikimedia users. I propose that among the minimal criteria for beginning this research, the researcher must have these things:

1. a registered Wikimedia user account on Meta (no IP)
2. a wikipage on Meta which names their research project and lists it on a dedicated page with all other research projects and links to resources about research
3. on their project page, the researcher must sign and datestamp a statement which says that they understand the research rules and will follow them.

If the researcher interacts with other Wikimedia users relating to the survey, then the researcher must provide a link to the page on meta.

I think that this system will prevent the following problems which I have seen happen with external Wikimedia research projects:

1. Having a registered user account will allow users who want to contact the researcher to do so in a way that other Wikimedia users can see. Making conversation public rather than forcing conversation to be private (for example, by email) empowers Wikipedia users by taking away power from the researcher, and this is good. Some researchers prefer to have conversation off-site and many Wikimedia users complain about them but have no easy way to organize on a Wikimedia project against an IP user if there is trouble.
2. Having a wikipage about the research project allows people to comment on the project in a central place and by sending people to meta connects them to resources relating to research guidelines. This also empowers research participants because it prevents them from being dependent on the researcher to explain research participant rights.
3. Asking the researchers to agree to read guidelines before beginning research prevents them from wasting Wikimedia users' time. A lot of organizations, companies, and research institutions come to the site and violate lots of rules. I think it would be good if before engaging the site they read some guidelines, whatever those guidelines are.

How do you feel about requiring this much, inflexibly, uniformly, and totally bureaucratically, from all researchers in all cultures speaking all languages? For languages with no research guidelines, the guidelines which the researchers should read are general Wikipedia guidelines. Blue Rasberry (talk) 15:59, 21 March 2012 (UTC)

I think it is a great step. Just requiring basing identification of the researcher, and the project, as well as referring to our research requirements (rather than demanding a report on human subjects protection) sounds very reasonable and does not impose excessive bureaucratic burden on the researchers and the committee. I'd add just one thing: if we demand having a Wikipedia account for the purpose of the research (which you well argued), let's allow sockpuppets in the case of Wikipedia activists, who wish to keep the privacy of their main account. Pundit (talk) 16:40, 21 March 2012 (UTC)

Done Socks should be allowed to protect user privacy. I am glad that we are in agreement about what is in this section. Blue Rasberry (talk) 13:39, 22 March 2012 (UTC)

Research review process

There are different levels of research review. I identify these and would appreciate anyone identifying more.

1. Self review - the researcher reads some guidelines then decides alone whether the research meets the guidelines. If the researcher thinks it does, then the research is allowed.
2. Form review - Upon completion of a form, the researcher is allowed to begin a project.
3. Community review - there is a dedicated research project page for each project. If there is a question about any project then the Wikimedia community can go to that page and discuss the research. The community may call for a research project to be disallowed.
4. Bureaucratic ruleset - there is a ruleset which a third party uses to evaluate a researcher's project. By the ruleset, the third party determines whether the researcher's project is allowed. The ruleset need not be invoked and researchers may begin their work without review, but if there is a problem, then any third party can check the research against the ruleset.
5. Research committee - a group of people, using criteria which they have not defined, make decisions about whether a research project is allowed.

I think that all of these things should happen, but the minimum is a self-review. I also think that researchers should use a Wikimedia account to declare the name of their research project and declare that they have done a self-review as the minimal criteria for beginning research.

I think a form is necessary because without sharing basic information about research it is not possible for anyone else in the Wikimedia community to discuss research. I am not saying that the form should ask any particular questions, but I do think that the researcher should share some information of some kind about the research project and the form should be a prompt to collect this information.

I think that the Wikimedia community should have oversight over research, just as the community has oversight over everything else. Anonymous Wikimedia users with no presence on the site cannot be tracked or easily discussed by the community. I think that a system should be in place to easily host a community discussion about a research project should the community want to do so.

I think that some guidelines ought to be in place and outside of debate. One guideline I would propose is that research has to be open access to the extent that most other licensing used on Wikimedia projects requires. Rules like this need to be somewhere, and if a researcher self-evaluates, completes a form, passes community review, but then any one person notices that the researcher is indisputably breaking a guideline, then the guideline still takes precedent.

I support having a research committee. Ethics is terribly complicated and no rule set solves all problems. Sometimes unfair decisions have to be made, and an ethics committee is a trust holder. In situations where a researcher follows all rules but somehow harms the Wikimedia community, then the research committee should stop the research. It is impossible to design a perfect system which prevents all problems and sometimes arbitrary intervention without explanation is necessary. The community should have some choice in who is on the research committee. Thoughts? What other types of research review exist? Are any of the types I proposed acceptable enough to be made mandatory for all research as a rule? Blue Rasberry (talk) 15:59, 21 March 2012 (UTC)

I'm inclined to say that all of these requirements may be needed in some cases (e.g. massive questionnaires to random Wikipedians, use of banner space), but there has to be a gradation of requirements. Clearly, one form of confirming the projects' validity is external approval (university's research board, national science foundation, etc.). Btw, I don't think that even in the American context it is that typical for an organization to have a research ethics committee for evaluation of proposals for researching the organization itself. Pundit (talk) 16:43, 21 March 2012 (UTC)

Let me clarify - everything I have written is about "human subject research", which is a technical term I am using in the sense of United States research culture. "Human subject research" refers to research which intervenes with people in a certain way and does not refer to all research which has some connection to the concept of humanity. I am not talking about an ethics code for researching Wikipedia as an organization if that research is not "human subject research", and I do not think discussion of that should be mixed with this discussion on this subject recruitment page.

In the United States research context, "human subject research" follows an ethics code and the most common protection of human subjects is to have some system wherein the entity conducting the research is not also the entity who decides which ethics code to apply to the research. For many other kinds of research which would not directly intervene in human lives there is no research ethics committee review.

The gradation I propose is that all researchers are required to first self review, then all researchers are required to complete a form describing their research. Completing the form has a dual purpose of forcing researchers to understand that there is some accountability on the site in that they have a minimal obligation to provide some information about what they are doing, and it also makes it possible for a third party to review their plans. At this point, it is possible for research to begin with no further review if the Wikimedia community permits this, but I think there will never be community permission for this in practice. Supposing that there is consensus for research to happen without review, then the self-review and the form is enough, but if there is community consensus that the Wikipedia community should review a research project then I think the community should be allowed to do so even if that infringes on any rights the researcher has to do research on Wikimedia projects. What I expect to happen is that some volunteers will insist that the researcher complete the form, so there will be reviewers who check to make sure the form is complete. There will also be the option for community comment on the research, and the community will have the option to disallow the research. Finally, I think in the end there will always need to be an option of some team to override community consensus in the case of popularly-supported research which harms the project, for example, by being illegal in the jurisdiction where the survey is conducted.

Thoughts on this? To what extent is this a reasonable outline? What alternative models can you imagine? Blue Rasberry (talk) 13:29, 22 March 2012 (UTC)

My intuition is different: I think that in many cases of research on human subjects (including e.g. a couple interviews with editors, or a simple, non-personal questionnaire for a limited number of Wikipedians) the community will not insist on the project being reviewed and approved prior to its execution (as it has been the case for many years). I think that there are projects which are either about sensitive topics or require massive participation, and these clearly have to be limited and closely controlled. You should also keep in mind the practical side - what exactly are you going to do if somebody asks e.g. 8 Wikipedians for interviews (say, on IRC) without clearing out with the forms? Would it be the same kind of offense as spamming on some wiki-list and soliciting massive participation in a poorly constructed questionnaire, asking for sensitive data? I think not and these cases should be treated differently, also in terms of our expectations towards forms/notifications/supervision. Pundit (talk) 15:22, 22 March 2012 (UTC)

I made a section below to discuss "researchers who do not follow rules" and responded to your questions about that there. For your specific example, I will say that if someone starts research without following a review procedure - forms or whatever - then if anyone finds that researcher they should tell them to stop research, comply with the rules, then restart research. Please continue this issue in its own section if you want to talk more.

In your example, it is my evaluation of the context of United States research ethics that 8 IRC interviews with Wikipedians would constitute human subject research. It is my assessment that researchers doing such interviews for almost any reason in the United States would need ethical review.

Let's suppose that there were no research review process of any kind, and someone started this IRC interview project. Suppose that some other user complained about the IRC project as being unethical. I would say that the researcher ought to stop their work until they address the complaint - would you agree?

Stopping to address a complaint would probably be an inconvenience for a researcher. Researchers would prefer to prevent complaints rather than address them. I think that researchers would prefer to have review if that raised the legitimacy of their work and gave them some protection against future complaints.

Supposing that the researcher completed a form describing the research, and the Wikipedia community could review that form. If any volunteer evaluated the research as meeting good research criteria (no one has proposed such criteria yet, I think) then I would be comfortable with research starting. The approval could be as simple as a template which one reviewer posts on the research page. If a research committee is formed and they become efficient at reviewing research, then they could also post their review; a counter proposal could be that the research committee passively approves projects by not posting notices on them but can actively post a notice on any project to stop it immediately.

However, if any community member opposed the research, I think that the research should be halted to check for community consensus on its legitimacy. Because I believe that any Wikipedia user ought to have the ability to halt research to check for consensus, I think that most researchers would request to get consensus on their research before they begin rather than have a higher risk of halting research in progress if an objection arises.

I think that reviewing research would be trivial for anyone who hangs out on a research board, and quickly detecting problems would become natural to regular reviewers. Just as many new Wikipedia users seem baffled by concepts like verifiability, NPOV, no original research, and no violating copyright, I think that many researchers will be at risk of making mistakes in proposing the conduct of their research. But just as Wikipedians quickly learn to detect problems with V, NPOV, and the rest, the research reviewers would quickly be able to spot problems in research projects once a rule set is made.

Does this scheme account for most outcomes? How do you feel about my saying that an IRC interview of 8 people constitutes research requiring review - should we start a section defining what constitutes human subject research? What do you see in the scheme which I am proposing which is impractical? I feel like what I am proposing is generally asking for two hours of reading setup time per research project from the reviewer, and the review time would usually be much less (hopefully 15 minutes) for a Wikipedia volunteer to check the project. Blue Rasberry (talk) 15:22, 23 March 2012 (UTC)

(1)Well, first I honestly do not believe that interviewing a couple of people on IRC should necessarily require an ethics committee review ON THE SIDE OF THE STUDIED ORGANIZATION. Even though in the US it is common to have such reviews on the side of universities, it does not change the fact that it isn't in most of the rest of the research world ((2)do you believe that European standards are inferior, or that they are undeveloped in this respect?). Even if we agree that an HSR is necessary, (3)it is still not entirely clear why should Wikipedia, and not the home university, be involved with it. (4)I find bureaucratic creep highly impractical. Also, besides what you write about the violating researcher's obligations, (5)I'm more interested in what we can practically do in cases of violations. Pundit (talk) 17:55, 23 March 2012 (UTC)

(1)It is commonplace for organizations to review research conducted on their client base. You emphasize "ON THE SIDE OF THE STUDIED ORGANIZATION" - researchers depend on permission from hospitals to access patients, from schools to access students, from places of business to access customers, and otherwise to people in charge of regulating a space for access to people using that space. A lot of research can be done on Wikimedia projects without entering the personal domain of users, but this board is about human subject recruitment. When people use Wikimedia platforms to reach Wikimedia project users I do not think there is any controversy in saying that the precedent in analogous settings is for the controller of the platform to request that researchers follow some rules.

(2)You ask me a pointed question - are European standards inferior. Forgive my interpretation if I am incorrect, but it sounds like you want to check me for sanity or prejudice. I offered to talk to you by phone, Skype, or otherwise voice if you need to do this. It is hard to check for irrational, unreasonable points of view in text. To answer your question - no, I do not think European standards are inferior. I think scholarly Europe developed standards with United States scholars and that the research standards are almost exactly the same. For example, the en:Declaration of Helsinki is a ruleset followed in United States research and obviously it is an international document.

(3)Wikipedia should be involved in HSR because Wikipedia has some obligation to protect users, as does any host which has guests. I would like the universities to actually conduct the review whenever possible, which is why I proposed having researchers declare their university affiliation and confirm that they are following their university's HSR policies. Any institution which reviews HSR will also tell the researcher that all research participants have a right to see the HSR review, and this is why I say that researchers whose institions require reviews ought to upload to Wikimedia a copy of their review because all participants have a right to access that immediately on request anyway.

(4)I think in shifting the burden of review to universities will prevent bureaucratic creep. The minimum I am asking is that researchers complete a form, not that anyone on a Wikimedia project critique a form in some personalized way.

(5)Please start another section on remedies for violations. In this section I want to assume good faith so that there can be a conversation about the experience of researchers who try to follow rules. Blue Rasberry (talk) 14:53, 31 March 2012 (UTC)

The fundamental difference is this: IF the researcher asks for active support from the organization side to recruit subjects, his/her research is often reviewed. If s/he doesn't and recruits the subjects him/herself, the organization does not meddle. Do you really think that the publisher of The World of Warcraft (one of the larger and more commonly researched virtual gaming communities) requires HSR review? As I've already mentioned, I believe your (1) proposal is very sensible: let's require researchers to identify themselves, confirm that they will comply to our (well formulated, detailed and explained) code of conduct and research ethics, disclose their affiliation, and explicitly state that their research is institutionally supervised. This is definitely doable, useful, and does not amount to bureaucratic creed (both on our and on the researcher's side). The only thing I'm saying is that going beyond that, that is conducting a real research project review by Wikimedia committee is not adding much additional value, is redundant (repeats what the university already does and in a way culturally compatible with the researcher's academic environment, rather than in a one-American-size-must-fit-all way), does not increase practical, non-rhetorical protection. But I'm willing to agree with you that separate, more detailed procedures should be used for all projects requiring Wikimedia support in recruiting subjects (perhaps this is the sensible compromise?). Ideas? cheers Pundit (talk) 14:40, 1 April 2012 (UTC)

I would think that the publisher of World of Warcraft governs research on a spam policy so that if anyone complains they treat the researcher like a spammer. That's not terribly different from what I am proposing here, except that on Wikipedia the moderation is done by volunteers who explain themselves and on commercial platforms moderation is done by paid staff who do not have to describe their motives. Also I think that Wikipedia attracts much more human subject research initiated on its own platform and between strangers than many other platforms.

Thanks for calling my proposal sensible. I really think that most of the HSR review on Wikipedia could be handled just by having a researcher complete a form which will typically take 30-45 minutes and rarely more than 1-2 hours.

I agree that a mandatory research project review by a Wikimedia committee would not add value, and would be redundant, but I think that if anyone in the Wikipedia community has an objection to research then they ought to be able to post on the discussion page of that research project and invite others to share in the discussion. If the consensus is that the research is problematic then I think the community should have a right to ask the researcher to stop, just as in other places on Wikimedia projects people who go against community consensus are asked to stop their behavior.

I am not sure if we are in agreement or not. Perhaps the point of contention is what researchers should disclose. Thoughts? Blue Rasberry (talk) 17:25, 11 April 2012 (UTC)

I think we're reaching an agreement. I believe that there is merit in gathering basic info from all researchers (more like 5-10 minute questionnaire, stating the affiliation, general project's scope, etc.). Then, IF there are complaints or IF the researcher needs help in recruiting subjects (both are big ifs and I'm sure that many projects will not fall there) we can require more detailed explanations. The possibility to comment on the project of course is a good practice. Pundit (talk) 20:16, 11 April 2012 (UTC)

Non-English speaking researchers

I have no idea what non-English speakers do on English-language Wikimedia projects. The entire world is unfair to non-English speakers. I think that understanding English is among the single most important factors in determining how much access to information a person can get when using the internet. I know that despite many Wikimedia projects being intended for multilingual use, most policy is in English and no language has extensive work on it like the English version. Non-English speakers have an unfair lot in life.

To govern research on Wikimedia projects I propose that a major check on research ethics be that the researcher must complete some form disclosing information about the research project. I do not necessarily think that this form should be reviewed in any particular way, but I do think that it should be publicly posted so that anyone from the Wikimedia community can read it when it is completed by any researcher.

The form ought to be translated into as many languages a possible. Researchers ought to complete their form in their own language and post it on their research project page. At that point, each language community can have some degree of self-governance about the legitimacy of the research and the community can use established procedures to govern the behavior of researchers as they would any other user.

In doing this, I feel like English and non-English speaking researchers would have near equal status in their research. In both communities researchers would be subject to the review of people who understand their own language.

I am not asserting what information the form should collect, but it should collect something. Perhaps the minimal acceptable form could be "Describe your research project in at least one sentence." Thoughts? Blue Rasberry (talk) 15:59, 21 March 2012 (UTC)

There can be quantitative researches requiring only minimal understanding of English, and still non-passive. Yet, on the English Wikipedia I believe most of the studies will not encounter this problem. I rather see the need for Wikimedia research policies to be planned with their propagation across project in mind, and for this your point no. 1 sets the good ground for a fair compromise. Pundit (talk) 16:45, 21 March 2012 (UTC)

You seem to be saying that self-review (my point 1) sets a good start. What happens if someone does self review, is actually following the rules, but a user claims harm from research? What is this user's recourse? I feel that without something beyond self review there is no protection for research participants in case of perceived harm. In contrast, having the researcher complete a form (my point 2) would offer a significant increase in protection to all participants at the expense of a low time-cost to the researcher. Thoughts? Blue Rasberry (talk) 15:01, 31 March 2012 (UTC)

harm such as?... How are you going to truly protect users? So far, I see these procedures mostly as a bureaucratic chore. We don't have any practical measures to do anything beyond them. Thus, I see a zero increase of real protection, with a significant increase in paperwork. Unless it is the rhetorics of protection that we care most about :) Also, as I've already mentioned, we're taking the role of research institutions. If there is harm done in research, clearly the university has many more possible courses of action than we do. Also, universities are usually better equipped in evaluating research than we may. We should not usurp roles taken by universities. Pundit (talk) 14:32, 1 April 2012 (UTC)

The most common harm in most research on Wikipedia is when half-prepared researchers send a flawed survey to users then later find after they get responses that the survey is unsound and useless. I can show examples of this, but if you look at research regularly I assume that you have seen many of these. It is unfair to tap into the time of Wikipedia users who trust that research on Wikipedia is legitimate, and then betray that trust by not following through with processing the results by producing a research report. Many researchers collect data through Wikimedia projects and then disappear, and this harms the Wikimedia community.

Asking for researchers to consider what they are doing is not a bureaucratic chore, and it provides a huge increase in protection, and the 1-2 hours it adds to their work is not a significant increase in paperwork. I am not in favor of a mandatory review of the information which researchers provide, and I do not think it would be good for a Wikimedia community to take the role of a research institution in reviewing this work. However, at anyone's discretion, I think anyone in the community ought to be able to call for a community review of any research project if anyone from the community wants to come together and discuss a specific project. I think most projects will not need such review, and generally the burden of review should be on the individual researcher and whatever university or research institution is supporting them. I think they should disclose any review they have had if they had one so that the Wikipedia community can contact their institution in case of trouble.

You are starting from different premises and coming to a different conclusion. I am not sure where we diverge in thought but I am sure I do not understand you. Perhaps we should choose a specific research project and talk about it as an example. Blue Rasberry (talk) 17:09, 11 April 2012 (UTC)

Yes, I understand the problem you're describing. Of course it applies to some categories of research (and thus not all of studies should be required to clear out this way). But my main concern is something else: I just don't believe that such a review will indeed cause self-reflection. However, I agree with you that setting up an ad-hoc review of a research study IF there are doubts about it makes perfect sense. So, in general researchers should be required to provide just the basic info about the project, and not be required to go through any additional reviews on our side. If there are doubts, a review on Wikimedia side can be arranged. Same principle applies to massive studies (we can even put a numeric threshold, like 50 subjects or more). Pundit (talk) 20:20, 11 April 2012 (UTC)

Researchers who do not follow rules

Pundit raised an issue about what to do if researchers do not follow rules for research. Most commonly this will happen when researcher begins work without knowing that the rule set exists. There could also be researchers or spammers who know the rules and ignore them.

"Assume good faith" is the Wikipedia policy. If someone does break rules, then that person should be asked to stop research, check the rules, then only restart research after beginning to follow the rules.

If there is a user who acknowledges the rules and then chooses not to follow them then the reaction should follow the precedent of dealing with other users who break rules. Blue Rasberry (talk) 14:56, 23 March 2012 (UTC)

That is? If the user does not even disclose their nickname on Wikipedia, there isn't much we can do, is there? Pundit (talk) 17:58, 23 March 2012 (UTC)

It depends on what they are doing. If they are posting ads to recruit for human subjects to participate in research then we can treat them like spammers, which they would be. What case did you have in mind? There are benefits to following community rules on Wikipedia and responses to users who do not follow rules; in what ways is the current system not enough for managing all classes of rulebreakers? Blue Rasberry (talk) 14:57, 31 March 2012 (UTC)

Posting ads is an entirely different matter. If we introduce your system, it'll apply to ALL researchers, not just the ones using ads. They can be recruiting subjects e.g. through "email user". I can't see how enforcing an all-encompassing system is significantly beneficial, and I don't see a practical possibility of any possible measures against violators, who do not reveal their Wikipedia logins. Again, do you? Pundit (talk) 14:28, 1 April 2012 (UTC)

I am not sure I understand you. Anyone who uses the "email user" function in a way which Wikimedia policy prohibits should receive the same response that is already established for spammers. Human subject research in every jurisdiction either falls under some all-encompassing system or it is unregulated, and I am not sure if you have objections to anything "all encompassing" here or why you might. For preventing violators, English Wikipedia has an "assume good faith" policy. I would like to start with the assumption that researchers will follow rules if asked to do so. Is there any part of Wikipedia which has protection against a user who is steadfast in their intent to violate rules? I see a ruleset as significantly beneficial and natural. I am not sure where you are leading this thread, but I think maybe this thread is splitting into multiple unrelated points. Blue Rasberry (talk) 16:55, 11 April 2012 (UTC)

The thing is that we have protections against recurring vandals. Recurring researchers are very unlikely to happen, most of this is a hit-and-run thing, which does not have any continuity. If somebody sends emails to recruit subjects, no measures taken post-factum will remedy this. I also would like to believe that just stating the rules will be sufficient in most cases, and this is also why I so clearly oppose a large procedure and obligatory reviews for everybody doing research on Wikipedia. I think we need this only for certain categories of projects (i.e. those which raise some doubts after initial descriptions, or those which rely on recruiting many subjects). Pundit (talk) 20:25, 11 April 2012 (UTC)

United States research versus other models

I live in the United States and work in human subject research ethics, particularly for clinical trials. I am not a scholar on the culture of clinical research in the United States, but after a few years in the field I feel like I can recognize what would be ethical clinical research in the United States and what is going to be widely considered as unethical by United States standards. Is there anyone with a broad understanding of global human subject research who can describe any differences between practice in the United States and practice anywhere else? I suppose it is ego-centric, but I assumed that most of the protections for human subjects were nearly uniform when they exist at all. Blue Rasberry (talk) 16:48, 11 April 2012 (UTC)

I live in the US now (I'm on a sabbatical), but I have a tenured position at European university. I've had 4 visiting appointments (lasting 1-2 semesters each) at leading American universities so far and I believe I have a fair understanding of HSR procedures here. My experience is only with the social sciences, and with sociological/organizational studies. On the other hand, I believe this is more or less what is predominantly done on Wikipedia, and I don't believe that clinical research rules are necessarily applicable (different risks are possible, different protection is needed, different levels of what you can assume of the informants able to make an informed decision applicable, etc.). The most general difference between Europe and the US is control over the research process in Europe is executed through peers (other scholars). Research projects clear out at university level and often also at the granting institution as well. There are no separate research ethics committees run by specialized administrative staff. I believe that this model is more flexible and more applicable to social studies than the procedures taken literally from medical research. Pundit (talk) 20:49, 11 April 2012 (UTC)