Referees/1

This is a continuation of Referees, which was getting too big to edit further.

Please feel free to edit either page.

Baselining

These concepts and facilities can be extended to provide a means of baselining Wikipedia, for example in support of Wikipedia 1.0.

Assuming that the above scheme of refereeing was in place, there are three extra things required to baseline a fixed edition.

The first is to set up an editorial committee. This means creating a new class of user, similar to the three classes of reviewers described above, but completely independent, with the infrastructure to support it, such as a flag on each version of each article that tells whether or not it has been reviewed by a member of the committee.

The second is to provide a special mode of Wikipedia access that only reads articles with this flag turned on. Links to articles that have been reviewed (for this baseline) will always go to the latest reviewed version. Links to articles that have not been reviewed will appear broken.

And lastly there are some niceties, all sorts of tools that would make the editorial committee's work easier. Let's leave those for the moment, it's not trivial to design them but it is doable.

But meantime, baselining has other uses. We could start on the baselining software by providing a user option that allowed control of the level of review of the articles displayed. For example, you could set this level to be refereed articles only. Then all links would lead to the latest refereed version, and if one didn't exist they would appear broken. In this way, a reader would be able to select a higher level of quality than provided by the existing Wikipedia.

At the same time we'd probably want to provide a user option that would not break links to unreviewed articles, but simply provide a warning message. In that way, a user could elect to view the latest refereed verion of an article if one existed, but to view a less reviewed version if not. This is one of the tools we'd want to give an editorial committee for a baselined version such as 1.0, as well. There are a bewildering number of possible options, and again designing the user interface is not trivial but is doable.

Wikipedia 1.0

If this is really to happen late in 2004, the software to support it probably needs to happen in the first half of 2004.

Other uses of the baselining software

Creation of a new editorial committee should be a sysop function, not a developer function. If this is done then other projects might be:

A G-rated Wikipedia

Again, it's just a matter of setting up an editorial committee, and setting the flags. However, it would be good to have an alias for this version of Wikipedia, so as to be able to encourage its use in schools, and to give us something to safely show our grandmothers. Maybe www.general_wikipedia.org?

This is an opt-in form of filtering, so it's in no way censorship of the main Wikipedia.

In view of the interest in this expressed elsewhere, it might be good to start this earlier rather than later, perhaps in read-only mode. That is, set up the http alias and the approver group, but allow no edits through this alias, rather direct people back to the main Wiki (which parents and schools could then restrict).

But it would be really good if the G-rated Wikipedia allowed editing exactly like the main one of which it is a subset. This is not trivial, it requires some software and administrative effort but it's very desirable. These are the Wikipedians of the future.

One way of doing this is to allow changes though the G-rated Wikipedia to only be made by logged-in users, and then to be only visible to this user (and to the unrestricted Wikipedia) until approved, when they would become visible on the G-rated one as well. But, there's a complication, a new sort of edit conflict. The G-rated Wikipedia is often going to be editing a version that is back-level on the main Wikipedia, as the one they see is the latest approved version, not necessarily the latest version. Subsequent changes to the main Wikipedia will have been unintentionally reverted if this is simply saved. There are several ways of dealing with this.

Initially, we could just make it a simple save, and it's then a function of the G-rating approver to decide whether to leave subsequent changes on the main Wiki reverted (and it's also possible that someone else already will have restored them), merge them, or what, and to make the necessary edits on the main Wikipedia. This could work quite neatly. The history as seen from the G-rated Wiki only includes approved versions of course. If there's no merging, these will just be the edits as done through the G-rated Wiki. If there is merging, they'll see another edit as well, merging in the changes made in the unrestricted Wiki.

If this overhead becomes serious, deal with this issue then, possibly by sadly and temporarily at least going back to read-only mode for the G-rated version (which means it's not a Wiki at all of course). But there are lots of other possibilities. Brainstorming needed.

Specialist encyclopedias

We could similarly define a subset of Wikipedia, such as a musical encyclopedia. With some of these, it might be OK to let any editor set the flag to include an article, ie the editorial committee is defined to be all Wikipedians by default (perhaps including guest/not logged in, perhaps not). This is an open, as opposed to closed, editorial committee.

This could continue alongside efforts to classify articles under Dewey Decimal Classification, Library of Congress Classification and other schemes. It would be helpful to give these projects software support, but this is independent of this proposal. Where articles are classified under any system this can assist the editors (open or closed committee) of specialist subsets of Wikipedia, especially if software support gives them effective tools to use the schemes. Tools developed for this might also be useful for general browsers of the encyclopedia.

Collegiate approval

Much of what is suggested above relies on the concept of a college of approvers who each take responsibility for each others' actions. At each of the three levels, this is the approach suggested here. This is similar to one of the traditional models of University scholarship. All degree holders are part of the convocation of the University, and are assumed to have an interest in maintaining its standards in all areas, not just the one in which they are qualified. Similarly with the college of PhDs who are a subset of convocation, but have extra authority, not just in their own fields but generally.

Thus there is continual peer review at these levels, just as we have in the current single tier system. This proposal is essentially to add three more tiers to this, one for immediate use and two for later use as we gain experience in it, all three to work in much the same way as our current single level does. All three new levels are clearly defined, the first being just willingness to review, the second adding expertise at an unspecified level, the third adding expertise at the level of a recognised and quotable authority.

This is a very important consideration. The desired culture of each level of approval is exactly the existing culture of our existing base level. It's just the membership is, frankly and unavoidably, more restricted.

If someone at any level consistently underperforms, then in theory at least they could be removed. Doing so would be traumatic. Any system will have this problem! A symptom of this would be their frequently appearing as a dissenting voice, or approving articles to which others dissented. That may mean they are not fitting the consensus model, or it may alternatively mean that they are contributing a lot and noticing mistakes others are making. To decide which, look at how these disputes are being resolved.

There is an additional problem that some areas of expertise will be better represented at each level than others. The approved articles will reflect this bias. If, for example, the tree of life articles are the first to be approved to specialist level (as is suggested above might happen), there is then a risk that these will set the tone for articles in quite unrelated areas approved to this level. To counter this it is to be hoped that other areas would quickly join them. Perhaps computing would be a candidate, we seem to have some good expertise here, or mathematics. Any Wikiproject is a strong contender for early appointment of specialist reviewers.

At both the basic review level and the specialist review level, it is quite appropriate for an existing article to be refactored by the reviewer specifically for the review. This refactor would be a normal edit, and would create a new version in the normal way. There is no separate namespace so it becomes the current version, and assuming it's done well that is good.

At the top level of referee this is inappropriate, there should be more independence between reviewer and contributors. If a reviewer wanting to act as referee feels a refactor is needed, they have two options:

• Do it themselves and then get someone else to referee it.
• Get someone else to do the refactor, independently.

Spelling, grammar and format corrections would be OK, but perhaps we'd suggest that someone else could be requested to do even these. But, none of this is enforced by software, just by goodwill and consensus.

See also The future of Wikipedia#Editorial_process.

Edit wars

Edit wars can still happen, at any level. But an edit war at the existing base level will be invisible to viewers who have set their preferences to other levels, because they only see approved articles. Hopefully they will be rarer the higher you go. For an edit war to happen at referee level, we've got a major problem with our referees!

Comments welcome

Edit anywhere! In particular if this is covering ground already covered elsewhere, provide some links to it please.

If there's positive discussion I'll come up with a more definite software proposal. Andrewa 01:13, 30 Oct 2003 (UTC)

I can see this proposal is in line with my proposals like Wikipedia needs editors or en:Wikipedia:Article scoring. I don't care much about technical details in implementations. You have my support.

The only difference I think significant is that I am rather interested in changing the culture while you seem not. I think it is important to make some change in culture. Many valuable contributers left wikipedia with the feeling that they have little support from the community and see it is wasting time to tackle those who attempt to implement POVs, those who only care if their points are included or not. I think there is a bad habit we have developed and I believe some systematical mechanism can cure or improve it somehow.

-- Taku 23:10, 30 Oct 2003 (UTC)

I do think that the culture can and should be changed, I'd just prefer it to happen gently. See my comments on measuring disputes on the Pump. And thanks for the comment, and support. Andrewa 23:34, 30 Oct 2003 (UTC)

Quality standards?

You gloss over this point: "Any reviewed article should be NPOV, attractive, accurate, meaty, grammatical, and probably some other things besides. No reviewer should sign-off on an article which isn't all of these."

Who decides what is quality and what is not? This is a tricky issue that you are avoiding. As I see it reviewers should have very limited ambit.

I agree it's tricky. But this is an issue for any approval mechanism.

I'm very sorry you see my attempts as "glossing over" and "avoiding". That's not what I want to do at all. How would you address the issue? Or do you see it as a case against having an approval mechanism at all?

I'm not trying to address here the question of whether we need such a mechanism. Rather I am saying that if we are to have one, here is one that avoids many of the problems of other proposals.

As to "who decides", this is collegiate, at all levels. Exactly as now, at the existing level. I thought I had made that quite clear. Actually, I think my proposal is the first to address this very good question.

You suggest that at the highest level you want people with PHD's. Firstly, you haven't asked the question whether people with PHD's are by definition better reviewers for Wikipedia. Do we want Wikipedia trapped with the constraints of Western academia? It may be they are better reviewers, but you've assumed it. It will also make the encyclopedia elitest. Alternatively, you could allow reviewers to evolve out of the system as is currently happening with sysops and reward reviewer status to those people who have proved in their actions on Wikipedia that they have expertise and understand NPOV. That seems more in line with Wikipedia ethos. Not everyone can afford an education, and I have meant plenty of educated idiots in education. Secondly, many subjects are not suitable to academic reviewers, i.e. Star Trek and other hobbyist subjects. Why would you want academic reviewers there? : ChrisG 16:12, 6 Nov 2003 (UTC)

Quite right. I want PhDs for their authority, not for their expertise. The expertise is exercised at the specialist review level. That's one very good reason for separating these levels.

No, I don't want Wikipedia 'trapped...' and that's one reason the proposal is designed as it is. Presumably you still see this as a danger. How would you better avoid it?

No, I haven't assumed this at all. Nor do I see how you can claim that a proposal that recognises (at the specialist level) expertise by a collegiate mechanism completely independent of all forms of academic qualifications as 'elitist' and supporting existing academic power structures. Rather, it extends and empowers the structure we already have which is a severe challenge to these structures IMO (and you ain't seen nothin' yet).

On the subject of Star Trek, firstly, in this proposal the expertise of those competent to do reviews in this area would be recognised and controlled by exactly the same mechanism as in any other field, at all levels. Secondly, in Australia such subjects are potentially studied by academics at all levels. I don't know whether anyone has yet done a PhD on Klingon or any other trekkie field, but there is in theory no reason why not. Thirdly, whether or not such authorities exist, it would be quite appropriate to appoint referees from fandom, remembering that at this level we are concerned primarily with their authority, not their expertise.

You make some very good points. I have not expressed the proposal perfectly by any means, nor is it perfect. IMO the issues you raise are already covered if you read the existing proposal carefully enough, but then I wrote most of it so I know what I meant!

In particular, I hope the matter of who would be the referees and what they would do is clearer.

I hope this clarifies them a little. Andrewa 19:35, 13 Nov 2003 (UTC)

Your openness to electing people "from fandom" as referees (if they can demonstrate that they are a recognized authority in the area) resolves one of my main concerns about the proposal.

Two other quick questions I was wondering about. Would it be appropriate for people to review their own biographies? And would the page setting out an individual's qualifications be edited by the individual or by the review board/person that assigned privileges to that individual?

Other than that my only comment is to think about the interface presented to readers and reviewers. Text across the top of an article can look ugly and is often far less effective than a simple icon scheme. That said, at this point in time, I think this is a reasonable proposal. Cedars.

Seperate Namespace

A seperate name space would probably be necessary:

• Reasons of simplicity. Users, especially corporate users, throughout the internet would want to link directly to the 'approved' wikipedia pages. Such users will not want a two step approach of linking to Wikipedia and then clicking to the last approved version. Such a different namespace would attract many users who would not trust information from such a 'dubious' and 'untrustworthy' source as Wikipedia.
  • This is already addressed. Corporate users wouldn't need to click on each article. They'd just need to set a user preference, and would then see only articles at the level they had selected. No separate namespace is required for this, at least, no separate article namespace.
    • This could work obviously with the more intelligent users of the internet. However, the concept of Wiki is scarcely known, and would act as a barrier to take up of the encylopedia. Many users of the internet never change settings because they are either too lazy or not technically aware.
• People who are interested in Wikipedia because it is a free, encyclopedia can have their own 'project', so that will not feel the need to interfere with the anarchic nature of the bleeding edge Wikipedia.
  • Hmmm. Can you elaborate? That seems to be counter to the culture of Wikipedia, and I'm not convinced it is necessary.
    • As I see it there are actually two tendencies hidden within the avowed common culture of Wikipedia. People who see Wikipedia as above all an encyclopedia and people who see Wikipedia as above all a Wiki. People differ in degree. 'Extreme' encylopediaists grow frustrated with the current compromise, because as they see it good pages get trashed, dislike that people create pages on 'irrelevant' things, and want more controls over who should edit. While 'Extreme' wikilovers dislike the fact that pages are often reverted by the Encyclopediasts and dislike the attempts to create more controls on the Wikipedia. Establishing a seperate namespace would widen the community, because it would bring in more encyclopediaists, because they would see Wikipedia as a means to and end; they would use Wikipedia to create the content, and then the quality material would be used to create the 'proper' encyclopedia. While Wikilovers would stop being irritated by Encyclopediaists wanting more controls and could just get on with creating content. See the section on Editing One's Subsequent Editors (and a proposal for "locking in" mature entries Wikipedia village pump for a clear example of these tendencies in action or at least the Encyclopediasts. One might describe the tendencies as Realists and Idealists as suggested by someone in the discussion.
• If Wikipedia becomes popular on the internet, the fact it is from one perspective only a 'test' or 'unofficial' encyclopedia will make it far less of a target for vandals.
  • I think this is a very real issue which we are only just starting to feel. It's addressed a little under Political dispute but this is still just a draft and we seem very reluctant to brand particular disputes as this or particular users (even hard banned ones) as behaving in this fashion. But I also think it's addressed by the proposal without using separate namespaces.
    • I think once Wikipedia grows exceptionally popular it may well come under organised attack by Faith based groups because it is so open. Unless we created a stable (semi-official) Wikipedia the current Wiki structure could be frequently overwhelmed. For my part I would always use the dynamic Wikipedia, and would see the static Wikipedia as a useful distraction so real work can get done.
• A refereed Wikipedia is going to evolve out of Wikipedia 1.0. It will need its own playground to prepare it for publication. You can use the existing Wikipedia for most of the work, but in the final stages you will need to see Wikipedia 1.0 in stand alone format, which will require a namespace. :ChrisG 12:02, 3 Nov 2003 (UTC)
  • The existing proposal provides a means of seeing 1.0 in stand alone format, so far as content is concerned. The need for another playground (or do you mean sandpit?) is not obvious to me. Certainly it will be needed for format, but not for content. The format can be largely developed independently of the content. And there will be some checking of the final product in its final format.
  • There could also be checking on a frozen version of the content, not necessarily in its final form. This is in addition to the proposal, and consistent with it. A new article namespace may be the best way to do this, agree.
  • Or, we may be brave (some would say stupid, but brave IMO) enough to see a stand-alone QA as not good use of resources. QA is no magic bullet. It can improve quality, it can assure quality, but it can't perfect anything, and it can even be a complete waste of time. I suspect that this final QA of content would achieve something but not a lot, and would be best kept to a minimum. Andrewa
    • I mean for final formating and organisation. It is important that that is all it does, because you don't want to see differences in content, because that would split the overall project.

Thanks for your comments, they are excellent IMO and right on topic. Andrewa 18:20, 3 Nov 2003 (UTC)

OK. Now that we've clarified a few things, I agree more or less with all these points except for one thing, and that is I'm not convinced that a separate article namespace is the best way to address them.

It seems to me likely to lead to a messy and inefficient database structure. For the final editing of a static version, where content changes are trivial or banned and there's no desire for them to appear back on the main Wikipedia anyway, I'd use a stand-alone copy of the database (or a subset of it actually). For the rest, I'd still prefer instead to use the versioning that already exists within the article namespace.

The point about users never changing user settings is a good one, but the solution is just one of the interface we provide. You will need to provide some way of directing them to the 'approved' namespace, as they won't know that concept either. So my proposal is, rather than have a different namespace, use the same mechanism to set their user option to only see 'approved' versions of articles in the main one.

Possibly the best way of doing this is by having a separate website with its own domain or subdomain name, just as I suggested above for a G-rated Wikipedia. But I intended this is simply another way of accessing the same database and software. Andrewa 15:57, 4 Nov 2003 (UTC)

A seperate domain with different setting satisfies most of my objections in the short term. However, if vandalism becomes a real problem in the long term then it will be necessary to have truly seperate domains to minimise the problem. : ChrisG 00:21, 5 Nov 2003 (UTC)

---

Comment by 128.175.112.225

at 20:16, 8 Nov 2003:

This proposal sounds pretty authoritarian to me. Don't "people with qualifications" tend to be biased in favor of orthodoxy and conservatism, not to mention even such people can have their own personal idiosyncratic biases. And besides, quite a number of topics here just don't have enough "expert contributors" as it is. Why not just create in addition to the "Discuss this page" talk page where people discuss the topics covered by the article in question and how to "improve" it a "Review/critique this page" open to everyone?

At the level of referee, the whole idea is to appeal to authority. This to me is the simplest and least interesting level of this process, but there is a call for it, and one advantage of this proposal is that it accomodates this in a framework that is anything but authoritarian. And there are some severe curbs on the authority of these people. Firstly, the idea of independence. They aren't allowed to approve their own work. Secondly, the lack of any formal recognition of their areas of expertise. There are no personal kingdoms offered here. It's collegiate.

No, what I meant was, even "people with qualifications" can be biased in their review and not so much abuse of power. And besides, even if they're not allowed to approve of their own work, wouldn't they be less likely to approve of differing opinions by others?

Certainly, but isn't everyone subject to this?

The other levels are rather egalitarian. I've suggested we offer reviewer status to anyone willing. The function of this level is just to flag versions that are "finished" and ready for public consumption. I've also suggested that readers should be easily able to turn this off and on. Whether they see the latest version or the latest reviewed version is up to them. That's authoritarian?

Why not just have a "rate this article" utility poll open to anyone? If you don't like a persons rating, you can give your own rating as well...

No reason at all. It's a different mechanism, and will produce a different and valid result, useful to those who value such things. It would address some of the same issues. A similar ratings scheme works well on Amazon.com and a couple of other places, and would be valuable to Wikipedia I think. But I don't think it would replace this scheme. I could be wrong. Perhaps we should do both. The main problem I see with this is that our software development volunteers are stretched enough as is.

There are a couple of things to work out. People at Amazon.com are evaluating a fixed version of the book review. The author doesn't normally update it after they've done it. Wikipedia is not like that at all. There are a couple of answers given already to this, but none completely satisfactory IMO. It would be a much more useful thing to add to a relatively stable version of Wikipedia, which is one thing my proposal can help to achieve.

Wouldn't you have the same problem with refereed articles? Unless we have tons of referees continually updating their reviews, wouldn't refereed reviews also become outdated? At least with a "Review/Critique this article" page, it's far easier and faster to update because anyone can do it.

No, you wouldn't have this problem at all.

One of the key features of the proposal is that all levels of review apply to a particular version of an article. I think you are just confusing things by talking of 'refereed reviews', just BTW. That sounds to me like a verbal review, which is exactly what I'm not suggesting, just a yes/no approval of a particular version.

No, these approvals will never get out of date. That's a very important point too. The approved version will always remain available, regardless of subsequent edits, exactly as it does now via the page history. The only difference is, there will be an approval process, and a mechanism for viewing only approved versions available as an option to those readers who want to use it.

Yes, the review process will take a great deal of time and effort, probably 'tons' of people, whatever system is used. This proposal minimises that effort by using a simple yes/no system, which also maximises the benefit.

The approved versions will lag behind the unapproved in content, as in any approval scheme. For this reason I hope and believe that what I have called the base or 'open' Wikipedia will always have readers. But it's obvious from the discussion to date that some writers would like to give readers the option of a reviewed version. That's all I'm suggesting here.

Thanks for the opportunity to clarify these points. Andrewa 09:21, 15 Nov 2003 (UTC)

I think the specialist level is where it will get most interesting. Yes, we have shortages in some areas. But is this possibly because these people are frustrated by the ignorant efforts of others? I recently had someone update the caption of a photograph I had added, changing it from "fretless" to "fretted". I have owned and played the instrument in the photo for over 20 years and would know if it had frets!!! It hasn't, but like many ABGs it has markers that look like frets. I have many other examples of similar edits which are just poor research, and have done one such edit (that I know of) almost as bad myself.

But in a "Review/critique this article" page, anyone can critique any other person's critique as well and it's left to the reader who, if any, to trust...

True. How would this be any different to what's already happening on the existing talk pages? There's also still the question of which version of the article the comments apply to. How would you handle this? Andrewa 20:52, 14 Nov 2003 (UTC)

Perhaps we all do? My definition of "expert" is "someone who has already made most of their mistakes".

Thanks for your input. I think it's what a whole lot of others are thinking too. Great to talk about it. Andrewa 18:31, 13 Nov 2003 (UTC)