Learning patterns/Recruiting contest judges

Learning Pattern Library

Recruiting contest judges

problemFind the right people and tools to judge a writing or photo contest.

solutionExperienced Wikipedians, content experts, contest participants and bots all make great judges.

creator• KHarold (WMF)

discuss

endorse

status:in progress

What problem does this solve?

During the planning stage of a contest it is important to think about how the contest will be judged and find the right people and tools to judge contest entries.

What is the solution?

Who should judge

Experienced Wikipedians

Invite past contest participants to act as judges. They may not have time to be a competitive participant, but judging the contest will allow them to contribute to the contest with a lower time committment.
Some contests ask administrators to act as judges because they have a good understanding of Wikipedia rules and standards and are trusted by the community for their neutrality.
Having experienced Wikipdians or administrators act as judges can be a good way to integrate new editors into the community, because it gives them a chance to form relationships.^[1]

Content experts

Consider inviting partner organizations to help judge a competition. This can bring prestige to both the contest and to the partner.^[2]

Experts may be able to judge content quality better and more quickly than Wikipedian who does not have expertise in a particular subject area. Try to limit the time commitment for expert judges to 30 minutes or less.^[2]
Ask expert judges to add comments to the talk page of the article or give suggestions for improvements that editors can use after the contest has ended.
If you choose to use non-Wikipedian expert judges, you may need an experienced Wikipedian judge to review articles for quality issues relating to formatting, style and structure.

How to manage judging

Consider using a Google Form or a free survey tool like Survey Monkey to collect scores from judges. You can set judging criteria in the form or survey, and automatically collect the scores in a spreadsheet.
Some contests score in rounds, others score all at once. If you score in rounds, the first round of judging may be on the quality of the information and the second round scoring may be based on formatting and wiki-markup.
Not all judges need to reivew each article, especially in larger competitions.
Consider offering prizes for a variety of different achievements. This can make it easier for judges to come to consensus about who should win each prize.^[3]

Bots and self-scoring systems

Not all contests need judges. You can use simple self scoring systems and rely on the honor system, or the contest coordinator can review a random sample of scored submissions to make sure everything is OK.^[4]
You can use bots to score and choose a winner of the competition or use bots to score before human judges conduct a review.^[5]

Add YOUR judging tips!

General considerations

When to use

Examples

Three stages of judging involving both content experts and experience Wikipedians.
“In the first stage, I tried to have researcher with expertise in a particular field go over the content and give grades on how a) accurate b) complete and (c) give remarks, which go on the talk pages of the articles. This is very helpful because afterwards, if people want to improve a page they have specific feedback from a subject area expert. The contest participants do not see the grades, but the remarks go on the talk page. I took all the reports and gave them to 2 judges from the Wikipedia community who knew physics at least at the graduate level who review them and chose the three best articles. Finally, professors from university who sponsored the contest decided on the ranking of winners.” - Yoni

Bots + Humans
“There is a bot pulls together all the different submissions, judges make sure they meet the requirements to be included: sufficient length, “good” article, etc. Judges work continuously throughout the year, in each round to ensure that no one is gaming the system. One year there wasn’t a through review process for some of the “good” articles, was seen as partial judging; then the people who submitted those articles volunteered to not count them for that round of submissions.” - WikiCup

Related patterns

External links

References

↑ “All judges have been administrators. This is because they need to be able to distinguish quality content. Also people already trust them to be neutral and unbiased. It is also a good way for new users to integrate with the community.” – WikiWomen
↑ ^a ^b “For big contests, we ask partners who have expertise to act as judges. This brings prestige to contest and to the judges organization. We try to make sure that it is a small task for them to judge, no more than 10 or 20 minutes in a week, even just to confirm that the monument or image has the right name." - Alex
↑ “We do not provide guidelines to jury members, up to them how to score articles, maybe just tell them we want to use a scale of 5. Usually every jury member reads all articles and then talks about them and come to consensus on which contributors are best. Because give out prizes for different categories, that usually resolves any arguments around the “best” contribution. Sometimes can ask partner contributing prizes to provide another prize if two contributors are doing well.” - Estonia
↑ "The organizer for the contest must set up the page for the competition and kind of act as a judge. Sometimes we take samples going through the edits, but we also rely on the honor system because the community will notice if something was totally wrong." – Lars
↑ “There is a bot pulls together all the different submissions, judges make sure they meet the requirements to be included: sufficient length, “good” article, etc. Judges work continuously throughout the year, in each round to ensure that no one is gaming the system. One year there wasn’t a through review process for some of the “good” articles, was seen as partial judging; then the people who submitted those articles volunteered to not count them for that round of submissions.” – WikiCup

Endorsements

I like the suggestions here. I am planning to write a pattern on being a contest judge. Shyamal (talk) 04:10, 15 May 2019 (UTC)[reply]

PS - Learning_patterns/Being_a_judge

[1] “All judges have been administrators. This is because they need to be able to distinguish quality content. Also people already trust them to be neutral and unbiased. It is also a good way for new users to integrate with the community.” – WikiWomen

[:0-2] “For big contests, we ask partners who have expertise to act as judges. This brings prestige to contest and to the judges organization. We try to make sure that it is a small task for them to judge, no more than 10 or 20 minutes in a week, even just to confirm that the monument or image has the right name." - Alex

[3] “We do not provide guidelines to jury members, up to them how to score articles, maybe just tell them we want to use a scale of 5. Usually every jury member reads all articles and then talks about them and come to consensus on which contributors are best. Because give out prizes for different categories, that usually resolves any arguments around the “best” contribution. Sometimes can ask partner contributing prizes to provide another prize if two contributors are doing well.” - Estonia

[4] "The organizer for the contest must set up the page for the competition and kind of act as a judge. Sometimes we take samples going through the edits, but we also rely on the honor system because the community will notice if something was totally wrong." – Lars

[5] “There is a bot pulls together all the different submissions, judges make sure they meet the requirements to be included: sufficient length, “good” article, etc. Judges work continuously throughout the year, in each round to ensure that no one is gaming the system. One year there wasn’t a through review process for some of the “good” articles, was seen as partial judging; then the people who submitted those articles volunteered to not count them for that round of submissions.” – WikiCup

[1]

[2]

[3]

[4]

[5]