Research:Teahouse/Pilot plan

Pilot

The Teahouse project is being piloted as a WMF Community Fellowship project in the first half of 2012 on EN:WP at WP:Teahouse. This pilot is intended as an experiment or proof-of-concept, and the outcome should demonstrate whether or not such an approach adds value to Wikipedia and the new-user experience by improving retention of good-faith new editors. If the project demonstrates success and the community finds the model to be of value, we hope volunteers will continue to maintain and improve the Teahouse on English Wikipedia. We also welcome partnership with members of other language projects to take the concept beyond English Wikipedia.

Targets

• At least 20 hosts active throughout the pilot.
• 75 new editors utilizing the Teahouse weekly.
• 50 to 100 new editors invited to participate in the Teahouse daily.
• A measurable increase in retention of new editors should be apparent via an increase in editing sessions of participants vs the control group. See below for a full list of comparative metrics being tracked.

Users

Hosts

In order to keep the space lively and active, the Teahouse is actively staffed by experienced Wikipedians called hosts. The role of these volunteer Hosts is to invite and welcome new editors to the Teahouse, answer questions, offer editing advice or support, and facilitate a peer-support environment.

Learn more about Teahouse hosts, and becoming one, at WP:Teahouse/Your hosts

New editors

All good faith new editors are welcome to visit the Teahouse. The focus for outreach efforts that invite new editors to participate in the Teahouse began with good faith editors who have already completed several edits, but are still in their first week of editing. Promising new editors are invited to the Teahouse through a few types of active outreach:

1) Partnerships with existing off-wiki outreach programs, meetups, and edit-a-thons (Global Education, GLAM, Girl Geeks, Chapter events, other women's editing events, etc)

2) Daily database queries for:

• users who submit feedback dashboard messages
• newly autoconfirmed users, or first day editors with >10 edits

3) As Teahouse gains confidence in its capacity to serve a large number of new editors, hosts and other Wikipedians are reaching out to good-faith new editors in a growing number of settings to point them to the Teahouse.

Note: DB Queries are intended to be scanned by humans to filter out vandals, as the intention is to invite good-faith new editors.

Features and user scenarios

Features

These are the main elements and requirements of the Teahouse space.

Q&A board

• user should see that people are asking questions when they first arrive at the Teahouse and is able to browse/post/reply to questions
• clear call to action to post a new question (big button to create new section ) - avoid "forcing" new users to have to search an archive first, like at help desk.
• easy to find most common newbie questions (near the top, clear headings, relevant topics)
• easy to reply to existing question
• edit screen should be free of as much code as possible.

Introductions space

• ability to see people (highlighting featured participants, image+text)
• ability to introduce self (user sees big button call to action, adds text)

Personalized outreach materials

• personalizable invitation template (with strong brand ID, colorful and friendly)
• personalizable thank you template (to be used by hosts to thank new visitors for stopping by)

Guide to preferences setup

• call to action link "how to setup your preferences", with small pic of email pref setup, and link to fuller guide pages.
• purpose is to encourage new user to enable email for notification purposes.

Hosts project coordination

• linked from main teahouse page.
• project space for hosts to organize their work (e.g. track tasks like recruiting, welcoming and answering questions), communicate with one another, and train new hosts.

List of potential invitees

• database report that refreshes daily.
• query of "promising candidates" (i.e. new users with 10+ edits in 1st day, new users who submit moodbar feedback) who should be invited to participate in Teahouse.

Automated notifications

• bot informs questioner on talk page of response, using a template that contains a link back to the question thread.

IRC channel

• in-browser, freenode, setup pre-launch, the idea is to link from the main page but will not necessarily do so for launch, until have need and ability to staff (see nice to have section below)

Other nice to have features (not planned for launch)

• Space to request peer review
• Bulletin board for opportunities to get involved in other projects, join lists, etc
• Freenode IRC chat space for hosted events, etc Done
• User box (linking to teahouse) Done
• Barnstar (awarding hosts or active peer supporters) Done
• Resources (links to existing help pages/resources, etc)

User scenarios

We have developed a set of user scenarios and use cases that describe some of the users we expect to benefit most from the Teahouse, and some common situations in which the Teahouse could support them.

Scenarios are divided into the following categories:

• Negotiating People, Policy and Process
• Finding Collaborators and Ways to Contribute
• Requesting Technical Help
• Online Followup of Off-Wiki Outreach

Project participants

Tea on the verandah, 1900-1910

• Sarah Stierch
• Jonathan Morgan
• Siko Bouterse
• Heather Walls

Timeline

Phase 1 – Planning, feedback, research (Dec/Jan)

Deliverable: Project plan complete. Assemble the necessary info, team, and buy-in to run with the project. Define participants, targets, and metrics for success.

Deadline: January 15 2012

Phase 2 – Design and build the space (Jan/Feb)

Deliverable: First version of the Cafe functional on EN:WP

Deadline: February 3 2012

Phase 3 – Outreach and set up hosts (Jan)

Deliverable: 20 Hosts trained and ready to help new user

Deadline: February 20 2012

Phase 4 - Launch (Feb, March, April)

Deliverable: Invite at least 75 promising new editors per week and turn them into engaged Wikipedians

Deadline: February 27 2012 (launch)

Phase 5 - Measurement and reporting (May)

Deliverable: Report on short-term outcomes measured against targets, plan for next phase of project (iterate, scale) if successful.

Deadline: May 15, 2012

Metrics

During the course of the pilot we’ll be evaluating success by collecting the following metrics.

As the pilot progresses, reports on Teahouse metrics can be found at Research:Teahouse/Metrics

Teahouse activity metrics

Quantitative

• invites sent per day/week
• visitors per day/week
• visits per week per visitor
• active conversations per day (measured via page views and unique edits)
• count/average of new editors interacted with per host per week

Qualitative

• examples of direct mentorship/coaching
• participant feedback on the Teahouse experience via surveys and/or interviews

Editor outcomes metrics

Comparative

longitudinal assessments of new users who participate in Teahouse in comparison to a roughly equivalent control cohort of new users who do not participate will include:

• edit count (broken down by namespace)
• bytes added
• logins (or editing sessions, however this is stored) per day/week/month
• logins since day 7, 14, 21, 30, 60, 90, and at 6 months and 1 year
• user rights gained
• blocks
• reverted edits
• articles created
• created articles deleted
• instances of collaborations with peers (editing on talk pages, articles, etc)
• warning templates posted per user page