Community Tech/Event Metrics/Archive

August 9, 2018: event tool feature ideas (part 1)—what do you think?

As described below, the decision was made earlier to approach this initiative through a series of interconnected tools that address the needs of different types of program organizers. These new tools will complement existing tools like Grant Metrics.

The first tool under consideration—working title Event Tool (ET)—is designed to help people organize and run wiki events. Over the past few weeks I’ve been interviewing editathon organizers about their process and problems. The sections below look at the parts of their workflow I think new tools might improve. For each, I describe the process as it is now and propose some ideas for addressing the problems associated with it. Just as importantly, in each section you'll find a list of questions. If you’re an event organizer, please use the talk page to help guide our next steps. We’re listening. —JMatazzoni (WMF)

Assumptions and caveats:

• Metrics a focus: In line with the  Wishlist proposal, easier and better data reporting about events is a core goal of this tool. I’ve described some efficiency and convenience features below as well.  But my expectation is that metrics will be the focus of initial product releases—unless event organizers tell us they have a different priority.
  • That said, this post doesn’t cover new metrics and reporting features per se; it covers the event workflow only from event creation through the day of the event.  Most of the ideas described below, however, set the stage for better metrics by providing new or improved ways for getting data into the system. Look for a post about data reporting in coming weeks.
  • Grant Metrics  will probably handle most or all data reporting for this tool. In fact, it’s not clear whether what we're presently calling the Event Tool (ET) will be a distinct tool in itself or simply a series of enhancements to Grant Metrics.
• For organizers only:  The features described below are for organizers. With the exception of the Signup page, event participants will not see or use the ET.
• Beyond editathons: My initial research has been with editathon organizers but we hope the functionality we create will be useful for organizers of other events as well—such as content drives. If you organize different types of wiki events, please let us know what types, whether the tools described would help you or not, and what we’d need to change to address your needs.

Step 1: Organizer creates an event in the system (and Event page on wiki)

How it works now

• Organizers manually build an Event Page on wiki. Some also enter events into the Dashboard, primarily in order to get metrics later.

Problems with current process

• Effort duplication / manual copy and paste of data: E.g, copying participants’ names from a signup database (on EventBrite, say) to both the wiki page and to the metrics system. Or copying a list of articles from the Event Page to a metrics system.
• Page creation is complex: Event pages can be difficult and time consuming to produce, which means organizers can’t delegate the process.
• The Dashboard and Grant Metrics don’t accommodate all desired data types: E.g., organizers would like to report on all events with a certain partner or to limit reports to a list of suggested articles (the “worklist”).

Proposed solutions/feature ideas

• Form-based data entry into a central event database: By entering event data into an Event database, we can achieve create-once-publish-many efficiencies. The Event Creator form can include new data types that organizers want but don't currently have in Grant Metrics.
• Automated wiki page creation: Based on the event info, the  ET could generate selected sections of a  wiki Events page (or perhaps a complete but generic Event page).  
• Automated wiki page updating: The ET might continue to update the automatically generated sections of the Event page. Organizers would still be able to customize the automatic wiki page by adding sections not under ET control.

Questions for organizers -- Post answers here!

• What types of information would you like to be able to report on?
• How important a feature is automatic wiki page creation and updating to you?
  • Are you willing to accept a new and perhaps more standardized page design to get auto  page creation?
  • What parts of the Event page change the most after creation so would benefit from automatic updating (e.g., the worklist?)? Is updating important to you?
• If participants’ usernames were in a database you could consult any time, would you still need to publish them on the Event page? Why?

Step 2: Participant sign-up

How it works now

• Organizers want participants to RSVP for event-planning purposes. They also need to put user data in their systems for metrics and outreach purposes. The data collected and the signup mechanisms vary considerably.

Problems with current process

• Data has to be transferred manually: E.g., by copying and pasting from EventBrite or a Google Doc to the wiki Event page, a MailChimp mailing list, the Dashboard for metrics…..
• No email opt-in: Third party sites like EventBrite provide participants no opportunity to opt in for future event notification from the organizer.
• Third-party services may not match Wikimedia privacy policies: EventBrite, for example, has its own privacy and security policies.

Proposed solutions/feature ideas

• Standard participant signup form: Based on event data organizers add to the system, the ET can generate a standard signup form to acquire basic user data—name, email, username. The form will not be on wiki; participants will find it by following a link from the Event page. Organizers will  be able to export user data—e.g., to give to partner organizations.
• Customizable form-setup facility: To accommodate the different types of user information organizers want to collect and track, we could provide limited flexibility in form configuration. E.g., organizers may be able to choose from a predetermined set of questions/elements, and to specify which are required or optional.
  • The form can’t include fully-configurable free-text elements, since organizers might use them to acquire data our privacy policies don’t allow the Foundation to store.  

Questions for organizers -- Post answers here!

• If we offer only a standard signup form (at first), would that work for you? What would a standard form need to include?
• If we can offer a tool that lets you select from a predefined set of signup form options, what would you need it to offer (e.g., questions about gender or age, a requirement to approve a safe-space policy...)? Be selective: the simpler this tool is, the more likely it is to get built.
  • In particular, should a checkbox for email-list opt in be standard or optional?

Step 3: Wiki account creation

How it works now

• Most organizers don’t require users to create a wiki account in advance, fearing it will discourage signups. This means that newbies often need to register on the day of. But for security reasons, the wikis allow only a limited number of accounts to be created from one IP during a given timeframe. This creates problems.

Problems with current process

• Account creation wastes time during the event: Organizers can apply for a the Event Organizer right, which lets them create more than the allowed number of accounts from one IP. But the person with the Event Organizer right has to personally register the participants, which can cause significant delays at the start of events.
  • Even if everyone coming to an event had pre-registered for the wikis, on English Wikipedia and some others new users need to have Autoconfirmed status in order to create new articles.
• Participants sometimes get blocked: This happens even when event organizers have played by all the rules.
• Restricted choice of event leaders: The Event Organizer right can’t be delegated, so the person with that authority needs to be on hand at events personally to register users.

Proposed solutions/feature ideas

• Promote and facilitate wiki registration during event signup: We may be able to integrate wiki registration directly into event signup page, creating a seamless process and ensuring that fewer attendees show up on the day of without usernames. If we can’t embed registration, we can create a signup flow that makes registration as attractive as possible.
• Bulk account creation tool: No matter how attractive we make advanced registration, there will always be drop-ins attendees on the day of. To help Event Creators move more quickly, we could make a tool that would let them register multiple participants at one time.
• Post messages to newbies’ user pages: To shield newly registered participants  from patrollers, we could post notices upon signup to their user pages announcing that they are part of an event and directing questions to the the event organizer.

Feedback from organizers -- Post feedback here!

Step 4: Participants check-in (day of)

How it works now

• Organizers want to know and document who actually came to the event. They may need to send this info to a local chapter or partner organization, and it can be useful later for when generating reports. Mechanisms for checkin vary widely. Some ask people to sign in on wiki. Some use a paper sheet. Others simply count anyone who edited certain pages during a given time as a participant.

Problems with current process

• Manual data entry: E.g., of usernames into metrics systems.
• Inaccurate metrics: Making assumptions about who attended can be misleading.
• Lack of participant data: By putting attendance records into the database, organizers will be able to know, for example,  that certain users attended multiple events.

Proposed solutions/feature ideas

• Class Roster with check-in facility for organizer: A class Roster for organizers would let organizers check in people who show up. The Roster should also let organizers add usernames to the records of users who signed up but didn’t register for the wiki in advance.
  • To help organizers monitor activity during the event, the Roster might include links to all attendees Contributions and Talk pages.

Feedback from organizers -- Post feedback here!

---

Reaching out to people who voted for this wishlist item:

This project—inspired by the Wishlist item you voted for—is in the research and design phase now. Input from event organizers is essential! Please read the post above and watch this page if you have a continuing interest.

@Joalpe:,  @Ijon:,  @Krishna Chaitanya Velaga:, @Discott:, @David1010:,  @Liuxinyu970226:,  @Jc86035:,  @Megs:, @Thomas Obermair 4:, @Shizhao:, @Pbsouthwood:, @Bspf:,  @Donald Trung:, @Wikinade:, @Nick Moyes:, @Zhangj1079:, @OrsolyaVirág:,  @B20180:,  @Exilexi:,  @Rachel Helps (BYU):, @Mickey83:,  @Becksguy:, @VIGNERON:, @Ozzie10aaaa:, @Superchilum:, @Romaine:,  @Theklan:,  @JenOttawa:,  @Ɱ:,  @Ckoerner:, @Arian:, @Kerry Raymond:, @Doc James:, @Muhraz:, @Hexatekin:, @Pamputt:, @Satdeep Gill:,  @NickK:, @Townie:, @Martin Urbanec:, @Vachovec1:, @Battleofalma: —JMatazzoni (WMF) (talk) 18:36, 10 August 2018 (UTC)

@YjM:, @Theredproject:,  @Mckensiemack:, @Hasive:, @Nattes à chat:,  @Winged Blades of Godric:, @Meredithdrum:, @Artchivist1:, @Paucabot:, @FULBERT:,  @Rhododendrites:, @JaxieDad:, @PointsofNoReturn:, @Annamariandrea:, @Halibutt:, @Masssly:, @Voltaireloving:, @Sarahobender:, @Davidpar:, @HugoHelp:, @Vallue:, @Medol:, @Tiputini:,  @Monikasj:, @Stinglehammer:, @Caorongjin:, @John Cummings:, @Fixer88: —JMatazzoni (WMF) (talk) 18:51, 10 August 2018 (UTC)

@Spiritia:, @Epìdosis:, @LornaMCampbell:, @Mtmlan84:, @Ouvrard:, @Mauricio V. Genta:, @Anthere:, @Kvardek du:, @Jorid Martinsen (WMNO):, @Uncommon fritillary:, @African Hope:, @Astrid Carlsen (WMNO):, @Richard Nevell (WMUK):, @Alangi Derick:, @Talueses:, @Wittylama:, @VMasrour (WMF):, @TrMendenhall:, @Ecritures:, @MichaelMaggs:,  @Pharos:, @Heathart:, @John Andersson (WMSE):, @Haxpett:, @Psychoslave:, @Anne-LaureM:, @Sylvain WMFr:, @Yohannvt:, @Lirazelf:, @ArtEmJJ:, @Jack who built the house: —JMatazzoni (WMF) (talk) 19:01, 10 August 2018 (UTC)

@Chicovenancio:, @جواد:, @Xavier Dengra:, @לסטר:, @Esh77:, @Bijay chaurasia:  @Shangkuanlc: —JMatazzoni (WMF) (talk) 19:41, 10 August 2018 (UTC)

@Jmatazzoni: —JMatazzoni (WMF) (talk) 16:08, 13 August 2018 (UTC)

July 17, 2018: Are you an editathon organizer? Let’s talk about better tools!

I’m Joe, a product manager at the Wikimedia Foundation, and I just inherited this project from Trevor (thanks for your help T!), As a first step I’m reaching out to people who organize editathons so I can learn more about your workflow and needs. I’d like to better understand what works for you and what doesn’t about your current tools and what ideas you have for tools that could make you more effective. I want to make sure I speak with organizers from countries other than the USA, though I need to conduct interviews in English.

If you’re an experienced organizer who could benefit from better technology for metrics, signups, event promotion or other parts of the editathon process, I’d I’d like to set up a time when we might chat via video conference (Google Hangouts). To respond, please:

• Send me a contact email where I can reach you
• Please tell me briefly how many and what type of editathons you’ve worked on and
• Let me know what time zone you’re in (I’m in California time).

You can reach me on my talk page or here: jmatazzoni[at]wikimedia.org. I'm researching this right now, so please don't wait to get in touch. If you have ideas but don’t have time to talk, please talk to me on the project talk page. I’m looking forward to learning more about your work! —JMatazzoni (WMF) (talk) 17:52, 17 July 2018 (UTC)

April 25, 2018

Sorry for the delay — a lot of progress has been made and I am looking forward to providing everyone with more information. A lot of this is duplicative of what I presented at Wikimedia Conference in Berlin last weekend, where I connected with an incredibly helpful handful of program and event organizers. The slides can be found here. The input and experiences were valuable and informative to my work on this project.

Survey

We wrapped the survey (see below) a few weeks ago. It received 31 responses from 27 different affiliates or organizations — thank you if you participated!

I'm working on getting these responses into a form appropriate to share on-wiki (per the WMF's survey policy.) In the open-ended survey responses, I've manually identified and documented 83 different times people described how they use the dashboard, 35 pieces of praise, 17 ideas, and 111 pieces of frustration. Some of these uses, praise, and frustrations are duplicative (which is great!) but there is a long list of one-off comments. The most common frustrations are about the limitations of the data available and the lack of connection to the wiki itself. The most common praise was about the Dashboard's ease of use.

Project scope

After reviewing the responses from the survey and discussing the bandwidth and strengths of the Community Tech team we have decided on a project direction. I've renamed this page and the associated project to "Tools for program and event organizers" because we are moving forward with this Wishlist project in a different manner than simply making changes to the dashboard.

Rather, our current project direction is to build tools that complement the existing Outreach Dashboard. We believe there is room for granular, tactical tools that address different use cases than the dashboard currently serves. There is a large amount of diversity among different programs from Edit-a-thons to GLAM content donation to 'Wiki Loves' type content drives. We believe we'll be able to deliver more value by building niche tools than expanding the dashboard. Additionally, a dashboard is only one type of tool and we will likely build out other types of tools. Some of these tools will likely work more closely on wiki, which isn't something the dashboard can do. In the end, we want to have a suite of interconnected tools that are useful to a wide audience of program and event organizers.

Our primary goal is to build highly useful functionality for concrete, defined target audiences (e.g. organizers of edit-a-thons, organizers of content drives, education organizers, wikimedians-in-residency, etc.) Many of these uses are somewhat available in the existing dashboard, when they are unique and could benefit from individual attention.

Next steps

The work falls on me (Trevor) as product manager to enumerate the potential target audiences and the pros/cons for focussing on each one. I'll then identify the most common use cases for these audiences. (Alex has a very strong suspicion that the two major areas will be cohort wrangling & data reporting, which I agree.) Once we all (WMF, and all project stakeholders here on wiki!) agree on a prioritized list, we'll start defining the specifics for each tool, including design so usability is a forethought of this project, not an afterthought.

As this "suite of tools" project crystalizes we will answer a lot of the questions swirling at the moment: How are campaigns represented in the tools? How do we (if at all) take advantage of our recent work on Grant Metrics? How does this relate to the hashtag tool or the worklist tool? etc. I'll be updating this project page as things progress and decisions are made.

It's looking like product definition will still take a few months, but I anticipate for us to be ready for development by August of this year, with something demonstrably by December. (Scope TBD 🙃)

Thanks for reading, please let me know if anything is unclear or if y'all have any questions! 🤠

March 15, 2018

The survey has been submitted to WMF legal for review of the survey policy.

I met with Sati, who gave me some good advice. She has documented interviews with grantees that may be useful and will help review our findings from the survey. She also suggested we think about Proactive vs. Retrospectively dashboard use (e.g. using the dashboard as you prepare and run an event, vs only inserting data after the event has ended.)

Also, we may want to build a full roadmap, even if we can’t commit to development of it. We'll need to be clear about our scope and commitments.

March 9, 2018

We have completed two in-person interviews over Google Hangout, and will be moving to an online questionnaire for program and event organizers. Our goal is to send this out next week (week of March 12) and have publish an anonymized summary the week of March 26. This input, along with feedback inside the dashboard itself and comments in the Wishlist proposal will help us determine which problems this project will solve. The target deadline for publishing these problems will be March 29, 2018.

The questions for the online questionnaire will be:

Basic/Demographic questions:

1. What is your username?
2. What is your background outside the free knowledge movement?
3. How long have you been involved with Wikimedia programs or events?
4. Which of the following are you most actively involved in organizing or facilitating?
   • Edit-a-thons
   • Editing Workshops
   • Conference
   • Wikipedia Education Program
   • GLAM Content Donation
   • Photo Drive or Event
   • On-wiki Writing Contest
   • Wikipedian in Residence
   • Hack-a-thon
   • Other Partnerships
   • Research Project
   • Tool Building Project
   • None
   • Other (Specify)
5. Please briefly describe this work:

Dashboard use:

1. How do you use the Dashboard to organize and communicate before a program or event?
2. How do you use the Dashboard to organize and facilitate during a program or event?
3. How do you use the Dashboard to organize and perform evaluation after a program or event?

Why the dashboard:

1. What do you think are the greatest benefits of the Dashboard?
2. What are your greatest frustrations with the Dashboard?
3. What can you not do because of poor functioning of the Dashboard?
4. What other tools/techniques do you use in addition to the dashboard? For what purposes?
5. Any other comments

February 18, 2018

We have completed two user interviews, and are awaiting a third. We will soon be moving to email-based questions to better manage our time. We will anonymize and summarize our findings here on wiki.

I hope to have identified some common problems we can address (#Problems to solve) by the end of March 2018.

We will also be speaking about the Dashboard and our work to improve it at the Wikimedia Conference 2018 in Berlin on April 20-22.

January 29, 2018

We would like to conduct 3-5 interviews with program organizers to hear first-hand about the dashboard. The script for the 60-minute interviews is:

Basic/Demographic questions:

1. What is your username?
2. What is your background outside the movement?
3. Which Wikimedia projects have you contributed to? What experience do you have?
4. Have you supported work with an Affiliate?
   1. What kinds of programs do you organize?
5. Which of these programs have you used for the Programs and Events Dashboard?
6. How long and/or often do you use the Dashboard?
7. Have you ever used project management software in other contexts? Are you familiar with classroom management tools in the educational context?  If yes explain:
   1. What was it? What worked well for you?
   2. Why?

Why the dashboard:

1. Event stages
   1. How do you do communications and organizing before an program or event? Does it include the dashboard? Can you show me an example?
   2. How do you do communications and organizing during an program or event? Does it include the dashboard? Can you show me an example?
   3. How do you do communications, organizing and evaluation after a program or event? Does it include the dashboard? Can you show me an example?
2. Why did you choose to use the programs and events dashboard for those programs instead of for other programs?
3. What do you think are the greatest benefits of the dashboard?
4. What do you think are the greatest challenges?
5. Would you recommend the dashboard to affiliates why?
6. Which features of the dashboard affect your ability to organize with an affiliate or campaign?
7. What can you not do because of poor functioning of the dashboard or an alternative tool?
8. What other tools/techniques do you use in addition to the dashboard? For what purposes?
9. What are your favorite metrics or tracking tools for other Wikimedia programs? Why?

January 17, 2018

Common workflows for Program/Event organizers:

• Organizers can create different levels of structure: Campaign > Programs (aka Events)
• Entry points to create Programs are not 100% intuitive, there’s room for usability improvements
• ‘When’ and some other data fields are not available when creating an event, only editing an event, there’s room for usability improvements
• Sometimes Volunteer activity (usually for Power Editors) will get into the system and muddle the data. This can be addressed by having the users sign up for specific articles so their other edits do not display for the program
  • E.g. Apples attends a ‘Books’ edit-a-thon but previously in the day they edit about Politics — only the edits about Books should show
• Goal of Edit-a-thons that the Metrics in the Dashboard doesn’t currently display: Walk away with enough skills to be useful contributors someday
• Attendees can provide feedback in the dashboard, but at in-person events this is done verbally
• Participants can add article, see list of articles, see what others are doing, see some data
• At the end of the event, organizers can…
  • Download data in a CSV file (one click download)
  • Only shows Wikipedia, Commons, and (some) Wikidata. Should show more of the projects to be fully useful
  • No visual display of the data, just numbers
  • What type of edits were made? Quality of these edits?
  • Metrics from Programs are rolled up into Campaigns
  • Is there a download for Campaigns data? TBD.
• Training modules may be buggy, there is no assessment (quizzes, completion data) and are not required

January 12, 2018

Based on initial conversations and reading, I've learned that programs are any type of time-bound organized contribution efforts, usually within a defined topic. The types of programs vary greatly in size, type, and complexity but the most common types are in-the-classroom education, Wikipedian-in-residence, edit-a-thons, and photo/edit/data drives. The Dashboard is used during three stages of program organization: planning, event logistics, and follow-up but excels at event management logistics and follow-up metrics gathering. It is user friendly (doesn’t require experience with wiki templates and markup) and is reliable for repeat organizers.

The biggest deficiencies in the Dashboard seem to be that it is built as a one-size-fits most but different groups want/need it to do things specific for their individual programs. I'm not sure if there’s anything significantly bad/wrong with the existing tool, but I expect there are some odd workarounds that users have developed which we could address. We’ll likely receive many specific yet varied requests from Education program organizers, and will also receive lots of requests for improving the Dashboard’s data/metrics reporting. We’ll probably hear some opposing pieces of feedback from organizers given the diverse uses, but that’s par for the course.

Next steps for this project: Alex will walk Trevor through the most common workflows for organizers (on wiki, the dashboard, and whatever other tools they use) when they start a new program, run their first event, and follow-up after the event/program has ended. From this I’ll create a user interview script, which I will use during calls with several real-world program organizers. I’ll hope to learn what parts of the workflow fail them, what they want to keep about the Dashboard, and what functionality is completely absent.

December 20, 2017

In an initial meeting with the Community Programs team, we discussed what this project will look like. We agreed to start with a user research approach that will help us evaluate and prioritize the use cases we need to support.

We discussed the existing uses of the dashboard — metrics gathering, event management, training, and multi-event (campaign) tracking.

We discussed that some of the risks for this project will be Design (Community Tech will have more visual design resources in 2018), community ripple effects and fatigue due to new tool adoption.

December 13, 2017

The team will start investigating this project early in 2018. If you've got suggestions or questions, please write your thoughts on the talk page!