Wiki Education Foundation/Quarterly reviews/2015-Q3 outreach pilot

The following are notes from the Quarterly Review Meeting for the Wiki Education Foundation's Outreach Manager Samantha Erickson on April 8, 2015.

Present: LiAnna, Jami, Frank, Samantha, Eryk

Participating Remotely: Sage, Ian, Adam, Ryan

Please keep in mind that these minutes are mostly a rough transcript of what was said at the meeting, rather than a source of authoritative information. Consider referring to the presentation slides, blog posts, press releases, and other official material.

Slide 1: Outreach pilot

Wiki Education Foundation Quarterly Review for its Outreach Pilot.

Samantha: Welcome! This is my first quarterly review. I was hired in November and since that time we have been doing a lot of work to launch this project. But why are we bothering?

Slide 2: Why an outreach pilot?

Slide: New ways to get students to contribute content (...)

Samantha: Idea sparked by LiAnna and Frank – will students edit outside the classroom? Believed if we targeted high achieving students, we could find some Wikipedians in our midst!

Slide 3: Outreach and recruitment intro

Samantha: Let’s review some outreach and recruitment work I’ve been doing.

Slide 4/5

Photo of Uof A Geo Club at Kartchner Caverns State Park

Slide 6: By the numbers

Slide: Outreach from Dec-March 190 student clubs contacted. 30/190 responded (16%), 4/190 converted (2%).

Goal was 16 clubs, an 8.5% conversion rate.

Samantha: All outreach was done through cold emails. I believed when starting the project that if we wanted to get student clubs on board before February, I had to focus my energies on high numbers in order to convert.

Samantha: Ran A/B email test in Dec & early Jan. Long versus short emails. Saw by the end of January that only 1 person had replied to the B test (long email) while 18 others had replied to the A test (short email). Disbanded the B test email with student clubs.

Samantha: Visiting campuses over the holidays wasn’t an option. Most schools didn’t get into session before end of Jan.

Samantha: Set the initial goal of working with 16 student clubs over the term. Early milestone was to have all 16 confirmed by the end of January, but we just weren’t seeing the response rate we wanted. Extended the goal to February and expanded our outreach to include honor societies and continued communication with new and old student clubs…

Slide 7: Honor societies

Samantha: Once we realized we weren’t having a great response rate from student clubs, we decided to more formally pursue help from the honor societies they worked with.

Samantha: Wanted to have a significant proportion of our outreach to be aimed at honor societies explicitly – the original goal of this being the “high achieving students pilot”

Slide 8: By the numbers

Outreach Jan-March: 24 Honor Societies

Samantha: Here are some numbers...

• 25% of our overall student club outreach was to honor society chapters.
  • Of the 24 honor societies we reached out to, 11 responded and engaged with us.
• 25% of them reached out to their students on our behalf in some way. (Upsilon Pi Epsilon, Pi Alpha Xi, NCA, Beta Phi Mu, Sigma Gamma Epsilon, AIGA)
• 2/24 are considering hosting us at their national conventions over the summer (NSCS, Sigma Alpha)
• Total: 8/24 or 33% supported our project
• 2/24 actually got us in direct contact with a student club

Slide 9 & 10: Student clubs

Samantha: Swing the messaging away from our work and onto the work of students.

Slide 11: Content goals

Slide depicts a point system based on the “content improvement index.”

Samantha: Frank and I set some content goals to help compare the clubs even though they were working on different projects. Used the content improvement index on the office wiki, which I’ve tried to capture here.

Frank: This is based on the WikiCup; basically a copy of the German version.

Samantha: Each club had a goal of setting 180 points.

Slide 12/13: Student photo, U of A Geo club

Samantha: The University of Arizona GeoClub is a graduate student club studying geologic sciences in many forms. We have 13 students on the dashboard from GeoClub. I met them in person once for two day event and one workshop (one field trip).

Students contributed 69 points toward our 180.

Slide 14/15: Geo Club cont

Samantha: One of our students, for example, helped expand the page on Lanthanite (a mineral) as well as pages on Mixite, and Agardite. She also created the page for Kovdorskite. All of these pages now include significant improvements to referencing and sourcing, as well as a couple photos from their mineral lab on campus.

She has permission to post the school’s archive of mineral images as well.

Frank: What’s interesting about this group – and we really only have one very interested student here so far – but they are involved with the NASA rover, we weren’t allowed to take photos of the screens because they were actually controlling the rover on Mars from there! So I want to know how to activate more of those people from that club.

Samantha: All of these people are graduate students; S. is a PHd.

Samantha: It’s also a question of how to get more than one editor from these trips and outreach.

Slide 16: Berkeley Water Group Ideas Lab

Samantha: 11 students on the dashboard for the Berkeley Water Group Ideas Lab. I’ve been working with this group since Feb 23rd. Recently hosted a field trip with their club, that was co-attended by students from a class studying California Water. That class is now on our course page system.

Samantha: A few students have already started to upload photos from our field trip this weekend and a new page has been created for the delta cross channel facility we visited. Excited to see where their work goes! We met on Monday and had a workshop; in a few days we’ve had 48 points of article editing.

Slide 17/18: Student photo

Samantha: One student created the page for the Delta Cross Channel and has some photos; another uploaded the photo of the Pardee Dam on the Pardee Dam page, which was added to the article. There weren’t any photos of the dam on Commons. So this is a content gap we’re filling.

We also partnered with a classroom, which is now running an extra credit assignment on the dashboard. A few other students are telling other professors about it, as well.

Slide 19: Upcoming clubs

• May 7th, UCSB Art, Design, & Architecture Museum Club
• May 15th, Oregon State Pi Alpha Xi Horticulture Club
• Also that week, Oregon State Hydrophiles
• TBD, UC Davis BioTech club

Slide 20: Did not pursue

Climatepedia and Stanford CISAC -- did not fit an ideal test case scenario

Slide 21: Learnings

Learnings: You don’t know what you don’t know.

Slide 22: Recruitment

Slide: Timing is everything!

Samantha: Outreach in December we felt was problematic because of the holidays. Students were not on their emails. Some follow up I did with the honor societies showed that most create their year long student programs over the summer, so while our outreach in the Spring was appreciated, it was hard to implement due to demand on student schedules.

Jami: What is a good time to launch?

Samantha: We had a higher response in February, start of spring term. But that was still challenging as many groups are carrying out first-term/semester goals rather than looking for new ones.

Frank: So if they start new initiatives in Fall, maybe we want to test that question with other honor societies. If timing is everything we want to make sure we know the timing is right. So if fall is when things start, when do we reach out to get on the schedule for the fall?

Samantha: During summer planning meetings, generally, where they meet with their board and set goals. We want to get on those agendas, present at conferences and meet people in person. We’ve been invited to meet with two of them already.

LiAnna: I assume this is a general honors society, not specific to fields? Some of those don’t have a board or meetings, just maybe a local faculty member. It would be useful to see a breakout of different kinds of honors societies and how they handle their planning differently. If there’s a packet that goes out instead of a meeting, for example, we want to get into that packet.

Samantha: Part of what I’m hoping to do is get more research into how those honors societies work and how to get involved.

LiAnna: I think that there are a handful of very big honors societies, but there are more on individual levels -- the UCBerkeley honors societies might number up to 80 if there’s 80 subjects being taught, for example.

Samantha: I’m open to ideas if anyone here has them.

Jami: Have we discussed this before? Didn’t we work with someone at Berkeley on this already?

Samantha: Yes, but she had a hard time getting this out to all of the student groups we talked about reaching, not all of them are active…

LiAnna: We could have Jami or Eryk go meet with honors society reps at MPSA?

Jami: We could do that.

Samantha: I’ve also not really thought of differentiating the honors societies, so that could be helpful.

Samantha: Next…

Slide 23: Engagement

Slide: Excitement doesn’t equal action. Expectations setting… how much is too much? Focus on the story.

Samantha: Students who I had the chance to follow up with all seemed engaged and excited during our conversations. Every club I’ve talked to we have decided to pursue!

Samantha: Honor societies on the other hand, ALL seem to think this is a great project for students. However, very few were willing to get us in contact with students. I believe that they genuinely were excited, but that concerns about student privacy and also creating demands on students during the year were primary road blocks. It’s challenging to get access to student emails to contact them directly.

Samantha: Because these students have no incentive to participate except their own volition, I wanted to make the project as easy as possible. I found the content gaps, found some sources, helped plan the field trips. About 40 hours of week per club so far for the two that we’ve worked with. This is not scalable. But how much of this can we put onto the students? We will discuss more in the change ideas section.

LiAnna: I know that community service is a tracked, crucial aspect of honors societies at the national level. I had envisioned this program tapping into the community service aspect of the student’s club activities. By replacing the community service options they do that may be the same type of thing they’ve done before, which may not be super useful, this could be the community service portion…

Frank: I support that this is true; but at the same time, the learning we had [to test whether students would edit of their own volition] was important. An award structure, for example, becomes the incentive to edit in the classroom program.

LiAnna: I see honor societies as holding latent Wikipedians… who might get hooked on editing, and lead to higher retention. But we need to get them over the hump. So, giving an initial incentive to get started is an important part of bringing people into Wikipedia, and perhaps then once that is activated, they’ll stay.

Frank: And that’s the point of this pilot. I will say, we did not engage with enough students to draw meaningful results just yet. That’s why we should be careful about what we take away from this. But I feel that having one person in a group emerge? There may be different ways to do that. But we can take some things away.

LiAnna: I was in no way saying that what was happening here was worthy of drawing a conclusion from – I want to encourage experimentation, but I want to express that my own theories, the ones I wanted to test initially, still haven’t been tested by the current model, and I’d like to test that out.

Jami: So is the learning that we have to incentivize students? Or is there not really enough data? And… wasn’t our field trip offer a form of incentivization? (They have both done field trips).

Frank: We took students in Arizona on the hope that they would upload images after the trip; they were good, they had excellent tools that they could apply to Wikipedia. Then, after the field trip concluded, we had one editor who also happened to be the president of the club, the active person.

Jami: Couldn’t we get a group of student group presidents together and get them to edit Wikipedia?

Frank: We could!

Ian: What excited me was the two students who told professors what an interesting idea this was. It’s a lot of work to schedule a field trip, but if we refined it we may end up with a high-impact tool to get into the door at universities. As that kind of tool it’s more effective at outreach than conversion…

Jami: As a by product it’s great, but it’s not really the mission. If it were the mission, we could focus this and make it much more effective.

(Frank departs for a meeting)

Jami: Also, what amount of data would make it enough to draw conclusions from? How many groups should we try?

Samantha: 16 was the low number we wanted. I think we want to see what happens after 20 to 25 student clubs. But we may already be learning that students aren’t interested in editing in their extracurricular time. I don’t want to push into 8 honors societies and have it be overgrown to the stage that we can’t handle it.

LiAnna: From my experience, you overshoot your goals, and you use them as an A/B test. You don’t want to overpromise, but if you can frame your work as letting some groups work with different levels of engagement on our end and instead, say, a dedicated instructor… but then we can say what is sustainable, scalable, etc. Testing alternatives is a good problem to have. Some things will drop, some will continue.

Jami: There’s a LOT of hand-holding right now. Which is useful for determining what kind of tools to use in your toolkit. When we on-board for the classroom program we don’t go to the schools, for example, we have it set up for people to take it on their own.

Samantha: I agree. But I wonder how we can do that knowing only six clubs…

Jami: What happens if we do only have six clubs next term?

LiAnna: This is Frank’s call but if we only have six clubs next term then we have a clear indicator that this isn’t working. This is the first time that I have seen this semester framed as a test of the intrinsic interest in Wikipedia as a motivator; that’s a different test from what I had in mind.

Samantha: Many of the honors societies have said they won’t take an initiative from the national level…

Jami: Is that something Eryk could help with in terms of communication strategies? Because if we aren’t testing intrinsic motivators in the next semester, and we frame it as a community service, would that change the degree of commitment? That could be a key part of what happens that makes this work. I am volunteering Eryk for that…

Eryk: (Busily transcribing meeting)

LiAnna: We also need to know what clubs are doing as community service; what are they doing that doesn’t work so great with what the students are interested in? Look at the classroom program: We replace a term paper that doesn’t fit real needs, and we replace it with something interesting and new. So we need to know what clubs are doing as a community service project that doesn’t actually engage students that this can be dropped in to replace. The first step is to identify your audience and their motivations. Are honors society people bored reading about volunteer work that doesn’t have anything to do with the organization? How can we make the argument that Wikipedia is a better option?

Jami: A real question: are we pushing for outcomes on Wikipedia right now? We just said we think the summer is the time to get in and make the pitch to student honors clubs at the national level. Is that research a better use of your time than going to school groups right now? Obviously, that’s not going to bring content to Wikipedia, but it could set the foundation for you to have the real numbers you want, and need, to test these outcomes in the next semester. If you want 20 clubs next term this may be a better use of your time than meeting a student club and getting – not to be negative – four photographs on commons.

(General agreement)

Samantha: During workshops and with phone calls, we have changed our communication strategy with Eryk’s help… to focus on the story of why Wikipedia matters. This approach seems to make a difference, especially to students who reported in their workshop surveys.

Slide 24: Partnerships

Slide: Need Mission Alignment

Samantha: When finding partners to work with – both clubs, as well as field trip locations and local community partners – we need to focus on mission alignment.

Samantha: Kartchner Caverns State park was a location decided upon by myself and the students in Arizona before we had contacted the park. Had some trouble getting helpful feedback, getting photography permissions, etc.

Samantha: East Bay Municipal Utilities District (East Bay MUD) on the other hand was extremely helpful. Sent us a book, 35 learning brochures, and had a staff member volunteer their time to meet with our students. Excited about the idea of editing and improving community information!

Slide 25: Change ideas

Slide:

A new flavor of push: in-person outreach

Creating some pull: Editing campaign / initiatives, Individual Recruitment

Samantha: In the Spring, we did the majority of our outreach online. This was impersonal and felt a bit salesy. Instead of trying to sell our students our project, we want to create an opt-in methodology. What would this look like?

Samantha: In the summer and for preparation for the fall, I think we should focus our outreach in person. Attend the conferences. Meet with student body presidents and take students out for coffee – tell them why this is cool.

Samantha: Bring in the idea of editing campaigns – if we launch a water campaign, what would that look like? Create a sense of community across clubs, make them feel a part of something bigger (even outside wikipedia).

Samantha: Also incorporate individual recruitment around editing campaigns – do you have something to offer to this topic? Click here to join! The goal would be to work with more than just students.

It also sounds like one thing we should be working on is getting groups to create initiative in these chapters.

Slide 27: Feedback

What are our opportunities?

What opportunities are we missing?

Adam: I’m not super sure about using the WikiCup incentive structure…

LiAnna: This was my suggestion originally, and how it’s being used here isn’t exactly what I’d hoped to see, but I wanted to use the WikiCup as a rough starting tool for measuring things. I’d like to focus on readership too.

Jami: The point system doesn’t even include sources; it’s a bit unrealistic to set goals based on featured article status…

LiAnna: For example, one student put in infoboxes, which is really elaborate and difficult, and didn’t receive any points for it. So… the WikiCup is just a short-term proof-of-concept that contributions can be measurable. It’s not intended to stay in the form it’s being used now.

Jami: I want to emphasize that, setting this up and evaluating it from a students perspective can be very helpful. Not just a Wikipedian’s perspective. Students have access to sources for example, let’s give them points for sources. Let’s leverage the things we know about how people use Wikipedia and how students can contribute to that.

Samantha: Sure, and I can re-build the metric and re-apply it to what’s been done to give us a better picture of what people are doing already (and continue to do).

Jami: In terms of campaigns, I embrace this idea of an editor campaign. I wonder how effective it would be geared toward students, rather than toward institutions? So many people I spoke to had heard of the art and feminism edit-a-thon, and librarians have listservs, connections, etc, whereas students can be unreachable. If we think about campaigns, we can test this…

Samantha: We’re looking at people at various universities who are leaders in water issues, and we’re going to talk to them about how to get people to participate in this kind of research. I don’t think it’s feasible for one person to be doing this campaign as half time.

LiAnna: I do want to flag that water is actually contentious…

Samantha: So is feminism?

LiAnna: Yes but the goal is to broaden the scope of topic, and get more information about women onto Wikipedia. That’s a fine line, but with water, the line between information and advocacy is thin. Some may say one side needs to cut water and one side doesn’t, some might have an agenda or funding conflicts tied to various positions on water conservation, for example. So you have to be very careful about how we go about this.

Samantha: Well, right now, we’re just talking about how some of these academic experts might see a campaign about water; we don’t have any plans on launching something right now.

LiAnna: So it’s important to look at content gaps, what kinds of conflicts there are, how many purely neutral sources there really are out there. And the question then is, how relevant is “ideas on how to conserve water” to Wikipedia at the end of the day?

Samantha: But there could be an entry on water conservation methods?

LiAnna: With art and feminism edit-a-thons, there’s not a big concern that adding a woman artist to Wikipedia doesn’t align with Wikipedia’s goals.

Renee: If we’re looking at this as outreach for the classroom program, shouldn’t the classroom program be tackling this?

Jami: You mean reaching out to our instructors to get them into the student groups pilot?

Renee: I wonder if that outreach could do two jobs at once … promoting both the classroom and student groups outreach? Going through a great instructor and saying “hey, we do other stuff too…?”

LiAnna: Well, the top-down model usually doesn’t really work anyway. We never want to do the top-down approach, which is a concern with tying an advisor to a club and then the club president is doing it half-heartedly because the advisor tells them to do it, there’s not going to be real enthusiasm for it.

LiAnna: A good question for the next round too is, how much influence to advisors really have? I don’t think many clubs have strong ties to their advisors.

LiAnna: As a final comment, I just want to say that the numbers on your slides, for response rates? Those are actually really good numbers and I think you should be proud of that.

Samantha: Thanks!