Wiki Education Foundation/Quarterly reviews/2014-Q1 Classroom Program & Educational Partnerships

The following are notes from the Quarterly Review Meeting for the Wiki Education Foundation's Educational Partnerships Manager Jami Mathewson on August 29, 2014.

Slides from the Quarterly Review.

Jami Mathewson at the Classroom Program Quarterly Review.

Present: Jami Mathewson, LiAnna Davis, Sara Crouse, Frank Schulenburg
Participating remotely: Sage Ross

Please keep in mind that these minutes are mostly a rough transcript of what was said at the meeting, rather than a source of authoritative information. Consider referring to the presentation slides, blog posts, press releases, and other official material.

Introduction

This is Jami’s first quarterly review so will speak about the motivations for decisions that have been made to get to where we are now with the classroom program – the centerpiece to Wiki Ed's work thus far – and about the direction of the program for fall 2013 and beyond.

Slide 1 – Overview

• Where we’ve been: November 2013 and spring 2014
• Successes
• A term in the life of
• Looking forward—separating the two roles
• Impact of partnerships (why important/positive steps)
• Wiki Ed’s needs

Slide 2 Since spring 2012, we decided to maintain programmatic level around 65-70 courses

• Ensure the appropriate amount of support for each course and student editor
• Do not overwhelm the Wikipedia editing community if we could not provide additional support

Frank: what were the drivers for the “organic growth” [vs. institutionalization approach] of the program?
Jami: the program primarily grew through word of mouth, and some growth may come from PR from previous terms. [During recent terms], it hasn’t been so much about growth [in numbers], but about connecting with classes more. Also, we have gotten a better reputation with the Wikipedia editing community, so a lot of editors are coming to the noticeboard and identifying opportunities/classes/instructors that have joined the program.
LiAnna: before and at the beginning of the Wikipedia Education Program, there were instructors already teaching with Wikipedia; now many of those instructors [within the scope of our program/geographic region] are doing so via/with the support of our program rather than individually. Now there is recognition [in both the academic and Wikipedia] community too, so that classes/instructors are referred to us.
Jami: People started to realize that instructors are teaching with Wikipedia anyway, so we want to make sure they have the resources and are doing it in the right/efficient way.
Frank: also, instructors are becoming more aligned with us as they [and/or their peers] recognize the value in having the support of both the Wiki Education Foundation and volunteers/Ambassadors.
Jami: [Regarding other drivers for growth,] anecdotally, I used to send some outreach emails to universities without much response but now that we have institutional credibility (website, programmatic reputation, funded organization), people are more receptive.

Slide 3 – Spring 2014 Goals

• Incidents: resolve 90% of Wiki Ed-related incidents within 5 days. (result: ZERO incidents)
• Highlight program successes: 4 new blog posts about instructor and student experiences (result: 20 blog posts; 4 profiles)

Frank: “Incidents” were a pain point for several terms [when Wikipedia community members took issue with the (then) new classroom program].
Jami: for context [an example of an incident in earlier days]; a community member would take issue with one student’s work (out of 1800 students) and post to the noticeboard that the incident was representative of the entire program. Frank wanted to proactively work to resolve such matters; therefore, the goal was not to minimize number of incidents, but to step in and try to resolve them within five days.
Sara: how do you qualify or define “incident”?
Jami: essentially, it was/is considered an incident when someone complains about student editor via noticeboard or other list; and [the time to resolution of issue is measured] from moment of posting to moment that discussion ends.
Jami: looking back, I might have set higher goals for incident resolution; moving forward, the goals are a little more detailed

Slide 4 – Spring 2014 wrap-up

• 65 courses (42 or 65% taught by returning instructors)
• 1,852 student editors
• articles touched = 2,539 edited (253 new articles)
• added 1,381 new files to Commons [n.b. 2 instructors made it part of their assignments, including Andrew Lih]
• 96M page views
• incidents: zero

Slide 5: Successes

• 65% retention rate of instructors
• Zero incidents
• Strengthen bonds with volunteers as the experts in teaching with Wikipedia: WikiCon USA, Wikimania
• Reestablished partnerships with ASA and APS
• Created usable databases
• Highlighted reach (page views)
• Split the Classroom PM into 2 roles

Jami: we are [now] seen as the link between academia and Wikipedia.

Slide 6 – Why [instructor] retention matters: [chart: Retention rate by courses]

Jami: the instructors who have taught with Wikipedia before are doing a better job and getting more advanced as they gain experience; therefore they require less handholding; they bring volunteers into the program from their pool of students; they seek out librarians and other participants at the institution. [When instructors become committed over time/engaged] as champions [of our work] it requires fewer resources on our end.

Slide 7 – Education Noticeboard Incident Tracker: [spreadsheet] Jami: [walks group through the spreadsheet and gives example of] when discussions are logged and where there are several incidents from one class, we can determine how to help (i.e. if they are sourcing incorrectly, etc.)
Jami: also, we want to log how many incidents are taking place [overall] in [distinct] disciplines, and assess what we learn from that [e.g. about challenges related to editing a discipline’s related content area on Wikipedia].
Frank: it also informs the kind of support materials we are developing.

Slide 8 – Salesforce demo:

[Sage created courses tab that shows various info: origins of class, etc.; will in the long term show the impact of individual volunteers; and eventually we can use/export the collected data to show instructors what their students did.]

Sara: If we are tracking partnerships, how do we connect institutional relationships to “more high quality content”?
[group discussion about this and Salesforce]
LiAnna: the dashboard will be critical to measuring [the connection between partnerships and high quality content]

Slide 9 – Master Article Database: [spreadsheet]

Jami: I spend a lot of time looking at what students are doing on Wikipedia. So how do we capture that? The master article database tracks articles, topic, bytes before and after, type of contribution (new or existing article), course ID, etc.
Frank: this tool is part of “building infrastructure for the future”; only when you track things and know what’s going on can you react early to trends i.e. warning banners/tags, etc. (and it can be automated)

Slide 10 – Why page views matter: [chart; refugee women and children: 11,000 page views…] Frank: clearly this is aligned with current events/politics, etc.
Jami: it is useful to share page view data back with instructors to share with their students (and motivate them).
Jami: it would be great if we could look at geotags [to show the reach/world regions where an article is viewed] and use that [to show the impact of student work].
Frank: this is the story about how Wikipedia effects people’s lives, and making connection between student and reader (the content somewhere in between).
LiAnna: also note that since the content is academic, it is more important in the grand scheme of things (vs., say, the article on Justin Bieber); also this is great example of diversification of editors through student participation (many females developed the articles related to refugee women and children).

Slides 11-13 – A term in the life of…[the Classroom Program Manager]

• assignment design: emails, phone calls, Skype calls, reviewing syllabi and draft assignments
• documenting in databases: course information, contact info
• monitoring student work: via the activity feed – keep an eye on class status
• highlighting student activity: Trophy case, blog posts, Twitter/Facebook
• Connecting Wikipedia Ambassadors to instructors and student editors.
• Supporting our partners: strategy about how to expand on campus? How to best support academic associations?
• Gathering data: evaluating course and term activity
• Training/mentoring university faculty to become Wikipedia Ambassadors
• Follow-up with past instructors: revise assignments and share info about new resources and tools.

Jami: re: monitoring student work/activity feed: from this, Jami updates every monthly report with what that term’s current classes’ activity looks like.
re: highlighting student activity: this shows what’s coming out of the program – great for outreach to partners)
re: training faculty as Ambassadors: these are people who already have training in the types of skills we want Ambassadors to have; it’s just learning the Wikipedia part that needs to be done, and this is [often best accomplished] when they (instructors) teach others to do so.
re: follow-up with past instructors: this ensures the workflow is right (e.g. that students are adding content to Article Namespace) and that the resources we offer help keep students engaged.

Sara: what are incentives for Ambassadors to participate? Jami: fellowships for some; for Wikipedians, learning to train others on Wikipedia gives them an additional skillset (and helps improve Wikipedia); for instructors who care about teaching, they are motivated by the skills conferred to their students (media literacy, etc.).
Jami: assignment design ensures that participants receive well-rounded training on all aspects of editing Wikipedia (e.g. are there enough resources on topic, do they understand notability, etc.). The assignment design wizard will be an important first step.

Slide 14 – Would spend less time on…

• Classroom Program
  • Logging student usernames
  • Gathering data
  • Developing databases
  • Monitoring student work

Slide 15 – Would spend more time on…

• Classroom Program
  • Pairing courses with WikiProjects
  • Training/mentoring Ambassadors
  • Analyzing data
  • Highlighting good student work
• Educational Partnerships
  • Coordinating with instructors/experts to create “content gap lists”
  • Building relationships with the organization, its instructors, and campus faculty
  • Outreach

Frank: re: content gaps list: once you have a list, you can classify the type of list and specific instructions as to what these types of articles need. [For instance, if] there is a class of articles that should be of a certain structure, we can provide an example of an article that achieves that structure so that students orient their work towards it. (e.g. articles about health in developing world would ideally list sociological impacts.

Slide 16 – Support from others

• Communications: blog posts highlighting good student work, campus PR and outreach, subject-specific materials for academic associations, articles for member magazines.
• Digital infrastructure: survey tools, API support.
• Fundraising: keeping the Classroom Program involved in any promises or leads, since we only have 2 chances a year to set goals (e.g., target gender gap courses?)

Note: Sage developed API while [working] at the Wikimedia Foundation to pull course info; the Wikimedia tech has not done anything beyond this wrt API.
Group discussion re: priority setting and program needs driving funding vs. the other way around. LiAnna spoke about experience with community colleges example; that the experiment helped us see what works/what doesn’t.

Slide 17 – Questions Questions/discussion
Jami: Big open question: how do we establish the baseline for content gaps?

Next three months’ priorities

Slide 18 – Looking forward…

Slide 19 – The 2 roles

• Classroom Program
  • Support Classroom Program day-to-day activities
  • Coordinate Ambassadors
  • Focus on bringing more student editors
• Educational Partnerships
  • Support Classroom Program for partners (e.g. sociology, LSU)
  • Coordinate Ambassadors on partner campuses
  • Focus on supporting more student editors

Jami: both [classroom and partnerships] roles are geared to bringing new students/more courses on the campuses but Jami’s focus (partnerships) is on bringing more support capacity to ensure sustainability and develop system within institutions so that they don’t need to continually rely on us [and so the contributions are still of high quality]); Helaine’s role (classroom program) is to supporting student editors through existing systems.

Slide 20 – Fall 2014 goals

• 85 courses
• 80% satisfaction rate for instructors
  • measure with an end-of-term survey
• Classroom Program Manager has developed a new Ambassador model to implement by December 1st (with support from Jami)
• Formalize partnerships with 3 academic orgs and 2 campuses

Slide 21 – Numeric course goals

• 85 courses
  • 50 taught by returning instructors (60% return rate)
  • 15 new courses from champions and academic associations
  • 20 courses of organic growth (or from champions)

LiAnna: challenge with organic growth is that we get one-off professors in random places that don’t have a support structure (though you never know when an individual might become champion etc.)

Slide 22 – Academic Associations

• 3 formal academic association partnerships by the end of 2014
  • ASA (sociology)
  • APS (psychology)
  • NCA (communications)
  • NWSA (women’s studies)
  • AFS (american folklore)

[Jami gave background on Wiki Ed's relationship with each association]
Jami: Academic Associations care a lot about the student experiences so they will collaborate with us on surveys, etc.

Slide 23 – Campus Partners

• 2 formal campus partnerships by the end of 2014
  • Georgetown
  • Louisiana State University
• Potential partners
  • Barnard College
  • Brock University

Slides 24-30 – Impact of Partnerships

• Bringing more student editors
  • our primary goal: expand the Wikipedia Education Program to more student editors
  • academic organization expansion will bring more courses, as will local outreach from campus faculty
• Expand support capacity for the Classroom Program
  • Instructors and student editors need support!
  • We can support more instructors when we have subject-specific assignments and training materials to on-board them
  • Campus faculty can manage more courses if they build the role into their job and departments
• Higher quality participants
  • Subject-specific training materials better prepare student editors before they edit ** Campus faculty can help student editors with their writing nd research skills in addition to their Wikipedia-editing skills.
  • Campus faculty can target high-achieving students and classes based on the campus culture.
• Better impact to Wikipedia: targeting content gaps
  • Academic associations members can develop lists of content gaps on Wikipedia within their specialty
  • Campus faculty can target disciplines via outreach to department heads and other support faculty
• Maintaining a high instructor retention
  • Better support materials = happier instructors
  • We can engage more instructors by asking them to contribute to content gap lists, meeting them in-person at conferences or on-campus, and connecting them with other instructors to advise
• Credibility within academic community
  • Academic associations are credible institutions already
  • Formal partnerships with university departments/centers will bring well-trained faculty members into our support infrastructure
• Credibility in the Wikipedia community
  • Targeting missing content on Wikipedia that is not being added otherwise
  • Look at the formal institutions who respect us as the experts on teaching with Wikipedia

Jami: re: subject-specific training materials: partnerships enable us to focus on subjects and create materials.
re: high achievers and identifying groups to target: we want to tap into whatever specific area of expertise of that university as identified by faculty. The people on campus know what resources they have and need in relation to our programs.
Jami: in the partnerships role, I will be traveling a lot more/on the ground; engagement with/support for instructors makes them more enthusiastic. The responsibility and visibility given to them also serve as incentives to be engaged.
re: Ambassadors: our experience is that Ambassadors with institutional support are doing better; they have good standing; it becomes part of their job. Also, these institutions see us as the bridge between Wikipedia and academia.

Slide 31 – Open Questions

• What are the baselines for content gaps so we can measure our impact?

Slide 32 – Supporting Partners

Slide 33 – Meetings and Strategy

• what are our numeric goals for your organization/campus?
• what do we need to do to support you in reaching those goals?
• how do we identify new participants and fully train them to be successful?
• how do we incentivize instructors? Ambassadors?
• who needs to support this initiative to remove roadblocks from expanding our reach, and how do we get their buy-in?

Slide 34 – Presentations

• teaching webinars
• attend conferences
  • at least 2 reasons to attend (exhibitor’s booth, teaching workshop, partnership meeting, pilot opportunities, etc.)
• lead on-campus teaching workshops

Slide 35 – Publications

• write blog posts about the partnerships and promoting the participants’ work
• write articles for academic associations’ magazines or campus newspapers
• develop PR materials for campus outreach, newspapers, mailing lists, etc.
• subject-specific help and training materials

Slide 36 – Data

• pre- and post- surveys for student editors and instructors
• evaluate student learning outcomes and satisfaction rate
• number of courses/student editors/articles edited
• amount of content added within their discipline/on their campus

Slide 37 – Supporting Wiki Ed

Slide 38-39 – Data Needs

• Retention of instructors
• Number of courses, instructors, universities, and student editors
• Number of courses, instructors, universities, and student editors by discipline/topic area
• Program’s reach: page view statistics
• Qualitative data (stories about impact and student/instructor experience)
• Number of Ambassadors
• Number of articles edited
• Number of new articles
• Number of incidents and median days resolved
• Number of Did You Know, Good Article, Featured Article
• Amount of content added

Slide 39 - Open questions

• How do we measure quality improvements?
• Create a baseline with content gaps and evaluate our impact to it? Some other measure?
• Formal adoption of program/curriculum through policy change at institutional level: how do we define “formal”?
• What needs do you have that weren’t listed?
• What is the best way to provide ongoing information and data for communications and fundraising needs?

Frank: Excellent quarterly review. Jami can be proud of what has been achieved.