Wiki Education Foundation/Quarterly reviews/2015-Q3 Digital Infrastructure

The following are notes from the Quarterly Review Meeting for the Wiki Education Foundation's Product Manager Digital Services Sage Ross on April 8, 2015.

Present: LiAnna, Jami, Frank, Samantha, Bill, Eryk, Karen Twitchell

Participating Remotely: Sage, Ian, Adam, Ryan

Please keep in mind that these minutes are mostly a rough transcript of what was said at the meeting, rather than a source of authoritative information. Consider referring to the presentation slides, blog posts, press releases, and other official material.

Intro

Slides from Wiki Ed's Digital Infrastructure Q3 Review on April 8, 2015

Frank: Because Karen Twitchell (board member) is here, I am going to repeat that these reviews are for everybody in the organization to come together to see each other’s work. Working in silos is never a good idea, so every quarter we look at our key initiatives and see our learnings, and then ask what the path ahead looks like. This creates internal accountability, but also accountability to the community – which is why we publish these notes, and who said what. When we started, many people asked us, “We don’t know what’s going on,” “You keep us in the dark,” etc. – so I said “Let’s change that.” That’s why we do this. So, Sage!

Sage: Big picture, last QR was December 1 – seems like forever ago – so I'll give a quick overview of what we did since then. Then I'll reflect a bit on learnings and areas for improvement, and then I'll walk through the current set of plans we have in terms of the next quarter, which is really all about course pages, and our main longer-term ideas for the next two years or so.

Assignment Design Wizard

Sage: In early December at the last digital infrastructure QR, we were still polishing up our first big tech project, the Assignment Design Wizard — which at the time, did not let instructors customize the timeline. We wrapped that up in time for the main bulk of Spring courses, and ultimately, the vast majority of our current courses have assignment plans that were drafted using the wizard. We don't yet have systematic feedback from instructors this term, but based on the user testing I've done and based on observing how instructors have used and modified the plans that come out of the wizard, I've been very happy with it so far.

Right after wrapping up the official launch of the Assignment Design Wizard, I started working with our development company, WINTR, on the Dashboard project. We did the soft launch of dashboard.wikiedu.org in early January, and we were building out the features of that until early-February.

In February, we really stepped back to take stock of where we want to go. We had recognized previously that in the long term, we should aim to move away from the Education Program extension and that we wanted to build our own system for course pages (and another system for managing other users of our program).

But we didn't really have a timeline for that, or a plan. So in February we fleshed out a plan for next steps, for the longer term roadmap of our tech projects, that we didn't have back in late 2014. So the upshot is that we decided we needed to build the course page replacement system sooner rather than later — in time for the Fall 2015 term. And in February we did a planning sprint with WINTR to basically lay out what we needed to do to get there, and asked for — and got — funding to do this project — and then we started doing it in March.

Dashboards

Work so far has really focused on laying the groundwork for transforming the dashboards into a fully-fledged course pages. So we did a lot of cleanup and bug-fixing to get to system ready to scale up, we deployed a few of the features we'd started but didn't have time to fully implement during the earlier Dashboard sprints – a nice mobile layout, and showing the ratings of articles – and just last week, we created the login system and got the basics in place to make edits through our system. We also spent a lot of time in the last several weeks working pinning down the user flow: how a user goes all the way from arriving at our site for the first time to setting up a project page to running their project — whether that's a classroom program course, or the activities of a student club, or any future programs that involve a discrete group of editors. I'll go over the current plans in the 'looking forward' section.

Frank: This is super cool.

(Agreement).

LiAnna: And I want to point out that this mobile tool is crucial for instructors who have told us they’re often using iPads or tablets to check courses – it means instructors can do this on the bus, traveling, etc, and so it’s extremely useful to work this into these tools.

Frank: What if someone walks in with their own user account?

Sage: Well, the idea is that the dashboard is its own experience, so you log in and see your specific courses rather than right now, where you see the entire global list. You would also be able to edit your lesson plan, respond to details of edits, offer feedback through the interface – either on WP or direct to the student.

Frank: So… the visual editor may happen on WP soon; what is the consequence for us? Could an instructor never touch wikicode?

Sage: They don’t touch much wikicode right now, but they will have to touch wikicode with their minds to review student edits.

Frank: Are we trying to set up a system to avoid talk pages?

Sage: For common uses, yes. There’s no real reason to have to use wikicode. If you’re engaging in a threaded conversation it’s different. But eventually even that will be supported. But common-use cases for instructors, that stuff is pretty feasible.

Jami: Will there ever be a VE version of diffs?

Sage: I expect it at some point. There’s already some visualizations that do that. We can potentially eventually see surface diffs in a dashboard, see the before and after. Right now we want to basically duplicate the functions of the Education extension, offer up tracking, and in some ways that’s just recreating existing work. But we want to improve the user experience; and it’s also a bit tough to improve on account of being embedded into the Wikimedia ecosystem.

Ask.WikiEdu.Org

One of the things the programs team has wanted is a Question & Answer tool that we can use to build up a set of answers to commons questions that we have to answer over and over. So after some research on the open source options out there, we reached out to the developer of a platform called Askbot to see if he could create a custom version of it for us. I borrowed a little bit of design time from our Dashboard project to have WINTR mock up a basic Wiki Ed theme, and the Askbot developer has been implementing it for us. It's up and running now, but we want to make the login process a little smoother – so that it really feels like just using your Wikipedia account rather than signing up for a random new website – before we start trying to send people there and seeing if we can foster a community where people ask new questions and answer each other's questions.

Samantha: Is this just for instructors, or for students too?

Sage: The first-use case is program managers and content experts looking to make working with instructors more efficient. It could be useful for students, but we need to develop it until it has a body of useful information. That’s the long-term goal, is for it to be a more efficient way to answer questions than the morass of wikipedia help pages.

LiAnna: In the short term, we don't see this tool as being useful for program participants. The idea is that, anytime we get a question, we answer it and put the answer into this system. So over a year, we’ll build up answers to questions. Then, whenever we get a question, we can send people to log in and get the answers from this tool. We’ll respond to people, of course, but we want to encourage people to use this – librarians, instructors, whoever needs it, not just instructors.

Frank: We also want to standardize the answers. We have seen in the past that, often, different people can give a different answer to the exact same question.

Jami: Instructors have often asked for connections to other people too. This might help.

Smaller Projects

FOSS Outreach

From mid-December through mid-March, I co-mentored an intern for the Free / Open Source Software Outreach Program, Anke Nowottne. Anke worked on a design research project — basically, trying to pin down problems that education program participants are trying to solve and sketch out some solutions. This was part of Wikimedia Foundation's mentorship efforts, so it was focused not just on Wiki Ed's program, but there's enough overlap that it was also a relevant project for my work — although not quite as relevant as I thought it would be at the time we were sketching out the contours of the project, since at that time we were still anticipating that it would be a while before we seriously overhauled the course page experience.

Salesforce

Sage: Over the last couple weeks, I've also been working on finding a good Salesforce consultant. We want to do a small project to improve and streamline our Salesforce setup, which could make us significantly more efficient. So I ran a brief request for proposals and talked with several consultants who specialize in nonprofits. We should be settling on a consultant soon and moving that project forward in the next month or so. Right now, it’s set up, but we don’t know what best practices to follow or how to best use the tools Salesforce offers.

Sage: The expert will come in to see how we’re using Salesforce, compare it to our goals, and help us figure out the best way to make it work.

LiAnna: The first step is a needs-finding. Salesforce is pretty powerful, so we’ll have someone change our setup to make the most of it in a streamlined fashioned that makes sense for what we do and where we want to go. As part of that we’ll also do some administration training for Sage and maybe a bit of internal training.

Frank: We always wanted to know basic details about the people we work with, how many courses are instructors teaching, etc. I hand coded a tool over a few weekends but that wasn’t sustainable. There are a lot of statistics and key metrics we want to see and should be able to see through Salesforce.

Jami: There are so many useful ways we can use this, such as seeing relationships…

LiAnna: Having communications records in the history is also super useful; our institutional memory is all in Jami’s head at the moment, as we move people or expand or change it’s a great tool for having instant access to the history and conversations we’ve already had.

Plagiarism checking

Sage: One of the goals we had for this term was to find a short-term solution to checking student editors' work for plagiarism. Our plan has been to use the work of James Heilman and Eran Rosenthal, the developer James hired to build a plagiarism bot. Having something working for this semester is still up in the air, although I think it's close. But we also firmed up our relationship with iParadigms, the company that runs Turnitin and whose API is used by this plagiarism checking tool. So when we're ready to create our own version of it as part of the course page system, we should be ready to go.

Frank: Who is the target group for the information about plagiarism? Who is told this? Is the instructor told, or do Ian or Adam have to clean it up?

Sage: Right now, when this goes live and is sorted by WikiProject, you can sort it by WikiEd Classroom project. Adam and Ian would then go in and triage those. Eventually, the first responsibility will be on the person who makes the edit. There will be false positives, so sending it to the student and asking them to adjust it is presented as a learning point. It’s useful to catch it based on small edits and get a student to adjust rather than go the whole semester and then check at the end and all their work is lost-- when perhaps a correction early on could have changed the course of what the student thought was OK.

LiAnna: Our first go, there are no automatic notifications aside from going to Ian and Adam. So we’ll see how many students are doing it; we’ll see if Adam and Ian can change that behavior by intervening, we can see the false positive rates. Having human evaluation first (Adam and Ian) will help us to automate in the long-term; maybe in two years, as students save an edit there’s an automatic check and it’s flagged before it’s even posted. But that’s a long-term goal.

LiAnna: Wiki Ed is also sponsoring a plagiarism awareness week; so I’m giving a presentation tied to them. We’re building a good relationship with them.

Sage: And, right now, we’re running it a few times a day, it makes sense to spread out the scans. When we implement it, we’d like to send every edit as soon as we know about it. At the current scale of our program, it won’t interfere with it. So, we’d like to go with continually checking. If we get such a high volume that it interferes with their paying customers, we may need to make adjustments or discuss a change in that relationship.

Reflections and learnings

Working with others

Sage: One of the things we've now experimented with twice is small projects with outside developers.

RecentActivityFeed

First was our RecentActivityFeed project, where we hired MediaWiki dev shop WikiWorks to build a simple extension that we wanted to integrate into the EducationProgram extension. In January we ended up calling it quits on this, after getting stalled for several months with the Wikimedia review process.

LiAnna: Waiting for useful code changes to go through an approval process was keeping us back.

Frank: We’re one of the first situations where we are developing our own software, with an outside developer, which goes around the bottleneck of that code approval process with Wikimedia. Other organizations in this ecosystems might like to learn from us and do the same thing.

Sage: They may not know they want to learn this from us yet, but they will. (Laughter)

ask.wikiedu.org

Second is the ask.wikiedu.org platform, which I feel good about at this point — although we don't have nearly the level of control we would have if we took the project on ourselves. I don't have a firm takeaway on this, but it's a tradeoff I think we'll need to be continually aware of in terms of predictability and control of the system — when we handle things more directly — versus expediency and low cost — when we work with remote contractors and systems that we aren't running ourselves.

WINTR

One of the things I'm really happy with right now is our work with WINTR. I had the opportunity to spend a significant amount of time in between our main projects during February, getting the testing in order and really coming to understand what's going on in our system, and that's really been paying off I think in terms of being able to bring my Wikipedia domain knowledge to the day-to-day development work. Now, according to the standard agile development anecdotes, that means I'm becoming just skilled enough to be dangerous. But I bring this up mainly in terms of professional development, I've been really happy with my own progress, and it's made the work over this last 4 months really fun.

LiAnna: I also heard from WINTR that having someone who really understood the software and program needs helped them in their work.

Frank: They even highlighted it when they came to our offices. Usually the relationship is one where the client says they want a blue thing with legs and two months later says they expected a red thing with wings. So having Sage work along the process has been really helpful for them.

Looking ahead

Sage: So I'll review the roadmap, in terms of the immediate goal of having full-on course pages in time for the fall 2015 term, and then the main projects we've sketched out beyond that.

Course Pages

Sage: For the course pages, what you'll see on the testing dashboard in the coming weeks will be a workflow for creating a new course timeline.

At first, it'll just be a blank slate, where you create an empty timeline and then you can add items to it, and publish an on-wiki version of it to Wikipedia, but make further edits and save them right from within the wizard.

And then we'll add on what is essentially the same workflow as the wizard, where you get to make choices to configure the details of a standard type of assignment, but instead of just publishing to Wikipedia and that's it, your assignment timeline actually lives on wikiedu.org, and changes basically get mirrored to Wikipedia.

From there, we'll add the ability to add yourself, or other users, to classes... just like we do now on the EducationProgram extension course pages... and to assign articles, sign up as a reviewer.

Transparency

And the way that it will work is that whenever you take an action on your wikiedu.org course page, your account will also make a corresponding edit on Wikipedia. You enroll as a student, and you automatically make an edit to your own userpage indicating that you're in such-and-such class, and you automatically add your username to the list of students on the course page. And then you add the article you're working on, and if it exists then you automatically add the assignment template to the article's talk page, and you also update the course page. This way, we don't really lose any of the transparency of the current course pages, and in fact, it should be a little easier to follow what's going on with student editors because all the pages on Wikipedia will now just be plain wikitext, and everything will just show up in Special:Contributions.

Dashboards

Beyond the course pages, we've got a lot of ideas, and a rough order of current priority. So there are some improvements to the dashboard features to make them really go beyond what the current course pages do — make it easy to visualize the individual changes editors are making, showing contributions to Commons. And then we'll get into automatically taking actions based on the contents of edits. So automatically checking for plagiarism and messaging people when there's a match, instructions for moving to mainspace that you get whenever your sandbox starts to get too big, DYK instructions when you've expanded an article five-fold or started a new one that meets the size requirements, these kinds of things.

Additional Tracks

After that, hopefully in time for the Spring 2016, we'll fold an updated version of the training into wikiedu.org... so better interactive tutorials, knowledge checks, and maybe professional videos.

And then we'll want to build something to address the challenge of finding Goldilocks articles: not too well-developed or too complex and broad, but not so small and narrow that you can't find any sources; just right.

The roadmap goes on from there, and I encourage you to check it out if you're curious, but of course, all of this is subject to change as we move forward and continually keep track of what our biggest current roadblocks are and what our development priorities should be.

Questions and Comments

Ryan: I’d love to see mock-ups or flow charts…

Sage: LiAnna has some very detailed ones right now.

Jami: Instructors are telling me that it’s hard for instructors to make a class look like it isn’t a course about Wikipedia. The course page is dedicated to Wikipedia; there’s no space to, say, show what the reading is, or what the class assignment is. Is there any way to say, hey, we want to use this to replace an external, probably closed-off, class management system?

Sage: That’s an important question about where we want to draw the line. Class management systems are notoriously unwieldy development projects. To get people to move from one to another means, likely, a whole lot of development of niche features that people are used to.

Sage: As a first-use case, one simple thing we’d like to add, initially, are things like reading lists for each week. You can move things from place to place, blocks of content for example, and move it around as you need to. But there are an endless number of unique things people like to do with Moodle or Canvas etc. We’re not looking to replace that, but we’d like to make something where, if you have a course where Wikipedia is a central aspect of it, you may not need to go outside of our toolkit.

Jami: So, is real-time student support something we’re willing to do in the next fall? I suppose this is the “just in time” learning model, sorry, not real-time.

Sage: The just-in-time-bot, OK. That’s a system where the platform responds with next steps or responses in reaction to what a student has just done. We detect that a student has made an edit, we test it for plagiarism; we see that a student has hit a point in their sandbox, we give them instructions on how to move that to mainspace. Or we see they’ve hit DYK requirements, we pop up and tell them how to propose a DYK.

One of the things, in broader strokes, some of what we’re doing is tied to this. You can see wikiedu.org/roadmap with a sketch of when we want to start tackling the various components that we’re aiming for. We’re looking at fine-tuning response points, for example, for when the just-in-time-bot might suggest things.

We’re looking at revamping the training, then moving it, around Spring 2016, into our own system, so we can create knowledge checks, ie, “Do you know what notability means?” etc.

LiAnna: And this training is super-adaptable to a variety of programs and uses, we hope. If someone wants to do editing around a specific topic we can customize the training for that topic, for example.

Samantha: So tied to subject-specific brochures, partnerships, etc?

Jami/LiAnna: Right.

Frank: The training, and just-in-time feedback, are crucial to getting people to do something new, that they have never done before.

LiAnna: And we don’t let people through until they know what we know they need to know to edit.

Jami: If they need an 80% grade to pass, we can see who got that and post it to the dashboard.

Frank: This entire idea though, training and just-in-time, can actually help the entire Wikimedia movement. Are we considering this in our architecture, maybe making it more flexible so it can be adopted to different uses?

Sage: Anywhere where it doesn’t constrain our work, we’re able to do it. But transforming a system for a variety of uses is difficult. So it’s not driving our development. Our needs are what we’re focused on. There may be some serious efforts to transform our platform to something that can be adopted for other purposes. So, we’re making things as modular as possible for an outsider to come in and see it and start working on it; but our priority is our own use and our own needs.

LiAnna: Also, the knowledge of internationalizing projects, for example, are maybe better handled by the expertise in the WMF.

Sage: And cultures and needs, different systems, these can be challenges. A Swedish course is using some of it right now. I can see collaborating on this with, say, Wikimedia if they were willing to put some time into it. I could see collaboration as the most sensible way to start universalizing it for other purposes.

Frank/LiAnna: Great work so far, Sage. Things are going well, and looking forward to seeing the new course pages system roll out this summer!