IRC office hours/Office hours 2013-04-15jw

Office hour with Jessie Wild on April 15, 2013
Times listed in Pacific Standard Time

[11:04] Pine OK it looks like we're done with the joins so let's start
[11:04] Pine Jessie, let me give some background for my questions
[11:04] jwild that would be great!
[11:05] Pine I have had concerns about AFT5 repeating mistakes from IEP, and I've been looking over some of the WMF monthly reports to try to get a feel for what's happening in programs and evaluations
[11:06] Pine I'm not sure I understand the whole picture, but it seems to me that at least some important info is getting lost and forgotten by WMF
[11:06] Pine On the other hand, I'm not sure that programs and evaluations are the unit that's supposed to be handling that kind of thing
[11:06] Pine I'm not sure that anyone is actually doing it
[11:07] Pine So can you start by explaining what you do in the evaluations part of programs and evaluations?
[11:08] jwild Yea definitely. So: the WMF structure around evaluation/learning is emerging and thus a bit confusing/messy still
[11:08] jwild As you probably know, there is a department developed in december called "Grantmaking & Programs"
[11:09] jwild within that, there is a group led by Frank Schulenburg concentrating on evaluating global programs as a whole -- basically trying to highlight which programs are achieving great impact and some indicators as to why
[11:11] jwild I fit in more at the organizational/strategy level with learning & evaluation … for example, what are the structures needed for impact? what capacity/skills are required? how are we 0x1dbuilding appropriate "suites" of programs?
[11:12] Pine Is the evaluations group looking only at external programs? I get the feeling that Programs spends most of its time on Brazil and the education outreach programs, with no one doing a systematic review of internal WMF-run programs like AFT5.
[11:13] Pine I also get the impression that WMF has no internal mechanism for sharing info learned from previous programs internally.
[11:13] jwild The Program Eval group is looking at GLOBAL programs. You are appropriately identifying a gap in the overarching view of programs solely within WMF.
[11:14] everton137 " I get the feeling that Programs spends most of its time on Brazil" if it comes to the edeucation program, we should compare the time and resources spent in Brazil, India and Cairo.
[11:14] Pine everton137: my comment is based on what I've seen in a brief look at some of the WMF monthly reports
[11:14] jwild So - one other funky structure thing within WMF is the regrouping of the Analytics team. Previously, they have basically been doing just the backend stuff (wikistats, report card, etc), but they are trying to build up the capacity to do more internal review
[11:15] everton137 Pine, nice that you have such impressionI will read it later.
[11:16] jwild BUT one thing that i think we need to be on the lookout for both at WMF and across all movement partners is looking at the lessons from the different programs. It seems to me the general mentality is that every project/program is non-comparable, and that is a limiting viewpoint.
[11:16] jwild does anyone else see that as being true too? in WMF and elsewhere?
[11:17] everton137 I would say it is a stupid point of view
[11:17] Pine Supposedly best practices in project management as a discipline should apply to many different kinds of projects.
[11:18] argenton the log is on?
[11:18] Philippe Pine is manually logging.
[11:18] Pine That was one of my reasons for my question about using IEP in the curriculum that Gayle is using for all WMF supervisors
[11:18] Pine It's also because I'm concerned that lessons learned at great expense from previous projects are being forgotten.
[11:18] everton137 I agree that there is too much to be improved on the learnings across all WMF programs and projects + how can we learn from the community (user groups, chapters etc.) initiatives.
[11:19] argenton thank you Philippe, in wich location do you gonna share this?
[11:19] jwild pine: so the best/good practice in this example case would be to develop projects in conjunction with the community from the onset?
[11:19] Philippe argenton: traditionally, logs go on meta at Office hours, I believe.
[11:19] argenton thank you again
[11:19] argenton
[11:20] Pine Well, that's one point. I strongly encourage people to read the IEP consultant's report. I have some concerns about some of their conclusions, but I think their overview of the facts is well worth reading.
[11:20] Pine One of those points would be the value of communication and development with community partners from very early in the design process.
[11:20] Pine Particularly for major projects.
[11:21] jwild I agree: I think that would be a great on boarding exercise. Not just for WMF staff, but even for the chapters/volunteers that are orchestrating projects
[11:21] FloNight Pine and jwild, I agree.
[11:21] Bence Pine: are those conclusions useful only to the WMF, or would they be useful for others as well?
[11:21] everton137 +1 to agree
[11:22] jwild one thing that we are going to be developing/compiling (i mentioned this on wikimedia-l, i think?) is a portal of some sort for lessons learned (qual and quant) and resources for development. I am making a note to be sure to include this document in there
[11:22] argenton Pine:What is the reason for people to read if there is no discussion around? If you can not understand the motives of those observations?
[11:22] Pine Benice: I think they would be useful for anyone involved in WMF related projects, including chapters and online initiatives.
[11:22] jwild +1 Pine
[11:22] Pine The bigger the project, the more important it is do do the planning well.
[11:22] Pine *to do
[11:23] Pine Another point that I think WSC raised in his email, and that I agree with, is the importance of having an exit strategy.
[11:23] everton137 Pine, and this was much improved in the Education Program in Brazil, as I said at wikimedia-l. We started with a smaller pilot to learn how to grow. For this the Program in India, although had problem, was useful for other places.
[11:23] Bence (I haven't read that report in detail, but I fear there is too much information out there and it is difficult to find what is relevant and applicable. A dashboard might help in that;but perhaps trying other media than simple text might be useful as well [videos, blog posts, bullet point summaries])
[11:23] jwild to all: how do you think we can develop more lessons like this documented one? I want all our movement partners to have these sorts of "learning hats" on when doing project reports/blogs/etc.
[11:24] Pine But my hope is that people don't just read WSC's email to get an idea of "answers". As a case study, I hope people look at the consultant's report and come to their conclusions.
[11:24] Pine *their own conclusions
[11:24] everton137 (Bence, good idea. To organize such kind of info would be really useful)
[11:24] Pine jwild: WMF could develop project management as a skill set for many of its employees and have systematic ways of developing that skill set, including case studies like IEP
[11:25] jwild Pine: I'm also thinking more movement-wide, e.g., with large chapters.
[11:25] jwild but, any materials developed could I suppose be used with/for both.
[11:26] everton137 I think we shold avoid as much as possible private wikis. That can be good for people to learn from the process and this will help not to loose information.
[11:27] Pine Some training materials might be applicable in more than one place. But specialization helps too.
[11:27] Thehelpfulone "we can develop more lessons like this documented one" -> sorry which documented one? link please?
[11:28] everton137 from the last Pine comment on project management as a skill, I wonder how to share about the process of building projects. Some times we want show just the results and while doing so much stuff, we loose also how things were actually *done*
[11:28] argenton jwild:sometimes the hat does not fit in everyone's head, I think one should document as much as possible, but also be something like, "this hat did not fit for me" and as Asaf said, I think the best way is the idea a cookbook, we have x resources, this the list of thing that you can do withit, no, do it because you have x.
[11:28] everton137 the cookbook idea is good, argenton
[11:29] FloNight Sharing learning about movements problem solving of common problems such as translation, global time zone management, using volunteers and staff mixture for projects but with the understanding that there might be a variety of possible solutions not one right answer.
[11:29] jwild argenton: totally. I think we should steer away from the idea of "best practices," but rather focus on lessons / case studies. This makes it more of an informed experiment environment!
[11:29] FloNight right like a cookbook
[11:30] Pine I think the term that HR people use is "tool set". Everyone gets tools and training in how to use them, and then people need to select the right tools for the job.
[11:30] everton137 Pine, to have this kind of tool set would be great. good ida
[11:31] everton137 idea*
[11:31] Pine Another term I've seen used is "skill" but to me "tool" is the better word because some tools are technological rather than just mental.
[11:31] argenton I already gave my 2 cents, I need to go, I'll read the log, if it is available. :*
[11:31] jwild yea definitely. to Bence's point about documentation, though, the BIGGER challenge I am currently wrestling with is how do you create an environment in which those cross-group conversations happen.
[11:32] Pine jwild: this is what continuing education is forIt's also what I hope is happening in Gayle's training for supervisors.
[11:32] jwild But i don't just mean for WMF
[11:32] Pine OK let's take an example
[11:33] Thehelpfulone sorry Pine / jwild in case you missed my question: <Thehelpfulone> "we can develop more lessons like this documented one" -> sorry which documented one? link please?
[11:33] Bence Sometimes making sure communication happens inside an enitity is difficult, so indeed, cross-group communication is hard to crack
[11:33] jwild I mean: how can we have a movement environment that is conducive to this -- e.g., WMHU (since you are here, Bence ;)) to share about a program that they totally scratched with the broader community . (note: totally fictitious example!)
[11:34] everton137 For this global shared learnings the workshop in Milan between people involved and interested in the Education Program can be important ** link substituted https://outreach.wikimedia.org/wiki/Wikipedia_Education_Program/Education_Program_Leaders_Workshop_2013#Agenda **
[11:34] Bence I think the Chapters meeting and Wikimania are the two places where it usually works, not so much online
[11:34] FloNight Bence, I agree.
[11:34] jwild thehelpfulone: http://en.wikipedia.org/wiki/Wikipedia:India_Education_Program/Analysis/Independent_Report_from_Tory_Read
[11:34] Pine jwild: also, WMF could suggest or require that people go through some online training as a condition for getting grant approvals.
[11:35] jwild could be. or an annual in person, ongoing training session
[11:36] Pine jwild: for WMF staff or large grant recipients, perhaps, but that's not practical in a lot of cases with small projects
[11:36] Pine or projects with tons of people involved
[11:36] Pine for example, RFA reform on English Wikipedia.
[11:36] Bence interestingly, the grant reviews might be one place where some cross-group communication is happening with a mix of people. Although I fear that at this point the tone is more accusatorial (questions about steps taken or not taken, etc) than cooperative
[11:36] jwild Right, great point
[11:37] jwild Bence - yea that's what my concern is too/
[11:38] Pine I hope that sharing info learned from grants is seen as a high value part of what the WMF Grants staff does.
[11:38] Bence I would also love if in-person trainings could take place for relevant skills - even if there are obvious limitations (people need to be in one place, etc.)
[11:38] Pine Also, it would be good to have a centralized place where lessons can be shared on Meta with links to the relevant grant reports
[11:38] jwild that conversation could also meaningfully happen on the different grant reports themselves, but i fear those will just be pure "reporting" and not evolve to true learning-sharing
[11:39] jwild Pine: learning & evaluation portal!!(I obviously agree!)
[11:39] Bence it might be useful to take some random lessons from recent grant reports and share it on wikimedia-l, those usually lead to some conversation and if relevant, get forwarded to many lists
[11:39] Pine Yes, WM-L is useful but I worry that people have short memories.
[11:39] jwild bence: that's an interesting idea
[11:40] Pine What if there was some regular discussion about the L&E portal on WM-L, monthly updates something like what's done for the other regular reports?
[11:40] jwild haha Pine is hitting on the problems of humanity!!!!
[11:41] jwild good idea
[11:41] jwild I don't know, though, how much discussion should be on WM-L for this … why not live in the portal itself?
[11:41] everton137 Pine, I like this idea
[11:41] jwild and use WM-L to just draw attention to where the discussion is happening?
[11:41] Pine jwild: it doesn't matter how much good info the portal has if people don't read it.
[11:42] Bence Reports tend to be long and contain multiple subjects - so I rarely see them generate mailing list discussions (although that could happen on Meta) - they do have their uses though
[11:42] Pine And hopefully if it gets regular updates people will return to it repeatedly to sharpen their skills
[11:42] everton137 Re: wikimedia-l to draw attention: I think it should work more like that most of the time
[11:42] jwild pine: yea totally - so we could post highlights on WM-L, but host discussion on meta
[11:43] Pine Yes, that could work. There will inevitably be discussion on WM-L also but those emails could be linked from the portal.
[11:44] Pine You may want to include additional lists besides WM-L, perhaps EE and tech lists.
[11:44] jwild ok here's a crazy idea that I haven't fleshed out: what if there was some sort of "rating" function built into grant reports? So, anyone could "rate" the grant with some sort of super simple rating system ("Super great project," "pretty good project," "ok project")
[11:44] Pine For the monthly and quarterly update reminders.
[11:44] everton137 I do like the idea Bence said above to share other kind of resouces from the projects, such as organize blog posts, videos, photos, news...
[11:44] Bence also, it would be useful to summarize the results of discussion once it does down and re-share it on the portal and wm-l; so that people who return later can learn without reading walls of texts <-- also useful for making it open to an international audience, as the summaries can be translated
[11:45] Pine Bence: yes, wall of text is a big problem and I am concerned that people who have English as a second language will get lost in heavy reports.
[11:45] Pine Wall of text also makes it easy for people to dismiss reading reports as "too much work" and "not worth my time"
[11:45] jwild Would that encourage people be more involved and thoughtful in reading? If used correctly, it could also help the different grant making committees decide what types of projects to fund
[11:45] everton137 jwild, Re: rating system: I think it is a good idea. In my opinion, it is tie for WMF software to try to go up to date to be aligned with Web evolution
[11:46] jwild bence: / pine summaries are always useful
[11:46] Pine jwild: I'm not sure about that idea. It's hard to rate projects that way.
[11:46] Pine I think you'd need a lot of different metrics, not just a single one.
[11:46] jwild Oh absolutely
[11:46] Pine Although you could do an average for a "final" score.
[11:46] everton137 Pine, we cannnot now if we don't do experiments
[11:46] everton137 know*
[11:46] jwild i mean this as one mechanism for feedback / community review
[11:47] Bence rating would probably be just a gimmick - but maybe useful for giving positive feedback in parallel to the free-text indepth critical review that is going on on the talk page
[11:48] Pine jwild: I suppose you could have a single metric for someone to give their impression of a project but I would be very careful about how that score is used.
[11:48] everton137 Bence, agree. And one trial doesn't excludes the other method
[11:48] Pine Yes, I think discussions are more useful, but then you have the "wall of text" problem.
[11:48] jwild pine: yep fair caution.
[11:48] everton137 or more than one trial
[11:49] Pine jwild: so let's say that WMF thinks all of these ideas are worth considering. Is there anything budgeted in the next annual plan to make progress on any of these issues?
[11:49] Bence you could also add buttons like "if you've done a project like the one in the grant, what were your biggest challenges; what worked surprisingly well, etc.", and then invite random people to the grants via wm-l or a monthly summary
[11:50] jwild bence: "gimmick" might be too harsh. one thing I have heard from other grant makers who are concerned about evaluating effectiveness of different programs is trying to do some sort of categorization of their programs into these sorts of general buckets ("high performers" -> "low performerS") and then doing some reflection on the commonalities between those grants grouped in those different categories
[11:50] Bence (clicking on the button would add a section to the talk page)
[11:51] jwild it would be interested if we could do something similar in this case: If we had categories of grants/programs/projects that we thoughts were really great, we could guide our learning conversations accordingly
[11:51] jwild (e.g., "Wow - it turns out that every project we think is the most impactful actually involved a huge mentoring effort")
[11:51] Philippe (FYI, and meant only as a time check: we're at 51 minutes for this conversation)
[11:52] Pine Philippe: yes, I'm on it
[11:52] jwild bence:really like that last idea
[11:52] jwild pine: regarding the next annual plan --
[11:52] jwild we are getting started with the portal ASAP … before the next annaul plan
[11:52] Bence might have been the wrong word – I just meant that a simple like button-like feature might not give us much info. Trying to categorise by impact, and other factors would be useful, perhaps it could already be done by the GAC themselves as part of their reviw?
[11:53] jwild bence: ohhhh yea! that would be a good testing group.
[11:53] jwild regarding the portal -- that's going to be a part of what my team is working on (toolkits etc.) and also the program evaluation team (in terms of good practices in program design, program similarities, etc)
[11:54] Bence adding some social features like : "notify me when this grant is funded/a review is ready" could also be interesting
[11:54] jwild pine you also brought up internal WMF onboarding/training materials
[11:54] Bence *report
[11:54] jwild in regards to these, I do not know
[11:54] Pine Benice: yes although I hope that lots of projects like AFT5 would go through a similar evaluation or discussion projects, particular major projects as they reach milestones. Have systematic ways of incorporating community review and adjusting course as needed.
[11:54] Pine The bigger the project, the more important this is.
[11:55] jwild bence - also an interesting idea. This could be built in most clearly, probably, into the IdeaLab since these are very preliminary ideas
[11:55] Bence Pine: sure, having some best practices on how to involve the community into projects done by "outside forces" like the wmf, chapters, etc. would be really useful
[11:55] everton137 Bence, these kind of notifications would be super helpful. I hope it can be build someday. It doesn't appear tehcnically difficult.
[11:56] jwild (I know we have a few minute still, but I just want to say this conversation has been really helpful to me, thanks for suggesting it, pine)
[11:56] Pine jwild: will the portal be included in the training or onboarding that Gayle and the HR team do for WMF employees?
[11:56] Pine Glad we're having it!
[11:57] jwild Pine: I think that would depend on which pieces were relevant broadly (outside of WMF). probably many are … in which case, YES!
[11:58] Pine Great!
[11:58] Pine We have two minutes left, anyone with any last comments?
[11:58] Pine I'll be posting logs to Meta today or tomorrow
[11:59] jwild I dowould you all be interesting in making this a monthly conversation?
[11:59] Pine Sure, monthly or quarterly.
[11:59] jwild idea generation as things are designed, review as they are implemented?0x1c
[11:59] everton137 if there is a monthly conversation, I think people should bring topics in advance.
[11:59] Bence This was fun and engaging - would be happy to repeat it
[11:59] Pine Yes, we can have a more specific agenda for future meetings
[11:59] Pine But I think this free-form worked well for this round.
[11:59] jwild yea quarterly might be enough
[12:00] jwild +1 pine
[12:00] everton137 Pine, agree
[12:00] Philippe imo.... office hours actually don't work as well that way.
[12:00] Philippe I am very fond of the free forms, because they encourage contribution from areas that we might not consider, without locking you to an agenda.
[12:00] jwild philippe: this wasn't a real office hours - it was more me saying that I would be online to talk about some of this stuff hehe
[12:00] everton137 Philippe, we can wikify hem
[12:00] Philippe But that's just input from the peanut gallery.
[12:00] Pine We can discuss format when we're ready for the next round.
[12:01] jwild But fair point Philippe - we will relabel (not office hours!)
[12:01] Bence Philippe: perhaps time to write down some learning about office hours for the portal
[12:01] Pine Yeah, maybe a different label, something like "Community quarterly review of Program Evaluations"
[12:01] Pine something like that
[12:01] jwild it could be what everton137 calls "coffee time"
[12:01] everton137 office hours is not a good name, too formal. I agree to hack the name too
[12:01] Pine Heh
[12:01] Pine OK, it's 12:01, so I'm stopping the log here
[12:02] Pine Thanks everyone.