Wikimedia monthly activities meetings/Quarterly reviews/Grantmaking/April 2014

The following are notes from the Quarterly Review meeting with the Wikimedia Foundation's Grantmaking team, April 2, 2014, 14:00 - 17:00 PDT.

Present: Haitham Shammaa, Winifred Olliff, Siko Bouterse, Jaime Anstee, Anasuya Sengupta, Asaf Bartov, Erik Möller, Sue Gardner, Tilman Bayer (taking minutes), Garfield Byrd, Rod Dunican, Tighe Flanagan, Jessie Wild, Alex Wang, Anna Koval, Leslie Harms, Edward Galvez, Janice Tud

Participating remotely: Jonathan Morgan

Please keep in mind that these minutes are mostly a rough transcript of what was said at the meeting, rather than a source of authoritative information. Consider referring to the presentation slides, blog posts, press releases and other official material

Agenda

Data: Grantmaking Overview from Q2-3
Integration: Grantmaking + PED + Education
Global South
Grants Programs: Individual Grants
Learning & Evaluation

Introduction

Anasuya:
Welcome
Biggest quarterly review ever? ;)
third of the grantmaking team new to the department, with PED and Education joining us (that completes the E in GLEE)

a lot of work on putting together the team
foundational stuff around grants for individuals and around evaluation
had a good team retreat last week

Jessie: focus for last Q has been on:

increasing support for individuals
evaluation reports were pushed out
work on grantmaking requirements, burdens on grantees
integration of new team

now in Q4:

individuals
clearer value add of specific programs and orgs
build tools etc. for self-assessment
Global South
grants requirements

Data

YTD: 170 grants in 64 countries
first quarter has highest number of grants because of Wikimania scholarships
location heatmap
80% spent in Q2
plan to spend full $8m budget this FY
Anasuya: overall budget being flexible, possible that FDC / APG less, more PEG, possible that Wiki Ed Foundation might return some of their grant
Sue: that's a normal practice?
Anasuya: yes, but very rarely happens this way
usually orgs ask for reallocation instead
Jessie:
trying to zero in on impact
done some of that for IEG, PEG
want to do a lot more
progress on strategic goals:
had set out 3 priority areas: Global South, individuals, gender gap
exceeding GS % target because of Brazil
under goal for individuals, gender gap
however, for gender gap, only counted grants primarily dedicated to that
GS: only 20% of money, but 51% of grants
gender gap[?]: 94% to orgs

Integration

Rod:
when we talked about integration of Global Ed program in the past, the word "awkward" stuck out ;)
started in outreach, then community, then programs department. now we may have found a home ;)
much stronger base now when working with Learning and Evaluation and Grants
education program is becoming a facilitative hub and grantmaking is about support through resources (money+)
transition of US/Canada complete, now can focus on other parts of the world
very much embedded in Arab world
Focus on Global South strategy and gender diversity within education programs
Synergy between the two teams:

Team and community tools: wikimetrics and fluxx, education can use these tools to help reach different communities and use to help in the classroom
design and evaluation
capacity building, understanding how projects work, learn communicating with education people, a lot of young community members don't have much experience with this
communication- LiAnna departed for Wiki Ed Foundation, and left a big hole for education team communications. Grantmaking team also has communication needs, so need to work on replacing her role.

high schools vs universities
Erik: that's a very broad interpretation of program, right?
Rod: yes, from workshops/individual professors to chapter-run programs
chapter programs are not very numerous, and don't need as much support
Sue: so this goes past the ed program as perceived earlier
Rod: around 20 of them just assignments
Sue: so are we rethinking our approach?
Rod: right now just evaluating
it might be high schools, orgs that have an existing networks of e.g. young lawyers
Sue: but we will try to aim support where there is highest impact, not just everything that is out there?
Rod: yes
Asaf: difference: having a campus ambassador
Tighe: Arab World data
difference to US/Canada: often after school, extracurricular: motivated students joining group, might not even have much impact on grades
our leaders in Egypt /Jordan pushed in different directions, also high schools
Sue: do we still do quality assessments?
Tighe: don't have an established measure, but look at FA numbers
Sue: do we think quality from 10 graders are comparable? In college there are professors reviewing work and making sure they aren't putting anything bad online. Tenth graders don't have that level of review.
Tighe: Gut feeling is yes the quality is comparable.
40% of contributors were tenth graders in Jordan
recall that it's often extracurricular, gifted students
Asaf: on enwiki, could rely on existing assessment system, not available on arwiki
Tighe: lost contractor in Egypt, but still cool results there, 88% women
Jordan smaller, but successful, too. 69% women (even though targeting mostly IT faculties, not arts like in Egypt)
Rod: still working through Saudi Arabia, working on program there
their work goes on CD-ROMs, need to figure out how to get content on WP
we were more involved in day-to-day work when Tighe was still there, now pulling out of that a bit now
In March, attended Global Ed cooperative kickoff meeting in Prague
at Wikimania, a lot of uninvolved but interested people joined, was kind of distracting
Team is 4 people, how can we spread their work out?

Global South partnerships

India

Asaf:
CIS site visit recently
upshot: they executive innovative language strategy, helped by involvement of language experts
multi-faceted: produced video, ran writing contest, ... - trying different things for different communities, gender focus on women scientists as well
at the moment high involvement, thoughts about sustainability/scalability are very much on team's minds
clear growth results in some smaller communities
e.g. ~40 active editors to ~70-80 sustained in Telugu
that's significant
felt changed energy in community
less successful: Hindi
tiny communities like Konkani: infrastructural work (Konkani encyclopedia release) but not yet community buildup
also reacting to community support requests
Erik: what measures did not work?
e.g. Kannada had a big peak to around 200 editors
Anasuya: they had a contest
Asaf: also, videos making rounds in social networks
Erik: duration of CIS agreement?
Anasuya: ends in June, current FDC proposal for 14/15
also, they are pushing content acquisition
Sue: the Kannada encyclopedia?
Anasuya: yes
that's significant, never done before
Kannada also had partnership with a university, not quite ed program like, but e.g. editathons

Brazil

Asaf:
two new contractors who were not on WMF team that transitioned to Acao Educativo grant
these two are doing well in the OER Mapping program. (See first row here: https://meta.wikimedia.org/wiki/Programa_Catalisador_do_Brasil/Planejamento_2012-2013/Parcerias/A%C3%A7%C3%A3o_Educativa/Proposal#Summary_of_activities )
but community passive or questioning
e.g. criticism of the decision to include CC-NC content (not usable on Wikimedia projects) in the OER Mapping project
e.g. no feedback until (or past) last moment
some concerns being raised about partnership model
geographical challenge (team split between Rio and Sao Paulo)
very engaged direct team contact at Acao, but senior leadership doesn't understand the Wikimedia community, and is not yet very engaged online
Site visit and review planned for early May.
Sue: What is the term of the grant?
Anasuya: November 2013-August 2014
Future directions could be partnering with other organizations; engaging and funding the new user group that has recently been recognized,
Anasuya: by now there is a long term relationship of PT WP community and WMF; the community exhibits feelings of having been let down by the fact that WMF did not establish office in Brazil. And not supporting work of the AE team enough
Could either be growing pains (partnership only a few months old) or it could be a problem with the model, we'll be working with Oona and AE to think this through
learning from CIS and AE could be clear guidelines on what kind of organizations we can work with (alignment) in the future
Sue: yes, we discussed that in the past, this might be the answer to the questions we pondered back then

Individual Grants

Siko: How do we distribute resources to reach as many individuals as possible to have the most impact on Wikimedia projects?
not much data yet about what works best
2 hypotheses we're pursuing : 1. microgrants, 2. community organizing
microgrants - problems and solution:

lack of data - what do IE grantees really need? consulted ARWP, result: people want sources
giving money triggers bureaucracy. talked a lot with Legal, Finance: e.g. need to check terrorist watchlist, paperwork (Legal was helpful in streamlining), but still remains a burden, transferring money.
- solution: give stuff without having to send money. e.g book travel
different cultural norms: e.g. about asking for money. solution, from WP library model: satellite page on home wiki. Currently almost everything on Meta. Will set up page on e.g. arwiki instead
limit in number of request staff can process. stuff needs a shipper, finance needs an approver. e.g. WMDE's Literaturstipendium program does around 200 book grants/year. solution from community organizing model : staff supports volunteers in processing requests

microgrants experiments: book requests on ARWP
ARWP library page, global WP library coordinator supports two local coordinators (Wikipedians), they are deputized and authorized to process these requests. two providers: amazon and neelwafurat
"trust in God but tie your camel" (Arabic proverb), - assume good faith, but make sure there are some controls in place so Garfield can sleep at night
need some simple requirements
for supported editors, e.g. minimum edit count (200 edits), link to the reliable source that they want to buy, and how you will use it for Wikipedia
for second request, require link to article that was improved with first book
for coordinators: get debit card (low-limit or prepaid), 1y editor in good standing, willing to sign NDA
when they want to continue, ask them to give a snapshot of well-spent funds
Sue: why the NDA - because they handle addresses of editors (i.e. personal data, like checkuser)?
Siko: yes, that's all
Sue: if I ask for this as editor, will people grill me? Tell me I'm a terrible editor? [troll]
Siko: it's a public process, but no community endorsement required
Asaf: we have not constructed terraces for trolls ;)
Erik: are there numbers already?
Siko: it's not launched yet, planned in two weeks from now
Anasuya: should have sense of impact in June

Siko: microgrants next steps:
Arabic pilot launched in April (April -June), look over metrics in July

measure the outcomes # of books requested, approved, received, number of editors participating and the articles improved, community satisfaction

building something simple actually takes much more time
around 150 hours of team's time so far
Erik: targets? what would a successful pilot look like?
Siko: set low at first, e.g. considering WMDE's numbers (200/year) and small community size
50 requests in 3 months is benchmark.
Erik: still all via wiki-based processes?
Siko: yes, preload template, etc.
Asaf: for next step, might repurpose Czech tracker software (that WMCZ developed for Mediagrant project)
Erik: is that still live?
Asaf: still used
Erik: move over to Tool Labs, use OAuth
Sue: question re "dollars -> bureaucracy": e.g. terrorist watchlist - tied to money, or grant agreement?
Garfield: tied to money
Sue: in last review, we thought about taking more risks, e.g. automatically reimbursing submitted train ticket unless someone raises concerns
Garfield: for reimbursement, we don't have to vet
still have fees for transfer though
e.g. reimburse $20, spend $20 on fees
can absorb that especially in pilots, but long term it's also about the staff time
Anasuya, Asaf: e.g. overhead when asking for ID: they don't have scanner, privacy concerns, bank details wrong
Sue: so we do need ID/terrorist check for train ticket?
Garfield: no for reimbursements, can get around that
scale travel and support grants
stepping up system for small payments right now
Anasuya: pre-approved debit card model much less overhead
Sue: so arWP books is a pilot experiment not for stuff-sending, but for this kind of devolvement to community?
yes
Garfield: tricky to get debit card program approval; right now working around it, but needed for scaling
Siko: also working on streamlining reimbursements, Wikimania scholarships, TS grants. Build better systems that work everywhere
Sue: also looking at this increased authorization for volunteers for travel program?
Siko: too early to say
Anasuya: (bigger picture) in the past worked to involve community in grant decisions via various committes; this is a continuation of that
Sue: so is the take-away: we could take the terrorist check out of the process in general?
Garfield: in some cases, the ability to get receipt to us is a limitation
Sue: technical?
Garfield: sometimes, but not just that
Asaf: e.g. in GS, not everyone can take a quick cellphone picture of receipt
Tighe: or, in Arabic world, got handwritten receipt

Siko:
what we learned about community organizers model of individual grants (IEG)
IEG impact report done by Jessie and team
main thrust:
online community organizing (building platform, testing on small scale, with possibility to scale) = most impact potential
grantees who know local community's needs
how to support:
take a risk, create safe to fail experimental environment, mentorship, lend project management expertise
money for people's time
learned from past mistakes
e.g. in first round, funded development of a tool almost without community involvement, won't get used- good concrete example of a grant type they won't fund in the future
successes involve multiplier effect:

Wikipedia library (1 grantee to 4 community organizers to 1500 editors)
Chinese social media project: WP not well known in China. funded Chinese Wikipedians to set up account on Weibo. Can't be sure about impact in terms of e.g. pageviews, but low cost
Wikisource strategy: a lot of community organizing. WS proofreading is a pretty solitary activity (compared to WP editing), so not a lot of community feeling. Grantees built bridges between language versions, conversations about needs,. outcome: 40 member user group. Contest, spike of new editors, sustained increase of active editors

Anasuya: a lot of GS communities seem to use WS in more strategic way that we imagined, e.g. in India - then use that as pipeline for WP participation
Siko: Next steps: double down on experiments with high impact potential (budget for next year based on renewal of successful grants)
select better IEGs in future: tweak proposal forms, criteria
aim to skew even more to community organizing
Testing group mentorship about e.g. project management (recruit more volunteer advisors)
I served as mentor to a lot of grantees, trying to reduce my involvement, get more volunteers to do that
work on IdeaLab: help turn idea into actionable plan
competition experiments, as it is something that does work in our movement
Erik: long term model for library project - handover?
Siko: lots of potential strategies, e.g. hand over to other group/organization, take in-house, etc
Erik: current money pays Jake's time?
Siko: yes
Anasuya: in general, it may make sense to bring successful project in-house on contract basis
Erik: arWP books project is under same umbrella? yes

--break--

Learning and Evaluation

Jessie:
priorities for Q4:

team integration: in the past constant questions/confusion about distinction between L/E and PED team, now unified voice
program eval: understand program impact
make things easier for grantees, e..g learning patterns library, IdeaLab, track key metrics

Jaime:
published first rounds of evaluation report(beta!)
covered 7 of the most popular programs (types), 119 different implementations
not all had sufficient amount of data
input: $/staff h/volunteer h
interesting that e.g. meeting space not reported as donated resource in education program
output: content added
also editor retention, quality
issue: inconsistency of quality ratings across projects
even where they are available, programs don't necessarily encourage to submit content for quality rating, because of the time lag until rating
provide tools:

tracking and reporting tools
Wikimetrics
survey tools

when polling program leaders, got more reporting from individual volunteers than from organizations
Sue: interesting ;)
Jessie: however, organizations may already be doing reporting for grants and not want to do additional reporting, so this is another useful outcome of bringing the teams together
Tracking and reporting toolkit- what are the 4 things we need to ask for, so grants reports can have pre-outlined fields (these are the important pieces to report, so Jaime has better data)
Jamie: good news: program leaders are excited, care about it, want to know what they are doing
average of 18 minutes for reporting each implementation
biggest challenge: track user names
provided Google Doc/Openoffice Doc they can use
available for translation now
Wikimetrics:
had problem with inconsistencies of CSV files
should become easier soon
Jessie: some of these improvements made to Wikimetrics are not just enhancements, but essential for getting correct data
Jamie: reduce burden
Anasuya: this should help a lot for next rounds of reports, enabling more conclusive results
Erik: looking at reports, there seems to be a lot of options for how to report data (fill out template, fill out doc..), will this be standardized?
Jaime: e.g. Word/Openoffice, tried to provide both for people who can only use one of them
spreadsheet from Google Forms, or own spreadsheet, or print
Erik: hardcopy format seems problematic for scalability
Jaime: The hard copy format is for those program leaders who may implement in the field as a tangibe collection sheet for inputing for reporting at a later time
Anasuya: again, hard to assume that everyone can be online all the time while preparing the report
Jessie: also, e.g situation at event where it's easier to fill out paper survey than submitting online
Erik: OK, but we shouldn't contribute to proliferation of hardcopy reports

Jaime: (on effect of evaluation work:) already seen people using their report to set up their goals
and more experiments, e.g. pairing editathon with classroom efforts
Analytics team's work (with Growth team) on Vital Signs will be huge for us too
blocker: still no access to geocoded data (Jessie will talk about that)
surveys: can offer accounts from our Qualtrics license
working with Wikimedia DC, Wikimedia UK, WMDE, WMNL who make use of this
for those not using Qualtrics, offer survey consultation
for WLM survey, people need more analytics support
also no open source tools
Erik: I have a lot of thoughts on that, so does Jessie
Jaime:
community coordinator gap, but online events still happening, people participating
most watched one was the one on SMART process
Jessie: intentionally timed with FDC proposal timeline
Anasuya: make sure that our capacity building/injections are timed with grantees' needs, be strategic about giving inputs rather than ad hoc
Jaime: participation ranged from 7-13 people
Learning patterns library: currently 34 patterns, Jonathan working on more community contribution
Program toolkits:
probably bring in contractor before end of year to work on adapt Wiki Loves toolkit
two sessions at Berlin conference (Thursday is a brainstorm session)
Anasuya: "blood sweat and tears" was not our title choice ;)
Jaime: but we can measure all three ;)
Followup strategies:
want to survey Budapest participants one year after that workshop
seeing a lot of activity, growth there
Anasuya: internal followup: make sure we have resources to clean up results, have strategy in place
next few months tricky because we will be out much, at Berlin conference, FDC meeting,...
Erik: has there been more momentum about learning patterns (contributions beyond from our own people)?
Jessie: not a lot, but some
supportive feedback about idea itself, "this could be the coolest place on Meta"
Siko: integrate patterns into IEG reporting, also travel/support as optional (give us a link to pattern you created or other outcome like a blog post), idea is that those could be more useful than long reports that no one reads
Winifred: also integrated in FDC reports
Anasuya: people come up with interesting ideas, e.g. Charles wrote a pattern about filling out FDC financial tables
Jessie: did a dogfooding exercise when doing the IEG impact report
Erik: agrees, integration into reporting could be very exciting
Jaime: some technical issues
Jessie:
geodata is major blocker
Asaf: we are talking about the number of active editors per country and language (project)
Jessie: still can access some internally
but we lose:

ability for self-monitoring, e.g. Brazil/India partnerships, APGs in Germany, Austria, and other counties, total >$3m in grant money
ability to make public data arguments. e.g. Asaf made decision to postpone funding a conference in Africa because he could see the lack of editors, but he could not point at public numbers to justify the decision

Erik: was made private around half a year ago
Anasuya: Asaf and Jessie and Toby have been working with Legal on this
Asaf: compare Erik Zachte's data which has been public for a long time
threat model: figure out a particular editor's country from recent changes
need to do bucketing
the necessary coding work was deprioritized recently
can't quite see why the technical effort needed would be that big
Erik: so you ask for?
Asaf: ask for publication of bucketed numbers, so I can point to them
I know it's not top priority, but I also think it's small
Anasuya: I also assume that the new ED will like to see these numbers
Asaf: also, "who are the top N contributors from geography X on ENWP"
Erik: now you ask for much more ;)
Asaf: yes, that's separate. But I want to point to this gap too, wondering when it will be filled
Erik: that's where you get into real privacy issues
Asaf: of course, this won't be public, but could e.g. reach out to these user privately
Erik: skeptical about the severity of the privacy problems re: basic aggregate numbers, but need to look into it more closely
Jessie: example: since FDC is under NDA, was able to share some data with them ("$2000 per active editor in this country")
Sue: to some extent Asaf should be trusted when making that kind of assertion
Asaf: I wasn't questioned, but if you shut something down based on such numbers, it's really much better to be able to point to concrete data
Sue: should at least be able to characterize the data we do have

Jessie:
priority next steps overall: (list)

Discussion

Asaf: will we ever be able to make direct Analytics requests? (small, without tight deadline)
frustrating that we cannot do that
Erik: a year ago, you had Evan, right now it looks you don't have anyone
when we started to centralize Analytics, the thought was that they would support the whole org, not just Engineering
reality is that the team is still building up
keep Grantmaking needs on Toby's mind
perhaps he should hire someone for that
Sue: determine how important is it to Grantmaking? in terms of FTE, how many
Erik: don't talk about persons, but needs
Asaf: I question whether we would even need half an FTE
we have expressed our needs already
Erik: where is the best summary where I can read about these needs?
Anasuya: we've shared that before, but can pull that together again quickly
Erik: Analytics is split into Research and developments teams
want to keep that distinction, avoid that researchers develop one-off tools themselves that won't be maintained
Erik: with Wikimetrics, Grantmaking needs were actually prioritized quite highly
Asaf: yes
Erik: good to rearticulate Grantmaking needs
Sue: so no one asks for dedicated person, but still this helps me to assess staffing needs in annual planning
Anasuya: change one eval person into analytics person? maybe, in any case needs are high

Erik: I like what Siko said about IEG as platform building
bringing in grantees as contractors / staff might change the math though
share e.g. GS success stories/numbers at monthly metrics meeting, also with Product team
caveat on peaks vs. sustained participation
Erik: at every meeting, we talk about technology
could be useful to look at libraries, labs tools instead of heavy, MediaWiki-integrated solutions
e.g. James Alexander's DMCA takedown tool for LCA
just contract that out, I'm happy to help with that
I know Jonathan did some such work
Anasuya: agree, a lot of cleanup work could be addressed with that
Erik: also with regard to the filling out of forms etc. that we discussed
Anasuya: about sharing learnings: had a good cross-department meeting about Global South work yesterday, find coordination points
Quim and Anasuya brought up OPW work, also in GS, could be relevant for this
Asaf: however, worry that we could become the bottleneck
Erik: would defer to Quim on determining how many more mentees can be taken on. it is indeed high touch work, as opposed to hiring a contractor
Sue: projecting into future (in the context of annual plan process)
some of the more recent stuff is really good, needs to be reflected for next year
Arabic project is really good
wish we had 3 pilots instead of 1 - 1 is a small sample size for making conclusions
Anasuya: if we had an additional program officer ... ;)
Sue: re Program evaluation reports, could go into various directions: do reports about 7 more areas, or deepen existing reports
increase number of programs (types) that we will evaluate?
Anasuya: planned 10: 3 new, 7 updated
Sue: possible to set global targets for Global Ed?
Rod: probably, for countries where we do specific work
Sue: other targets are mostly dollar amounts?
Anasuya: yes, but that's where the impact analyses come, will enable other goal setting
Sue: regarding GS target, how much % of that is Brazil?
Asaf, Anasuya: bit more than half
Sue: not a direct concern, but in theory, target could incentivize us to waste money to reach percentage ;)
Jessie: recall grants may be smaller in GS
Anasuya: and, dollars go further there
Garfield: is there work with community to establish basic expectations about average costs for e.g. holding editathon[?]?
there are outliers
Jaime: people are taking hints already
Sue: e.g. for WLM, expenses were wildly varying, right? yes
Asaf: e.g. Canada budget was $2k, ZA was much higher - but there may be differing needs, e.g. difficulty relying on public transportation in a country
Anasuya: might just state range of expenses from past grants
Asaf: pop quiz: country with most pictures per capita? Armenia