Wikimedia monthly activities meetings/Quarterly reviews/Grantmaking/January 2015 -1

The following are notes from the first of two Quarterly Review meetings with the Wikimedia Foundation's Grantmaking team (focusing on the Experimentation & Community Health area), January 20, 2015, 10:00 - 11:30 PST.

Present: Lila Tretikov, Siko Bouterse, Alex Wang, Tilman Bayer (taking minutes), Jonathan Morgan, Sati Houston, Haitham Shammaa, Erik Moeller, Geoff Brigham, Anasuya Sengupta, Garfield Byrd, Janice Tud, Jaime Anstee, Floor Koudjis, Tighe Flanagan, Katy Love, Kacie Harold, Anna Koval, Asaf Bartov

Participating remotely: none

Please keep in mind that these minutes are mostly a rough transcript of what was said at the meeting, rather than a source of authoritative information. Consider referring to the presentation slides, blog posts, press releases and other official material

Intro and overview

Siko: Welcome

slide 1

[slide 1]

This is one of two quarterly reviews for Grantmaking

slide 2

[slide 2]

Agenda

slide 3

[slide 3]

Anasuya:
We're in the middle of a big pivot in Grantmaking
from just giving out money, to giving out money together with a lot of support to help grantees achieve goals
Lila: number of affected people means...?
Anasuya: those we give grants to, and those we had more than one touchpoint with
e.g. participants in grant-funded activities
Asaf: also those that grant programs affect
Asnasuya: yes, that's the second level
so that's a (very crude) measure of impact

slide 4

[slide 4]

many individual grants, but small amounts
these (rows in the table) are overlapping

slide 5

[slide 5]

Background: Overarching goals for Grantmaking, Q3&4
Global metrics will give us much more rigorous data
Lila: 10% increase in participants means?
Anasuya: YoY, per Wikimetrics (it includes those who participate in events, and uses current definitions of active editors as 5+ edits/month)
Erik, Jaime: ...
Lila: based on what?
Jaime: based on reported numbers from grantees for last year
From January on, all grant (reports) have to include Wikimetrics
Asaf: be aware this figure does not refer to the 70k-80k global active editors count. (It includes reported attendees at events, many of whom never edit at all, or do not meet the Active Editor criterion)
Lila: why can't we (calculate these figures) ourselves?
Anasuya: (explains Wikimetrics/cohorts, e.g. WMUK holds an editathon, collects participant account names...)
Jaime: 3 parts: active editors, new accounts created, ...
Lila: so we may have increased participation
might make sense to (anchor?) global metrics by...
Anasuya: we are doing that already
Siko: we don't have a baseline yet because (global metrics) weren't reported so far, but will have it now
Anasuya: and we're in sync with Analytics team on what we ask to be reported

slide 6

[slide 6]

Siko: we decided to split department's QR into two: Experimentation & Community Health / Community Growth

Seed Grants & Experiments

What we did

slide 7

[slide 7]

in Q3, we will have a lot more integrated frontend, story gets simpler
Lila: what percentage spent not on FDC?
Anasuya: about a million US dollars
Siko: last quarter, 130k dollars.
Anasuya: so about 5%
number of non-FDC grants is large, but dollar amount is still a small percentage.

slide 9

[slide 9]

Siko: sourced new projects in Q2
deployed workflow for IdeaLab
main focus this q: Inspire campaign
along some work that needs to continue

slide 10

[slide 10]

Alex: two examples:
Wiki Camp Armenia: outside our normal understanding of who contributes
most important outcome: continued participation (participants initiated wiki clubs in their schools, started edu type programs...)
voted coolest project at Wikimania
Lila: what's retention like?
Kacie, Alex: 18% 2 months later (14% is correct number)
Anasuya: but in general we look at retention after 6 months
Lila: 18% is great
Siko: issue with retention (as metrics): takes a long time to measure
Lila: in Analytics they run predictors (if sticking after two months, likely to remain longer)
Erik: in edu program, retention of professors valued higher than of students
Siko: WMCZ's worth with senior citizens (caveat: this is early, from just after midpoint report)
a lot of us in the movement had kind of written off outreach to this demographic group (digital literacy issues), e.g. after WMDE's pilot
but these guys were pretty smart about it. E.g. they go to senior centers that alrady have computer lab
Facebook ads seem to have worked
takeaway: worth revisiting assumptions
similar to edu program in some ways
Anasuya: communities are not organizing on-wiki as much as we might imagine (i,e they are on FB or WhatsApp...)

slide 11

[slide 11]

(Siko:)
sourcing next projects
Open call to ~600 wikis via global village pump msgs
examples: revision scoring (see Aaron's presentation at metrics last week)
Lila: what's the ETA on this?
Siko: IEGs are 6 months
so final report by summer
Erik: this is about a labs prototype, which will enable [user-facing] implementations by others
Lila: ok, but what about bots etc.?
Erik: that will be up to users
Asaf: wikis will need to do work to implement it locally e.g. on revision page
Erik: it's been very helpful to have Aaron as interfacce between these volunteers and the engineers who work on Services
Siko: so support model here: time from engineers and Aaron, plus money
Telugu catalogue digitization project:
supported with some money and advisor time from Jake (from TWL), as well as from CIS
WikiProject X: organizing effort for WikiProjects on ENWP -- https://meta.wikimedia.org/wiki/Grants:IEG/WikiProject_X
support: money + time from original teahouse project team
Lila: which WikiProjects?
Siko: will depend on community interest (and size/type of project)
Geoff: are you working with Communications on telling stories like these - there are great?
Siko: thinking about this, could still do better on telling our stories
as Comms builds up their capacity, see opportunity
we already encourage our grantees to blog, do some blog posts ourselves

slide 12

[slide 12]

Alex: last year, IEG open call --> Art and Feminism editathons, not very structured yet, but already very successful in uptake
meeting them halfway (so they don't have to fit into one of our models entirely)

slide 13

[slide 13]

Learnings

slide 14

[slide 14]

Siko: some grant proposals come in to a program that's not the one best-suited for them
some get attached to the front door through which they had walked in
Anasuya: the way we got structured is a legacy issue (e.g. the way FDC was set up, ...)
Siko: one way to address this: source proposals one level up, at IdeaLab
problem with e.g. gender gap organizers: always the same group of people, risk of burnout

slide 15

[slide 15]

proposal endorsements are not votes, but an important input on committee's decicsions
improving workflow with simple gadget more than doubled the number of endoresements (with about same level of talk page comments, ie non-endorsement in put)

slide 16

[slide 16]

IdeaLab scale is still untested
spikes in proposal submissions are still during open calls (and the one at the end from mens' rights activists' reaction to gender gap focus)
ideas to improve reach:

focused outreach (by topic)
community organizing
integration with tech

What's next

slide 17

[slide 17]

Things should get a bit simpler with focus on proactive grantmaking in Q3

slide 18

[slide 18]

Focus on Inspire campaign

slide 19

[slide 19]

evaluation will be a huge part of this campaign
priority languages: all languages are welcome, but make sure that e.g. translations are perfect for this set, where we know the're[?] good community organizing
Skud has years of experience organizing such spaces
Lila: how did we choose these priority languages?
Anasuya: let's talk about that in the Global South part

Goals

slide 20

[slide 20]

concern: no baseline for e.g. new women editors (there were some past projects, but without measuring this, and not part of global metrics)
gendered content: based on Max Klein's work (via Wikidata, can assign gender to article) [cf. http://notconfusing.com/preliminary-results-from-wigi-the-wikipedia-gender-inequality-index/ ]
Lila: just biographies, or also other content?
Jonathan: yes, that's the only type so far (and photos of people on Commons). But we're using that as a proxy indicator
Jonathan: (on second column:)
upgrading IdeaLab for Inspire campaign
highlight number of ideas submitted, leaderboard of ideas with most endorsements
task recommendations
these are ways to scale up number of ideas, filter content that one migh want to participate in
want rich pool of potential ideas
Right now it's challenging to measure gender participation, and to capture women-written content
So in grant reports, we ask for numbers on participants' gender
Want to be able to tell more accurate story about impact of these

slide 21

[slide 21]

Siko: what we're learning already

slide 22

[slide 22]

Jonathan: this from Max Klein's work
Lila: changes due to growth or deletion?
Asaf: we can't know
also note that classification (on Wikidata) is ongoing - roughly 40% done, 60% to go
Anasuya: interesting that smaller communities are doing [better on] this now
Lila: if you want to compare, would need to freeze and compare on exact same dataset
can probably abstract away 4th column, but not 5th
Asaf: precisely.

Concerns to monitor

Siko: community concerns that reactive grantmaking is still needed, capacity (see e.g. recent Wikimedia-l thread)
CentralNotice is a blunt instrument
e.g. Meta-Wikimedians dislike "bump" when banner loads [cf. https://phabricator.wikimedia.org/T52865 ]
solution proposed earlier: add code into banner that filters by edit count
but still concern
Lila: so is it a technical concern or?
Siko: no, a cultural one
Erik: let's talk about that - would rather like to fix that one bug than build a whole new notification system
idea to use Echo sounds great, but raises a whole new set of issues
Lila: come up with proposal, figure out how much it will cost
fine with allocating some funding - this is obviously needed

slide 24

[slide 24]

Siko: key needs:
research support for measuring gender outcomes
on-wiki frontend developer:
so far this work has been done by e.g. contractors on an one-off basis. Want to avoid reinventing the wheel
Lila: in my past life, had internal services team for that kind of thing
you and Erik/Damon should talk, develop reasonable model

Community Research: Global South

slide 26

[slide 26]

Asaf:
Biggest opportunity for WMF to affect the largest number of people:
get GS-non-readers who are already online to use the free knowledge we offer (non-readers -> readers)
for other groups, had e.g. workshops, events
but nothing much historically done for this group
suggest awareness campaigns (with and beyond WP0)
for offline people: offline WP exists, but doesn't scale well, in terms of the actual logistics of getting physical media to the people who need it.

slide 27

[slide 27]

Lila: how do you pick these regions?
Asaf: large population size, and existing active editor base that we can work with
Lila: and what about picking languages, e.g. in India?
Asaf: In India, we actually work with several language communities
e.g. Telugu (mentioned earlier)
Hindi community recently got more organized
in Philippines, English much more so than Tagalog
In Indonesia, Bahasa and English, not the others
South Africa: English, besides some possible work with Afrikaans community
Brazil no longer on this list, after termination of catalyst program
still conducting non-priority work there, but concluded catalyst grant
added Ukraine
Lila: it would be great to have a version of this slide with numbers (language breakdown, comparison with last year)
Asaf: OK

What we did

slide 29

[slide 29]

in Brazil, we're kind of back to square one (no longer paid contractor/grantee work) in terms of activity level, but restored community trust
Lila: insights on readership drop in South America? no
what about editor numbers?
Anasuya: it's an uneven map in that region, e.g. Argentina growing significantly recently (but: seasonality)
Asaf: India community consultation
important message arising from the consultation: up to the community which activities (from "roadmap" list) get taken up and supported
CIS has slowly built up credibility

What we're learning

slide 31

[slide 31]

Main lesson from both catalysts: community buy-in is important (this was a design flaw of the original catalysts)
continue doing community consultations

slide 32

[slide 32]

movement orgs (user groups, chapters) are in need of support, benefit from e.g. regular checkins with staff
Geoff: what do you discuss in these check-ins?
Asaf: agenda driven by group
e.g. governance problems, program design, or how to motivate volunteers
Anasuya: we also have calls with individual volunteers, separately
Geoff: are there any differences between user groups and chapters visible in these checkins?
Asaf: not significant ones
important: contact breeds trust breeds initiative
e.g. "I don't know how to use CentralNotice"
without this guidance, groups often resort to obvious and ineffective activities (e.g. giving talks)

slide 33

[slide 33]

community maps
Swahili: a handful of core contributors with discussions among them; each with a cloud of single-interaction new editor communication
Tamil, Arabic: stronger core with internal discussion
also, on Arabic, more non-core interactions
caveat: does not quite correlate to community health (e.g. does not show whether interactions are positive or negative, does not distinguish between discussion and conflict)
Anasuya: Haitham did great work on this, identified contact points for Asaf and for LCA
Lila: should also do semantic analysis

slide 34

[slide 34]

Asaf:
also interesting results from Global South survey
e.g. Arabic Wikipedians do more habitual tranlsation than Tamil Wikipedians
[Erik leaves]

slide 35

[slide 35]

Asaf:
we need to find out more about each community's norms: e.g. how are admins elected? are stubs allowed? what's the bot policy? what dispute-resolution mechanisms are there? How are newbies recognized?
we are going to conduct 90min interviews with core community members from each priority-region community
as result will know e.g. "this community never did press work, but has harmonious editing collaboration"
then address these needs (e.g. workshop on media relations)
Lila: this is [envisaged] for any language (not just English)? yes, for all significantly active languages in the GS priority regions
Asaf: enable deployment of "canned" solutions (ready publications, toolkits, resources) in some cases, custom solutions (mentorship, site visits, project planning) in others

What's next

slide 37

[slide 37]

Goals for Q3 and Q4
example of regional conferences: WikiArabia in Tunisia

Questions and Discussion

Lila: I think this goes into the right direction
I always like more qualifiers on goals
whom they benefit and how
for contributions from GS to English Wikipedia, need to monitor e.g. deletion rates
incentives for enwiki community to diversify content
Anasuya: yes, started doing that kind of segmentation
Lila: enwiki is a different beast
Asaf: wondering about starting conversations, or providing data/research, to inform policies on enwiki
Lila: yes, work on solutions with community members
this is also why identifying connectors is so important
in particular, emerging connectors