Wikimedia monthly activities meetings/Quarterly reviews/Wikipedia Zero, 2013-02-20

The following are notes from the Quarterly Review meeting with the Wikipedia Zero team on February 20, 2013.

Present: Tomasz Finc, Kul Wadhwa, Frank Schulenburg, Amit Kapoor, Dan Foy, Sue Gardner, Erik Moeller, Tilman Bayer (taking minutes)

Participating remotely: Brion Vibber

Please keep in mind that these notes are mostly a rough transcript of what was said at the meeting, rather than a source of authoritative information. Consider referring to the presentation slides, blog posts, press releases and other official material

Proposed agenda

1) Program Vision
2) Strategy
3) Year-To-Date Snapshot
4) Highlights
5) What is Wikipedia Zero (product)?
6) What We Learned
7) Where We’re Going
8) Groundswell Impact
9) Open Questions
10) Measuring Success
11) What We Need
12) Q&A

Presentation and discussion

slides

Kul on Program Vision & Strategy:

Scale as quickly as possible, first sign up operators with greatest reach
"Partner access" - mobile users with access: closing in on 410 million subscribers
Actually launched to date: 12 countries, reaching 69 million
Only part of them use data access
Most recent: Botswana, next: Russia (big)
Uptick in monthly pageviews
Seems to be gaining traction some months after launch
Partner pageview growth:... larger than WP mobile in general<

Dan on technical challenges that were overcome:
Testing is different than usual, because it depends from which country one accesses WP
Reduced launch time (alpha -> production ) from 2 months to 2 weeks
Issues with Opera Mini (widely used in developing world)
They route traffic through their proxy, so we couldn't identify partner traffic
Met with Opera people
Worked fine in-house, partners beta-tested remotely
Tomasz: Opera supports some Indic languages (e.g. Kannada) that users wouldn't be able to use otherwise

Amit on "Globalizing the Mobile Experience"
Language choice on landing page (non-WP0 just see English TFA when accessing wikipedia.org), catering to the bilingual usage pattern in those countries

Kul: People do want the local content
Lots of media traction is starting to make our job easier in terms of engaging and education potential partners

Amit: country success stories: pageview growth for Orange Niger and Ivory Coast
Erik: Opera Mini traffic included there? Amit: yes since January. Previous numbers include conservative Opera estimates.
Amit: Set up dashboard for us and the partners. Partners don't allow us to make all of the traffic data public

Kul on WP0 UX
Demo by Dan:
Banner on top: "Free Wikipedia from Wikimedia", links to a portal by partner
Important indicator that page is free
Images not loaded, but can choose to view them (with fees)
Warning "Standard data charges may apply...", also when following external links
Frank: how do people continue using their phone once they end up being outside the free-of-charge system? Do they go back to the place where they got information at no charge (i.e. WP)?
Amit: need to go back. people are quite familiar with how to browse

Kul: difference in user behavior - many have smartphone but don't use data, or only Facebook.
Erik, Sue: how about give WP0 to people who don't have a data plan at all?
Kul: for partners, one value of WP0 is getting people used to using data
Amit: existing example :...
But hard to communicate in advertising etc (with instructions on how to use it)
Kul: lower pageviews of WP. Investigating why (less local content?)
Growth takes time
unique users aren't coming back after launch (in several countries)
Amit: Tunisia example: many customers used Wikipedia one time and did not come back
Possible reasons?
Not a lot of locally relevant content. Example: Swahili WP in Kenya
Showed partners WikiProjects (e.g. WikiProject Uganda) to demonstrate to partners that there actually is local content
Kul: Access slow, people often want to see snippets, but Wikipedia is pretty long-form
Sue: (recalling CBC.ca) - could also work the other way around, with users willing to wait once for a long text but not repeatedly for snippets.
Erik: In US and Europe, large part of traffic comes via search. Haven't really optimized our own search much, which could be a reasons for users not coming back. Do we know about difference in search behaviour in these countries?
Sue: Does Google search give WP0 results? Kul: No for zero.wikipedia partners. Yes for m.wikipedia partners (most current launches)
Frank: TFA as landing page not enticing (unless article's topic is locally relevant)
Sue: Natural search traffic has been really important for WP, if WP0 loses that, that's significant.

Kul on where initial estimates were off:

• data usage percentage in partner countries was smaller than average for all developing countries (e.g. China drives up that average) - 13% vs 10% actual
  
• Percentage of Wikipedia users among all Internet users much smaller in developing country mobile users (global 35% comScore, Annual Plan assumption:..., 12% actual)
  
• Cultural response to "free" can vary greatly by country, e.g. "free" made much greater impact in Uganda than in Cameroon (free = cheap?)
  Erik: obviously, local market conditions differ a lot too. Ideally would have a data driven approach to decide where to invest
  

(Amit):

• page views per reader were estimated too high (40 for existing readers on desktop per comScore, AP assumption 12 per mobile., actual closer to 7)

Kul: Learned about partners:

• Partner capabilites vary greatly on every level. For some of them, have to call them every day.
  
• Markets differ: India very competitive, low margins. Brazil: local affiliates eager to preserve margins, reluctant to offer anything for free
  
• Managing partners is delicate. Contracts don't equal compliance.
  
• HQs don't control affiliates.

Where we're going:

• Improve operations, faster launches
  Dan:
  • reduce # of people and departments involved
  • standardize processes

Sue: does technical work involve the mobile site itself too?
Dan: yes
Erik: wonder if we should phase out zero.wikipedia.org (as separate offering), adds complexity, low traffic
Kul: partners ask for stripped down version
Dan: tech work already done
Erik: Potential end user benefits (no-image version) should be considered separate from whether we accept it as a singular partner offering
Frank: goal for launch time (from currently 2 weeks)?
Kul: We plan to reduce the launch time but also do more launches in parallel, 5-10 per month instead of the current 1 - 2

• Technical debt

Dan: bugs/features backlog (2 months), add APIs for the partners - secure website where they can configure (2 months)

• Scale ("every single person on the planet")

Kul: 29 launches to go on current deals
Erik: Public list of countries / launches? https://wikimediafoundation.org/wiki/Mobile_partnerships#Where_is_Wikipedia_free_to_access.3F
Dan: portal for partners: configuration, dashboard, marketing assets
streamlined signup, network settings
Kul: activate the community to promote / and ask about WP0 locally, like the South African students' letter
want to influence marketing more
other internet companies that run mobile partnership programs tend to divide partners into three categories, if by size. we don't have as many resources so we plan two categories (large >30 million subscribers/smaller)
we spent 60-70% of our time custom-managing large partners, but for smaller partners, want more standardized approach
we get requests from operators as small as 50k
Smaller partner market share 2.6billion vs. 3.4billion subscribers of top 20 operators globally
Portal has three audiences: general public / partners / movement
Sue: merge movement and general public?
Erik: branding - we use WP0 for all of it now?
Kul: had lots of discussion with Communications(Jay): Zero means free. FB0 problematic for lots of operators
Sue: agree we should phase out zero.wikipedia.org, also because it lowers our proposition
Noticed in Australia: everyone loves the idea, but expect it to be the same and very simple. distinction dilutes the value
Kul: partners' main concern is often conserving bandwith, images take a lot
Frank: how many partner might we lose?
Kul: maybe half of them
Dan: It's also about meeting them technically, demand for text-only
Sue: If so, should have that for everybody
Tomasz: we already have that: the image-free version of m.wikipedia through link at bottom
Sue: should not implement separate version solely based on needs of mobile firms

• Other delivery channels

USSD demo by Dan:
In chat interface, enter search term ("India"), drill down via numbered menu choices ("...3. History 4. Geography ..."), then request actual article content per SMS
Kul: try to get partners to do first one for free, difficult (this is cash cow)
right now: first 3-4 SMS free
Sue: would be only for wealthy people
Kul: imoprtant becaue: 87% of dev countries mobile users don't use data, 80% dont have phones with modern browsers
Erik: worried about diluting our brand (free knowledge), need clear path
Sue: Any friction (including insecurity about cost) prevents usage, be cautious about these charges. our service (general knowledge) should never be paid for. what if the article is a stub?
Should we subsidize it ourselves?
Kul: argued a lot with people already. Free SMS packages?
Sue: suggest rule: don't do in situations where people have to pay for text
Erik: agree, user could rack up huge bills
Frank/Sue: frame as "gateway" to data plan? ;)
Amit: disagree a bit, SMS does incur variable cost for operators
Erik: USSD launched yet?
Kul: not yet but soon, plans with 3 first SMS free
Erik: drill down menus (search) free? yes, only SMS with content costs
Frank: Is that kind of billing easy to understand for users?
Kul: we don't know
Dan: that's what we learn by doing pilots
Sue: how about small pilot where we cover the costs as an experiment? how expensive could it be?
Erik: important theme here: not introduce non-free versions of WP. may lose some relationships. If they worry about costs, we may cover it.
Kul: could do such a pilot
Sue: I am uncomfortale with us entering into partnerships in which people may be charged for some of their Wikipedia usage. Even if we ensure signposting is 100% understandable, we still run the risk that a user will misunderstand. We don't want to see newspaper stories about someone poor in a developing country who accidentally chalked up a crippling phone bill due to Wikipedia usage.
Dan: we are really careful about those warning banners on WP0. USSD service clearly states the price.
Sue: Recurring theme in conversations about WP0, e.g. with journalists: What's in it for mobile firms? Partly it's offering Wikipedia as a kind of loss leader to encourage people to use their data services, and also it's corporate social responsibility. The trouble with the corporate social responsibility aspect is that it leaves mobile firms with no incentive to actual market the service or increase its actual usage.
Amit: (on subsidy pilot) Operators surely haven't done it before...
Kul (continuing with alt delivery): Apps for feature phones

• Accelerate usage

Research, Awareness, UX/features, Efficient management
Erik: do you need UX resources?
Kul: not consistenly, but some
grassroots: ZA students letter
beginning of a movement: letters from officials who want WP0 for their country, awards

• Measuring sucess:

pageviews (-> money saved. but that is hairy in terms of net neutrality and accuracy)
unique visitors
others: free vs generated (indirect through exposure) pageviews, local language readership, active editor count in those countries. anecdotal: frwp spikes coinciding with launches, but unclear if there was a causal relationship
Sue: focus on driving unique users. Important difference between users (in countries where they are likely to) have alternatives for accessing information, and those who don't.
Frank: do people know what the product is - "online encyclopedia" in countries where most people are not accustomed or don't have access to printed encyclopedias?
Sue: we try to be close to a search engine. Wherever people are expressing a desire to find out information, that's where we want to be. We're not survival information (news, weather, traffic) -- we are reflective, context.
Erik: Realistically, not going to have our own uniques soon (technical and privacy challenge)
Tomasz: If we had a mandate and permission to set cookies could get them relatively fast
Sue: mobile walled gardens are a moment in time, won't last
Sue: going to actually measure pageviews, but they are just a proxy for uniques, which is a better measure of public service
Erik: large investment (growing team) for a few million pageviews, how to determine success vs failure in long term?
Sue: know now that initial estimates were largely off, for reasonable reasons
Recalibration with board, "who needs us more" conversation
don't want WP0 as marketing scheme (getting accolades but not actual usage)
get closer to real target audience, would be easy to backslide towards audience that already can read us anyway
Erik: set numbers for July 2014?
Sue: now is an opportunity to take stock. We now know some (not a lot) about how people in developing countries are using their mobile phones to get information. We now know a lot about mobile firms: they can be slow to make decisions, slow to execute, their goals are of course not 100% aligned with ours, they skimp on marketing for a variety of reasons. We know that the WP0 deals take a long time to bear fruit: first is the negotiation of the deal, then the technical implementation, then the marketing to the end users, and only after that final step do we even begin to see results. We have a backlog of signed deals, a mix of zero.wikipedia and mobile.wikipedia, waiting to be executed upon, the execution is the choke point. Therefore it seems to me that if zero.wikipedia is less desirable from our vantage point, and signing-of-deals is happening faster than executing-of-deals, then we should stop negotiating zero.wikipedia and focus solely on mobile.wikipedia. USSD is fine, but users should not be paying. we should focus energy on the most promising areas
Erik: Still orders of magnitude below overall usage. path to scaling?
Kul: some countries might be phenomenal successes
Sue: two schools of thought: 1) any progress is great no matter what it costs and 2) the goal is to move the needle in a big way. okay to experiment but we need to know what success looks like. tiny incremental change is not success. Key questions about this program: can it move the needle? Can it scale? This is why the WP0 portal is interesting, because it could make it scale, could bear fruit while we divert resources to other work.
Frank: need clear number
Kul: or combination of numbers
if we could get to 10x in next few months, I would say it is working
Sue: Maybe worth experimenting going full-on with one firm, to see what a best case scenario could look like. what if we did a pilot with a chapter, asking them to advocate for wikipedia zero in their geography and to promote it via the media?
Dan: Measure improvement wrt developed world, or (...?)
Sue: Board: easy/organic growth in e.g. Germany is done, notion that remaining work is less organic. on the other hand: mobile explosion in dev.world
Erik: We would then have to deliver numbers in that frame of thinking (billions not millions). Need to make the case that the program is worth to have in that frame.
Sue: Reminder about ongoing Annual Planning process, goal-setting

• Needs: (Kul)
  

FTEs (marketing, relationship manager), contractors (USSD/SMS implementation/software, feature phone dev, research - maybe on the ground-, UX/product, portal design - already some budgeted, start involve Heather)
Sue: who does mobile UX currently?
Tomasz: Vibha and Munaf do, but not WP0
Sue: WP0 banners should be done by the same people, consistency. and mobile designers should be connected in general with the WP0 work so that they benefit from whatever WP0 learns about mobile usage in developing countries, and can integrate those learnings into our general mobile offering
Erik: depends on how much work it is
Sue: question re research: would we commission research or do ourselves?
Kul: local contractors
Frank: in the past, the outcome of our own research efforts wasn't very convincing
Sue: we shouldn't do research ourselves or commission custom research -- most of what we need is the same as everyone else's needs, so we should be able to buy existing research. (User testing is different.)
Erik: article-level analysis?
Amit: done some in 3 launched countries

Tilman: Communications aspects?
Kul: Coordinated a lot with Comms team. Emphasis on reducing barriers. Not a lot of pushback on net neutrality and censorship apects (even though we can't satisfy NN purists)
Sue: Responding to media requests about whether we offer partners customzed version, I emphasized we don't change content
Erik: Most people understand that it's a legitimate tradeoff between net neutrality vs. access to education as a human right, as evidenced by public recognition of the program

Amit: marketing via centralnotice targeted to countries
Erik/Kul: focus areas...

Erik: Evidence that there are UX issues with mobile search?
Amit: they wouldn't be different from desktop
Sue: Won't move the needle if we do too many different things at the same time. don't focus on "walled garden" UX, solutions that work for all are going to scale better. Team should focus solely on making mobile.wikimedia available for free for people in developing countries.
Amit: This would be a useful mandate to internalize for team
Sue: kick things over the fence to Tomasz' team
Erik: WP0 has specific mandate to remove cost barriers to free access. Myriad of things we could to do improve access in so many places
Sue: not resourced to do market-specific things

Erik, Sue: this was great, thanks