New Readers/Raising Awareness in Mexico/Short-term Impacts

This page summarizes the findings regarding the short-term impacts of the Mexico video campaign to a custom landing page and then onto Wikipedia via that landing page, per phabricator:T210813.

Details

• Dates for analysis: November 15 - December 16
• This month covers the ad purchases, with most of the spend happening from 11/21 - 12/5 and then again 12/13 - 12/14.
• The campaign focused on the videos advertised via Youtube and Facebook, where the total number of paid views (which comprise >96% of all video views) was 2.3M and 3M respectively.
• For landing page statistics, I use Piwik. For log data regarding Wikipedia page views, I use the webrequest table and focus solely on the views that can be geolocated to Mexico (1393 out of a total of 1518 unique devices) so as to focus on the traffic most likely generated through the campaign (as opposed to e.g., WMF employees).

Results

Click-through rate between pages

We see substantial drop-offs both between views of video -> landing page, and then again with landing page -> Wikipedia.

• Video to landing page: per Piwik, the 5.3M views from the campaign resulted in 35,888 views on the landing page, which is less than 1% of video views resulting in a visit to the landing page.

• Landing page to Wikipedia: Of those 35,888 visitors to the landing page, only 1393 (4%) from Mexico clicked through to Wikipedia, making the landing page the ending place for the vast majority of traffic. Of these 1393 visitors, most went to the Música de América Latina page (1191, or 85%), with the remaining going to the Main Page.
• Landing page to app: Of those 35,888 visitors to the landing page, we also did not see many app installs. There were 340 (1%) clicks on the install links but no noticeable bump in daily installs from iOS or Google.

New vs. Return Visitors

We also find that most of the Wikipedia visitors that can be attributed to the campaign are return visitors.

• Mostly return visitors: Of the 1393 devices that did reach Wikipedia, only 34 (2%) of them were devices that had not visited in the last month (with the vast majority already having visited that day). This suggests that most people who click-through are already familiar with Wikipedia.
• Expected levels of exploration: Of the people who click-through to the Main Page or Música de América Latina Wikipedia articles, the vast majority do not read any further. For example, we see a total of about 100 page views to other music-related articles from the 1191 people who visited the Música de América Latina page. This is not atypical, but it would have been nice if we saw evidence of exploration beyond the initial page.

Impact on traffic

The views to Música de América Latina did not have a noticeable impact on daily traffic. As can be seen from a chart of page views (permalink), which compares the views to the Música de América Latina article during the campaign to the views to other es-wiki pages of the form Música de..., there is no clear bump in traffic from the campaign.

Miscellaneous

• Piwik's geolocation was not fully configured: The Piwik portal indicated that a large proportion of traffic to the landing page was from the United States (55%). This turned out to be almost certainly due to not fully configuring Piwik's IP geolocation capabilities. Piwik by default uses browser language to geolocate visitors to a country, which is very noisy. In the future, it can be configured to look-up the IP address with MaxMind's GeoIP2 databases (see the docs), which should be substantially more accurate. From our internal webrequest logs, 92% of the devices who reached Wikipedia were geolocated to Mexico via their IP address, which is further proof that this was lack of configuration as opposed to real traffic from the United States.

Conclusion

Though the video campaign may have generated awareness (and potentially increased long-term traffic) in Mexico, this analysis of short-term impact suggests that the videos themselves did not lead to substantial page views based on people clicking on the links embedded within the videos.

Queries

/* In order to collect all of the landing-page-related page views to Wikipedia, I used the following query. Due to the overhead of this query, I actually did this for each day individually and used INSERT INTO instead of CREATE after the initial table was created. */

CREATE TABLE isaacj.bienvenida AS
SELECT REFLECT('org.apache.commons.codec.digest.DigestUtils', 'sha512Hex', CONCAT(client_ip, user_agent, 'HASHKEY')) as uid,
       CONCAT(LPAD(year, 4, '0'), '-', LPAD(month, 2, '0') , '-', LPAD(day, 2, '0')) AS date,
       (x_analytics_map['WMF-Last-Access-Global'] IS NULL) AS new_user,
       pageview_info['page_title'] AS pg_title,
       geocoded_data['country'] AS country
  FROM wmf.webrequest
 WHERE year = 2018 AND ((month = 11 AND day > 14) OR (month = 12 AND day < 17))
   AND referer LIKE "%bienvenida.wikimedia%"
   AND is_pageview
   AND agent_type = 'user'
 ORDER BY uid, date
 LIMIT 100000;

/* In order to collect the page views that are associated with the devices that reached Wikipedia via the landing page, I used the following query. This query was executed for each of the days in the analysis timeline. */

CREATE TABLE isaacj.campaign_requests AS 
SELECT w.uid, w.pg_title, w.last_access, w.country, w.dt
  FROM (SELECT REFLECT('org.apache.commons.codec.digest.DigestUtils', 'sha512Hex', CONCAT(client_ip, user_agent, 'HASHKEY')) as uid,
               pageview_info['page_title'] as pg_title,
               x_analytics_map['WMF-Last-Access-Global'] as last_access,
               geocoded_data['country'] as country,
               dt as dt
          FROM wmf.webrequest
         WHERE year = 2018 AND month = 11 AND day = 16
           AND is_pageview
           AND agent_type = 'user') w
 WHERE w.uid IN
       (SELECT DISTINCT(uid)
          FROM isaacj.bienvenida
         WHERE date = '2018-11-16');

/* In order to count the page views associated with landing-page readers from Mexico, I executed the following query. */

SELECT pg_title, COUNT(DISTINCT(campaign_requests.uid))
  FROM campaign_requests
 WHERE campaign_requests.uid IN
      (SELECT DISTINCT(bienvenida.uid)
         FROM bienvenida
        WHERE country = "Mexico")
 GROUP BY pg_title;