The Wikipedia Library/Processes/Partner metrics

Collecting Link Metrics

Link metrics are collected automatically on a weekly basis and graphed on the TWL Tools website.

The Numbers captured sheet of the 'Wikipedia Library Metrics Number of Links' sheet controls which links have their data collected and displayed. To add a new partner simply add a new row at the bottom with the partner name (must be spelt exactly the same as in Partner Flow), URL descriptor (can be the same as partner name if the URL is for the whole website), domain (in the format *.foo.bar/abc/123), and language code. You can optionally add notes if there is something worth explaining about this partner's data.

To control which partners/domains are displayed on TWL Tools, add or remove an 'x' in the Display? column. The live site downloads a copy of this sheet once per day, but TWL staff can log in to the site and force a download with the Reload button at https://tools.wmflabs.org/twltools/metrics.

The Search URL column is not required for the tool.

Comparing link numbers

When reporting link usage the numbers should primarily be communicated via the change since the last report as well as since the start of the partnership. To collect these numbers, open the 'Changes for Reporting' spreadsheet; this sheet is mostly automated, only requiring the designation of which columns to compare:

When adding a new partner, also add a row at the bottom of the sheet, entering the column of the 'numbers captured' sheet corresponding closest to the date when accounts were first distributed to users
When preparing quarterly reports add the column letter(s) for the last update in the Report Column column under 'Last report' and then report the values found in the 'This report' section.

User Metrics

The private project tracking sheet Sum Users Across Wikipedia Library aggregates recorded Wikipedia username information from Wikipedia editors that we collect in the sign up sheets. The only information aggregated is the username information. All tracking materials are private to the Wikipedia Library, and access requires a Non-Disclosure Agreement. We recognize that this current method offers adequately secure, but not optimal, collection and tracking of user information. We are currently designing and developing a Library Card Platform that will offer a more secure environment for collecting these metrics.

Access the respective partner collection spreadsheet
Using the named range function on Google Spreedsheet: name the entire column that contains usernames (even blank cells) "User)
Go to the sum user page, and add an additional column with the partner name. In Row 2 add a "COUNTA" function with the range of the column, and in Row 3 add =IMPORTRANGE("GOOGLEUNIQUE", "Users") where GOOGLEUNIQUE is the unique string for the spreadsheet
Approve the usage of the imported range in this particular spreedsheet (Note this gives permission for all accessors of the central metrics sheet to see the particular individual sheets: neither the GOOGLEUNIQUE range for a particular signup or the Sum Users page should be shared without express permission of the TWL Lead or Project Manager, and a signed NDA with the WMF.)

Partner metrics updates

The MOUs signed with partners include the provision of quarterly updates of the relative usage of their resource on Wikipedia (which we estimate with links, per the information above in #Collecting Link Metrics), and the relative number of users participating (see #User Metrics section above).

Review the private "TWL Partner Flow" spreadsheet's "Account Distribution" table for when the next quarterly updates are due. We try to get the updates out approximately 3 months after the last update or the initial account distribution.
Fill out the draft email below (if you are unsure about the quality of your report email Sam, Nikki and/or the Account Coordinator to find out more, and ensure that the materials are correct)
- Make sure to review the email several times for typographical errors and unfilled parameters
- Make sure to represent the metrics in percentage growth to help the partner recognize the scale of change; this representation of the metric communicates the numbers in a way similar to other reports the partners receive.
- If the partner's accounts are exhausted or close to exhausted, make sure to ask if the partner would be interested in including more accounts. Noting the number of users that are waitlisted allows the partner to recognize the relative demand for the content. If you plan on asking for more accounts, give Sam or Jake a heads up before doing so: they may know something else about our relationships with that partner.
Send the email, with the TWL Partnerships Coordinator (User:Samwalton9) and the volunteer account coordinator for that partnership cc'ed. This allows them to clarify, provide additional commentary if there is something missing in the email, and respond to replies if necessary. Moreover, this provides all coordinators in TWL an opportunity to follow all the correspondence for the partnership, understanding how the relationship is evolving.
If the update email is the second (6 month, or every 12 months thereafter) quarterly update, make sure to remind the account coordinator that they need to complete the access check in with volunteers who have received access. Make sure to track this on the partner flow spreadsheet with a date when this reminder was sent.
If the update email is the third quarterly update, an enquiry into an extension/renewal of the partnership may be appropriate.

Email template

Subject: The Wikipedia Library metrics update - [Partner name]

Text:

Hi [Contact Name],

I have a quarterly metrics update from The Wikipedia Library on the donation from [Partner name] as part of our metric-keeping process.

A graph of the link usage for the [Partner name] domains can be viewed at [link]:

[Donation Domain] links rose by [Change in raw links] since accounts were distributed in [Date] ([Percentage] growth) and by [Change] since the last update in [Date] ([Percentage] growth).
[Donation Domain 2] links rose by [Change in raw links] since accounts were distributed in [Date] ([Percentage] growth) and by [Change] since the last update in [Date] ([Percentage] growth).

The link metrics only account for direct links [on the English Wikipedia], and some citations to your sources may not include direct links. Additionally, we currently cannot say for certain how many of the link additions were made by users with access through The Wikipedia Library, but this is data we're hoping to start reporting soon.

According to our records, we have distributed accounts to a total of [number] users for [Name of Donation] as part of the partnership. Moreover, [status of available accounts and/or waitlist - do we need more accounts or a renewal?].

Do you have any questions or thoughts on the partnership? How has the partnership affected your own metrics?

[Salutation]

[NAME]
[ROLE]
The Wikipedia Library

Check-in reminder text

6-month

Hi [name],

This is just a reminder that it's about time to check in with [partner] users. Please use the guidance at https://meta.wikimedia.org/wiki/The_Wikipedia_Library/Processes/Check_ins if you haven't done so already.

Let me know if you have any questions.

Yearly thereafter

We're at approximately the yearly mark with [Partner] and so need to send out the annual user survey. Please use the guidance at https://meta.wikimedia.org/wiki/The_Wikipedia_Library/Processes/Check_ins and send the previously used survey (this one: [link]) so that we can collate all the responses in one place.