Global Data and Insights Team/Movement Data/Equity Landscape/Pilot & Consultation/Directed Review Questions
|Question 1. Considering the output metrics developed to understand Wikimedia presence & growth; are there major domains we have failed to identify?||
The metrics of the pilot and initial public dashboard will be limited to our first round of data exploration and mapping and include:
The domain-level metricsEdit
The project focuses on triangulating our best available signal metrics across key engagement and enablement domains. All metrics are scaled 0-100, averaged across triangulation points, and then scaled 0-100 again so that the global percentile rank is 50 and those scores nearer 100 represent geographies signaling the most engagement in any named domain and those nearer 0 signaling the least. All scores are comparative indices in which no further modeling has been implemented beyond averaging and scaling triangulated signal metrics. The initial domains we have been mapping are:
In addition to the main domain-level metrics, for those domains in which we have robust enough data (Readers, Editors, Grants, and Affiliates) we will also share sub-metrics along two facets:
The high-level metricsEdit
In addition to each of the domain metrics we will share three high-level roll-up metrics:
Additional metric domains under consideration which we hope to progress towardEdit
General Inclusion & EquityEdit
|Question 2. Are there input measures that could be improved from the landscape of existing data or measures which should not be part of the measurement framework for Wikimedia presence & growth?||
|Question 3. Are you aware of any key data sources we have missed for understanding Wikimedia presence & growth?||
|Question 4. What are possible use-cases of these metrics for you?||
|Question 5. Do you have any other thoughts you would like to share with us about the Wikimedia presence & growth measurement framework or the planned dashboard elements?
Here we go a little deeper with design considerations on naming and analysis that are intended to be responded to as they are relevant to you as a potential data partner or user.Edit
|Do any of the proposed project, metric domain, or facet labels present potential challenges to you or your group’s planned projects and partnerships?
|To what extent do the metadata classification systems enable or inhibit potential cross-use with datasets you or your team use to understand Wikimedia communities?
||We plan to catalog the following metadata for cross-referencing across datasets:
|Does referring to equity or inequity resonate more with recent patterns of use you have seen in research and evaluation of inequity in your operating space?
||Referring to equity vs inequity. Some comments have been shared that we should focus on improving equity rather than reducing inequity and labeling our aims and all coefficients positively in this way.
|Kolm categorized absolute and relative measurements of inequality as “rightist,” “centrist,” and “leftist,” depending on their treatment of inequality as an absolute or relative concept. Leftist measures are sensitive to absolute changes; they do not change when all incomes go up by the same absolute amount. An example of this type of measure would be the absolute gini, which is a standard gini coefficient multiplied by the mean of the distribution. Kolm defines centrist measures as measures which show increased inequality when average incomes rise and the relative distribution stays the same, and decreased inequality when all incomes rise by the same absolute amount. Rightist measures are purely relative; when all incomes go up in the same proportion, they are unchanged.
Using percentile ranking relies on a relative view of each group as opposed to an absolute view of capacity. When it comes to measuring change over time, we will rely on measuring the distributions of the input measures. For this we have several options:
For Diversity & Dissimilarity
If you are experienced with any of the above, please share your thoughts on these options via direct comment.
|For scaling and weighting we have identified some key options and are leaning to the options noted below. Please share what, if any, concerns or alternative suggestions you may have.
Scaling: For relative comparisons and roll-up ranking we need to triangulate across more than one input data point to estimate a more generalized global ranking. This means scaling for relative comparison and combining for relative presence and growth.
We considered raw ordered ranks across the distribution, percentile ranks across the distribution, Z-scores, and ratios. With the exception of some background calculations to determine limits to room for growth which apply ratios, we are currently applying percentile ranks 0 to 100 across all inputs for triangulation and calculation of output metrics.
|Weights: Grants dollars must be weighted by the local economy and all input measures must be weighted by some population factor in calculating coefficients of inequity.
We currently propose to apply as follows:
Considering what engagement and resourcing gaps are most important to the strategy work you or your group engage in, please share what excitements, concerns, or curiosities you have about the above weighting options.