Global Data and Insights/Movement Data/Equity Landscape/Pilot & Consultation/Outcomes

There were ten Directed Review Questions and Design Considerations posted for review and input during the duration of the consultation from February through early April 2022. Twenty community members indicated interest in joining the pilot on the talk page and nine entered and completed the pilot process in the first cohort. A tenth individual joined near the end of the original pilot period and the other nine interested individuals never responded to our NDA sign-up next steps request and did not engage further via their talk pages.

We were able to capture and analyze input from the meta portal comments from five community commenters as well as nine pilot participants who shared feedback over the course of our first pilot sessions. Overall, we received 29 comments for inputs out of which six comments could be integrated right away for improvement, ten comments were flagged to be integrated in due time and availability of resources, six comments were held off from integration for further strategic exploration, and seven comments were beyond the scope of the current project and could not be integrated.

Across the input, there were several recommendations for change. We have categorized those according to our original analysis schema and outline those observations along with examples and outcome next steps in the tables below.

Integrate right away for improvement

There were several observations that emerged from the pilot consultation sessions on clarifying terms and schemas, support for data literacy and shared sense-making and potential for data politicization or misuse which we are taking into immediate consideration and design of next steps as we work toward a technical end-point for the first public dashboard.

Observation

Example(s)

Outcome

The politics of data and potential risk of bias

The data can be politically misunderstood and might create disagreement rather than a shared vision of change in direction is needed.

There might be some concern with the expectations around the data and how it is going to be used.

Some people might be cautious of some of the data from certain sources, especially if they are not reliable data sources like the World Bank, for example. It is recommended to stick to intergovernmental agencies, such as the World Bank & the UN, OR to provide the rationale for using some of these sources that gives some culturally appropriate explanation.

(Paraphrased from feedback shared in pilot.)

We are reviewing additional sources as optional replacements for Freedom House’s Freedom Index, potentially using measures from the Varieties of Democracy dataset. We will either replace it or provide an inclusion justification if the index is maintained for triangulation.

Further, we are regrouping at the strategy table on some potential partnerships to ensure a carefully supported roll-out and engagement support related to these observations. Stay tuned for more details by the end of June.

Data Communication & Data Literacy

Include infoboxes directly in the dashboard that can provide brief explanations of definitions inside the tool. (e.g., what an average percent rank is and what criteria is used for maps and categories). This will make the tool easier to understand.

The tool is going to require a time investment to understand what’s going on and how to look at this data.

People need to have data literacy support for working with evidence in decision-making.

This tool needs a good data communicator, otherwise, it might not get used.

(Paraphrased from feedback shared in pilot.)

We will be working to develop the initial public dashboard to include clearer explanations within the tool around these terms and schemas. We have also implemented a scale reduction from 0 to 100 point scale to 0 to 10 point scale in order to simplify the interpretation of meaningful score differences between locations or metric areas.

As above, we are regrouping at the strategy table on some potential partnerships to ensure a carefully supported roll-out and engagement support related to these observations. Stay tuned for more details.

Metric labels

“Count of organizing hubs engaged” might be confusing/problematic, since "hubs" are now being discussed as part of Movement Strategy. Ref

Replaced the "hubs" term with generic "groups" and added to the category of metric labels needing further discussion.

Hold on integration - needs further discussion and strategic exploration

There were also some suggestions to increase the transparency of specific data points and geographic intersections, as well as a repeated theme of wanting more robust inclusion of movement organizers which exist outside the official recognition spaces.

Observation

Example(s)

Outcome

Financial strategy and transparency

Report WMF financial investment in all measured demographic communities Ref 1, Ref 1

We appreciate the detail shared about the data and intersections of interest.

Currently, the project is focused on triangulated scaled metrics; however, we are engaging in conversation and experimentation about how we might safely surface some of these details in future iterations.

Specifically, grants dollars and actual unique editor count are the first raw data points we are in discussion about. However, these data are limited in terms of demographic intersections and breakouts which can be shared.

Metric labels

“Count of organizing hubs engaged” might be confusing/problematic, since "hubs" are now being discussed as part of Movement Strategy … Maybe call them "regional networks" or "affiliates networks" Ref

“Equity” resonates more in my operating space and it is more in play than inequity Ref

Consider and discuss with additional pilot groups the options suggested for "regional networks" and/or "affiliate networks"

We continue to hear that equity is the preferred focus rather than “inequity” but we will continue to explore labeling as we work to expand beta testing more broadly across the community.

Movement organizing

Affiliates' presence: I feel like when identifying affiliates' presence in the movement, the metrics of affiliates that are emerging but unrecognized by AffCom should be presented and this would help in forecasting the direction of the movement in the domain of Affiliates presence Ref 4

Recognition of emerging and unrecognized affiliates Ref 2

All information is tracked through the on-wiki contributions of individuals and groups … is it practical to include conventions, meet-ups, and capacity-building sessions of affiliates since these types of events are also considered as work within the movement? (Paraphrased from feedback shared in pilot.)

The purpose of the program leadership domain separate from affiliate and grant leadership domains is intended to provide this signal about other movement organizing activity.

However, we do not have very robust geobased data on unrecognized and or volunteer organizing happening outside of the grants and affiliates monitoring data which is only tracked for Foundation grantees and recognized affiliates.

We hope to improve the metrics available to input to the program leadership domain as a movement organizing domain beyond the bounds of grants and affiliate recognition.

We are currently examining the Program & Events Dashboards and the Event Metrics tools as potential solutions to help here.

If you are aware of existing datasets that would contribute relevant signal metrics, please share about them on our talk page.

Hold on integrating - currently beyond the project scope and resourcing

There were also some suggestions to adjusting the mode of consultation for more community inclusion as well as suggested metrics for accessibility, new editor retention, admin infrastructure, non-affiliated movement organizing, incubator projects, and more granular data sharing specific context. Many of these suggested metrics have been on the radar, however, they are still beyond the current project stage and resourcing. We share highlights in the table below and continue to await opportunities to pursue improvements in these areas as better data pipelines become available.

Observation

Example(s)

Outcome

Self-organized Wikimedia community conversation is missing

The Wikimedia Foundation should support regional and thematic Wikimedia community groups in asking questions to their own communities, and those communities can discuss among themselves in the way that is natural for them.

The Wikimedia Foundation should greatly lower the barriers to access for support for communities to organize their own conversations about social and ethical issues which guide movement strategy. Ref

We understand and appreciate this challenge. We purposely designed the pilot and consultation as a dual effort with an immersive pilot in addition to the open general consultation on meta due to the fact that many of the consultation points for future design decision-making are rather niche and piloting is aligned to the use cases which the initial metrics schema is aligned.

We continue to enroll community members in the pilot for this style of pre-alpha dashboard engagement. Once an alpha dashboard is made available, there will be a continuous feedback mechanism in place. In addition, we are currently at the strategy table discussing potential partnerships to ensure a carefully supported roll-out and engagement support related to these observations. Stay tuned for more details.

Admins

Main Domain: I feel that administrative rights should be considered there and for my community, there are no Admins despite of the many experienced Wikimedians. Ref 3

We are hoping to be able to develop a geo-based metric capturing the signal of those with administrative rights and activity, this among the wishes for metric improvements.

Accessibility / Disability

People living with disability could be carefully considered and onboarded successfully in the movement. Ref 3

Diversity, Equity and Inclusion: the ecosystem seemingly ignored and underrepresented. - Accessibility could be included in planning, executing and reporting projects. - Most Projects particularly Wikipedia lack accessibility features making it almost impossible for people with disabilities particularly the blind to be included into the community. - The lack of adequate accessibility features show u from sign up, creating an account to editing, their inability to use screen readers to edit or the provision of adequate guidance on how to navigate the interface. Ref 2

We are hoping to develop better metrics in time here related to contributors, however, we have no available geo-based data at this time specific to disability or accessibility beyond internet and mobile access.