Talk:Movement Strategy/Forum/Community Review Report

FORUM metrics should include non-Meta users as distinct group

I think FORUM metrics should include non-Meta users as distinct group. Ideally even more detailed users who did actively use Meta ever (less then 10 edits), who did not use it actively in the period when Forum was activated and maybe compare these numbers to size with other groups that took part in discussions over Forum. --Zblace (talk) 09:37, 20 August 2022 (UTC)

What goal(s) and what question(s) motivate the queries you are suggesting? These queries as they are formulated now are not trivial to produce, but before discussing the details of you suggestion it would be useful to agree what we want to measure and why. Qgil-WMF (talk) 19:46, 20 August 2022 (UTC)

@Qgil-WMF think it would be useful to know how many people would likely not interact (much) with MS or related initiatives if there was no Forum as a channel, as well as how many people find it easier to follow and engage with MS work (although they know and have used Meta).

I do not expect huge number but I am curious how does it compare to the vocal Meta-is-the-only-advocates...

Personally I do not think it should be one or the other, but serving at least two different entry points.

--Zblace (talk) 13:17, 21 August 2022 (UTC)

@Zblace this makes sense, thank you. We are taking note, and we will see what we can do. Qgil-WMF (talk) 13:34, 22 August 2022 (UTC)

Daily Engaged Users

Is it possible to get a breakdown of the Daily Engaged Users statistic. There are three basic problems I have with how this is presented.

• Are the Daily Engaged Users an average (if so, mean or median) over the week, or the sum total for the week.
• The Daily Engaged Users aggregates "likes" and posts. Given that a like constitutes less engagement of writing the post, conflating likes and posts may make users seem more engaged than they are. What are the number of likes versus the number of posts?
• For those Daily Engaged Users, how many are forum admins and (WMF) users posting in an official capacity?

I'm a bit skeptical that Diversity, Equity, and Inclusion are reasonably criteria to evaluate the success of a project, but even if they are, the only concrete measurements we have about how well those goals are being met is the actual level of meaningful participation if the forum. That should have been made more obvious before reaching a conclusion.

I did look at the forum over the last week, and among posts, I counted 44 WMF posts and 48 community posts. Over a quarter of those community posts were about forum errors and embedding content. TomDotGov (talk) 16:07, 20 August 2022 (UTC)

Just for context, thanks to the Discourse Data Explorer plugin, it is possible to create custom queries to the database as sophisticated as the SQL skills of the person creation the query. So far, the metrics we have published come from the reports available out of the box in the admin dashboard (except the Meta - Forum regional comparison). If other metrics are deemed useful and feasible, we can produce them as well. To your questions:

• Up to 30 days, the Daily Engaged Users data is shown daily. For longer periods, it is shown weekly (and then monthly, etc). This is why in this report the data is grouped weekly. Since it's called "average" I assume it is the sum of the 7 days divided by 7.
• Yes, it should be technically possible to extract the data about who has posted on a certain day vs who has reacted but not posted. We don't have this query though, it should be written. The "posting is more engaging than reacting" reasoning sounds... reasonable at first, but it has to take a lot of nuance. For instance, someone just landing on one topic in one day and saying "I agree" or "Thank you" (and then leaving) would count as engaged, while someone reading during a full hour every single new post (or even visiting the forum at different times to read more) and then selectively reacting or not with judgement wouldn't count as engaged. When it comes to dopamine, it is proven that authors get a good dose with those emojis reacting to their posts. There are more examples. Bottom line, the query about people posting can be useful depending on what is the question we want to answer.
• We don't have a query to see what ratio of WMF-nonWMF daily engaged users. Given that all Foundation users have the string "WMF" in their username, it should be possible to create it. We (the Movement Strategy and Governance team) are also interested in metrics that compare WMF-nonWMF engagement. In a healthy forum / community, Foundation staff should be a clear minority. Based on observation and basic calculation, we are not there at all. Admins and moderators currently are 100% Foundation staff, quantitative participation (just numbers) we are at best on par, and qualitative (let's say who starts topics, how lengthy/elaborated the replies...) Foundation staff has probably the majority. We need to look at the trends now that the initial phase is over. We need to see what data can we add in our monthly reports that helps answer this question. Let's see what we can produce for the first monthly report, for this month of August.

Qgil-WMF (talk) 20:32, 20 August 2022 (UTC)

@Qgil-WMF Thanks for that, and also thank you for replying to various posts on wikimedia-l about this report. On wm-l, you wrote "it would be helpful to share the excerpts or the absences that prove this bias."

I'd say something that really seems biased is "In summary, if the MS Forum can maintain and improve the suggested components, it will be deemed successful, irrespective of the number of users." I'm not sure if this is how you mean that to read, but to me it reads like "regardless of what the user metrics say, the forum will be deemed a success. That's certainly not true - zero users, or zero non-WMF users, would certainly be a failure; and I think the presence of the Metrics section show an understanding of that.

I'd be interested in comparing the July and August stats to Wikimedia Space (at its end) and Telegram, to get a feel for where this stands. A big thing is that a forum, like a wiki, has non-zero costs just for existing - it needs admins and support to function. Looking over the past week, those costs (mostly) seemed to be a large fraction of the posts on the forum, but the last week might not be representative.

For the three points:

• Makes sense, and seems inline with what I saw. Thanks.
• While there are a lot of things that can be used, I'd say posting is one of the less ambiguous metrics - it's hard to ascribe motivations to pageviews. I'd go so far as to say that probably the standard should be non-trivial posts (say, more than 10 words long), though that certainly would lower the numbers and tilt them more firmly in favor of WMF staff.
• I concur.

With incomplete data, my current thoughts are the MS Forum should stay open for another 2 months or so, and then there should be a (meta) community straw poll toward the end of that time with a question like "The MS Forum has made valuable contributions to movement strategy." Since that's the fundamental question that underlies success - is the Forum providing value to the Foundation that makes it worth the costs (in time and attention, for the foundation and community) to run it? TomDotGov (talk) 00:42, 21 August 2022 (UTC)

@TomDotGov You make some good points. I'll check with the team during the week and come back with a reply.

As the report says, we plan to organize a survey every 6 months, starting next January. That looks like a natural point to ask about the MS Forum also outside of the Forum, on Meta and other relevant venues (The Movement Strategy channels on Telegram come to mind). Qgil-WMF (talk) 10:29, 21 August 2022 (UTC)

Next steps

Based on the discussions on Wikimedia-l and the Meta talk page (see the links above), we will:

• review the report summary to reflect better the content of that summary in the first paragraphs
• propose a way to include Meta in the periodical surveys mentioned in the report
• review the MS Forum goals and metrics to better define minimum expectations on participation
• address any questions related to Wikimedia Space when the site is back (in addition to Tgr’s reply, see SJs related proposal)

(In case you are interested, I have left more comments in my reply to Wikimedia-l today.) Qgil-WMF (talk) 13:56, 23 August 2022 (UTC)

About propose a way to include Meta in the periodical surveys mentioned in the report. The Movement Strategy and Governance will conduct a survey about the MS Forum every six months, starting on January 2023. The purpose of this survey is to collect qualitative data about forum use and to complement the monthly metrics reports. The survey will ask respondents whether they want the Forum to continue. As a complement to this survey, we will open spaces for free-form feedback on Meta-Wiki, and on the Movement Strategy channel on Telegram. The main purpose of these spaces is to receive feedback from people who aren't using the Forum or prefer not to fill the survey. Qgil-WMF (talk) 07:10, 29 August 2022 (UTC)

About review the report summary to reflect better the content of that summary in the first paragraphs, see this diff. Qgil-WMF (talk) 12:52, 29 August 2022 (UTC)

Thanks. Tweaked. What do you think? Andreas JN466 18:37, 29 August 2022 (UTC)

@Jayen466 It's good, thank you. Qgil-WMF (talk) 06:35, 31 August 2022 (UTC)