Talk:Community Insights

Active discussions

Costs for this surveyEdit

Hello,

how many money does this survey cost. And why do you use a external company for this survey. I think the most questions of this survey are multiple choice questions and there it does not take a long time to know how many persons have chosen what for a answer and how were the questions translated. I think there have been translated some things into German where it doesnt make sense. For example Village pumps was translated with Dorfpumpen. This is a translation but does not exist in the German Wikipedia as a Diskussion site. Is it possible to get the text of the questions in German so that I can check the questions and tell you when I think that another translation of a word were good. -- Hogü-456 (talk) 18:30, 10 September 2019 (UTC)[]

As Hogü said, the translations are terrible. For example there is a question about how I deal with new users and one option is "I post a message on their talk page, using a barnstar or a predefined message" while the german translation is "Ich poste eine Nachricht auf ihrer Talkseite mit einem Barnstar oder einer vorformulierten Nachricht." The translation is very close to the original, however there is one problem: Barnstars don't exist in the German Wikipedia. Another example: Ihe part where I am asked where I am from I can select a country from a list which is sorted alphabetically in english - but obviously is a mess when viewed in german. "Deutschland" can be found under "G", "Österreich" can be found under "A". It works in english but not if every entry in the list is translated individually. It's obvious that this survey is made by enwiki users for enwiki users. I hope that the surveys taken are not all grouped together but are instead analyzed in seperate groups depending on the language the survey-taker used. On another note: The final part asking about demographic data is pretty weird. For example the question about gender has the options "male", "female", "transgender", "non-binary". Sooo apparently trans people are neither man nor woman? And when asked about age the minimum available is 18. Are people below 18 not allowed to participate on Wikimedia projects? Both of these don't affect me (I am neither trans nor below 18) but I wanted to point this out anyways. ーŒ̷͠²ð·¨´´̢́̕͘³͏¯̞̗ 【🅱alk】 17:44, 21 September 2019 (UTC)[]
Hello Œ̷͠²ð·¨´´̢́̕͘³͏¯̞̗! We are aware that there are some issues with the translations-- they were completed by a third-party service. We had hoped to have volunteers make corrections to the survey translations, but as this is such a large request many of the languages did not get reviewed, and German is one of them. We are manually fixing errors (like the one mentioned at the top of this page) as we can, but some we are stuck with--in particular the problem of alphabetization you refer to. We intended to make the list of regions and countries alphabetical in each survey language, but it is not possible due to the survey infrastructure. Please know that this is a known issue, and we are looking into ways to fix it for next year--same goes for overall translation quality. As for the gender question, we ask respondents to select all that apply in order to make the question as inclusive as possible. For the age question, we are bound by legal restrictions on surveying people under 18 without parental consent. I appreciate your feedback! I do have one question for you-- is there another "vorformulierten Nachricht" that we might use to substitute for "Barnstar" in the German version of the survey? I'd like to fix the error you are pointing out, but I'm wondering if there is an example of a "predefined message" I can substitute that is more relevant to German Wikipedia. Thank you again for your time and feedback! --RMaung (WMF) (talk) 22:06, 27 September 2019 (UTC)[]
@RMaung (WMF): Thanks for your reply and sorry for my late answer. There are several templates in dewiki to greet new users, see de:WP:Begrüßung#Standardisierte Begrüßungsvorlagen. I think the phrase could be translated as: "Ich hinterlasse eine Nachricht auf seiner/ihrer Benutzerdiskussionsseite unter Verwendung einer Begrüßungsvorlage." (that would be the literal translation of "I leave a message on his/her user talk page using a greetings-template.")
Oh and about the age thing: if you are not allowed to survey people under 18, why don't you just add the following option: "I am below 18 and my parents or guardians are answering this survey for me" That way kids their parents of course know what to pick and you got your age data in a totally legal way. :P Then again adding options to questions after some people already answered them is not fair, however in the next survey this could be an option. ーŒ̷͠²ð·¨´´̢́̕͘³͏¯̞̗ 【🅱alk】 15:57, 5 October 2019 (UTC)[]
If the software does not allow to use different alphabetical orders for different languages you might want to use international codes in future, like "DE - Germany", "DE - Alemania" etc. Respondents should be able to find their own international code in a medium (internet) where such a code is commonly used. → «« Man77 »» [de] 16:38, 27 October 2019 (UTC)[]

QualtricsEdit

If you want your data to be representative, maybe you should choose another service provider. Qualtrics has sanctioned all those living in Iran. 4nn1l2 (talk) 03:56, 10 October 2019 (UTC)[]

@4nn1l2: I realize this response is coming incredibly late, but it is still relevant to this year's survey: Qualtrics does not block Iran, it is the result of the political relationship and resulting data embargoes between the governments in Iran and the US (and thus any survey that originates in the US, no matter which provider we use). We are hopeful that the situation will change in the future. RMaung (WMF) (talk) 15:16, 9 October 2020 (UTC)[]

Survey designEdit

The failure of this survey to explain to ordinary editors what "...implementing your project or scheme..." actually means made it virtually impossible to answer one whole set of questions. Having then moved forward through the survey to try to understand the context of those questions, I found there was no back button (just before the personal info section.) It was thus impossible to go back and give proper answers. I'm now feeling I've spent 25 minutes answering a survey that's not applicable to my level of editing activity on en-wiki only. Nick Moyes (talk) 20:08, 17 October 2019 (UTC)[]

2019 Survey?Edit

What is the current status of the 2019 Survey? Antrocent (talk) 06:16, 13 February 2020 (UTC)[]

Confusion in questionsEdit

Hi,

@RMaung (WMF): I'm not sure where to provide this feedback, so I'll be putting it here. As I'm answering this gigantic survey, it's sad to see that again there's confusion on terminology - some on rather important questions. Especially:

  • "For the Wikimedia projects you participate in, how much do you agree or disagree with the following statements about user conduct policies?" It is unclear what is being referred to. I would guess community policies, but the question is asked right after a question about WMF's policies ( Code of Conduct for Wikimedia technical spaces, Trust and Safety) and right before Wikimedia Foundation’s Friendly Space Policies. It's likely that different respondents interpret this question differently, with a bias based on how much they dislike WMF's policies.
  • When talking about actions towards newcomers, one line states: "I post a message on their talk page, using a barnstar or a predefined message" which is a super wide array of possible interpretations. The context suggests you imagine this to be positive, but the most common 'predefined message' would probably be templates giving them a reprimand/warning. Some people may interpret this question as such, and some may think you're only talking about positive messages.
  • For the question "About how often have you done each of the following:" there's a huge gap between "A few times per year" and "once or twice per week". With such vague scales, it's likely that more modest people round down by a lot, and people in less modest cultures round up a lot. This may make a discrepancy in support seem apparent that is not there (or make a discrepancy the other way around disappear). This is especially relevant for 'rare events' such as "Mentored someone closely to help them implement their Wikimedia program, event, campaign, or partnership". If you do that once a month, that is HUGE. But it would still possibly disappear in the noise of 'a few times a year' which may be also selected by the also admirable volunteer that closely mentors someone once every two years (because it's more than never).

In general, the survey assumes that the respondee fits one single profile. That I 'lead' one project. That I contribute to one community. This is maybe true for some, but many people fill multiple roles. It is unclear to me how I should fill out questions like 'Does the community you organize or lead have established policies and/or procedures for group decision-making?'. Is that always the same community as I responded about before? Sometimes a response can be multiple communities. I imagine you could give instructions to pick one and stick with it, but I totally didn't see that kind of instruction this time.

The number of questions should really come down a *lot* :) This is just an exhaustion exercise and it's impossible for me to stay focused for this number of questions. It's definitely longer for me than the 40 minutes max that you promised (I think it may have taken about an hour). Maybe you could warn people more fairly about this, and allow them to only fill out certain sections (choose at the start) and then allow to continue to other sections at the end.

Finally, I would have appreciated a note that states the obvious: the WMF is running this survey, but no individual responses will be shared outside the survey-data-folks (even within the organization). This is a bit implied in the privacy policy that was hidden behind a link, but not explicit. This is the equivalent situation of the WMF being landlord (runs the projects), employer (runs the grants programs), supervisor (runs the movement process) and also runs the survey. I definitely presume a firewall situation, but can't realistically rely on it. Even when you remove personal identifiers, there is plenty of data points in the demographic questions to identify any community member based on their responses alone (combination of language, country of residence and start year would be plenty in my case).

I appreciate the efforts, but I wish that these surveys were much much shorter, and easier to digest to make them more accessible.

Effeietsanders (talk) 22:10, 23 September 2020 (UTC)[]

Yes - I agree with much of the above. About an hour in and I found myself answering questions as if I were major lead in a whole host of teams and projects, rather than a humble contributor. Maybe the fact that I did try to answer them was unhelpful? - I don't know. Yes, the timescale breakdown for things like frequency of mentoring were ridiculous and bound to produce wildly inaccurate results. Nothing asked whether the respondee helps new editors in any help fora, either. And terms were bandied around as if we all know what they are - not a hyperlink in sight (for me to work out what an affiliate is, for example, or to check whether or not I had read or remember the relevant policy being referred to.) Over all - far too long-winded. Nick Moyes (talk) 22:23, 23 September 2020 (UTC)[]
Hi @Effeietsanders: @Nick Moyes: Thank you for your feedback! We are working to improve the survey's length, reliability, and validity every year. The majority of respondents do take between 20 and 40 minutes to respond to the survey, but some take less and some more, depending on how a respondent participates in the movement. I apologize that you feel you saw questions that were not relevant to your participation, @Nick Moyes:, you likely saw those because you had indicated that you lead a project, event, campaign, or group. I'd love to hear more about how we might better phrase that question so you don't see irrelevant sections.
I appreciate the concerns about data interpretation-- please know that we keep this in mind when we report and interpret data. For example, the question about how often the respondent has "mentored someone closely." I agree-- just doing that once is a big deal, and it is not lost in the data. We ask 'how often' so that we can better track change over time, both in the proportion of organizers who mentor others at all, but also how frequently they do so.
For future iterations of the survey, we will add hyperlinks for those terms which might not be apparent to all. I will also work with our Legal department to make sure that section of the survey's privacy policy is even more clear. Thanks again to you both. [[User:|RMaung (WMF)]] (talk) 15:09, 9 October 2020 (UTC)[]
Thanks for the response. Just a sidenote: this nuance is lost in the data, when a not-appropriate scale is chosen that confuses the respondent, and people err on the side of caution, or the side of bragging (not sure how else to phrase this), depending on their cultural background or social status. I think in general it is sub-optimal to have respondents interpret what your scale means, but that is a more subjective discussion. Either way, by now it's moot to try to improve anything, even if you wanted to.
I hope for a significant improvement on both length and reliability - as these introduce biases that will then be relied upon for many strategic decisions. This is a sustained problem for quite a few years now. Effeietsanders (talk) 16:29, 9 October 2020 (UTC)[]
@RMaung (WMF): Darn - I drafted a long yesterday, but lost it all in an edit conflict. No time to rewrite it today, but, in essence, I'd suggest that because you guys probably think much bigger and more strategically than the majority of us mere mortals, that you create a hierarchy of questions from simple involvement to more deeply embedded in WMF work. Then the right questions only need kick in at the relevant point. So, if I've helped participate in or helped to run a local editathon/meetup/online training sessions once or twice, I don't get marked as some major project organiser, as if I ran Women in Red or a Wikimania project. Yes, hyperlinks would be good. Having terms defined or explainable as you go along is essential to respondees understanding what it is that you are asking, and for you to get the right answers. Do you have a panel of non-WMF staff from various worldwide projects on whom you trial these surveys before they're rolled out? If not, getting a few volunteers together might help you iron out problems or gaps in understanding. Regards, Nick Moyes (talk) 10:15, 10 October 2020 (UTC)[]

How to add a translation into Russian?Edit

Hello! How to add a translation into Russian? I wrote it, see: Community_Insights/ru Зайва Игорь Леонидович (talk) 06:30, 26 September 2020 (UTC)[]

@Зайва Игорь Леонидович: Hi! what in particular are you translating? This meta page, or the survey, or something else? Thank you! RMaung (WMF) (talk) 15:11, 9 October 2020 (UTC)[]
@RMaung (WMF): Hi! https://meta.wikimedia.org/wiki/Community_Insightshttps://meta.wikimedia.org/wiki/Community_Insights/ru --Зайва Игорь Леонидович (talk) 18:37, 9 October 2020 (UTC)[]

Access DeniedEdit

It's a little bit sad, given the stated aims of this research and other things coming from the WMF, that Qualtrics prohibits anyone responding to the survey, or in fact accessing any part of their site including their privacy policy and ToS, over the Tor anonymity network. And no nice message either, just a raw 403. -- zzuuzz (talk) 20:22, 28 September 2020 (UTC)[]

@Zzuuzz: Thanks for letting me know-- I'll reach out to Qualtrics to see if there are any ways around this. Qualtrics has a variety of benefits and functionality, but I acknowledge that there are some drawbacks. For this year it was the best available option for us to use. RMaung (WMF) (talk) 15:13, 9 October 2020 (UTC)[]

2021 surveyEdit

(Pinging User:RMaung (WMF).)

The Insights survey often has some issues which only show up after the survey is running. This year, could we perhaps have an open/public review of the survey questions, structure, procedures, and plans, in advance of the survey being run? Even better, could it be open to suggestions for additions? Local projects and regions often want to know something about their own people, and I wonder if the survey could specifically pose certain questions to those who give a specific answer on country/project/language.

Also, could the survey be built in such a way as to allow the publishing of the raw data afterward, without running into privacy issues? --Yair rand (talk) 04:45, 25 August 2021 (UTC)[]

Hello Yair rand! I have transitioned roles at the Foundation and I no longer project manage the Community Insights survey. Looping in TAndic (WMF) who is managing regular community surveys going forward. Best, RMaung (WMF) (talk) 16:57, 25 August 2021 (UTC)[]
Hello Yair rand - Thank you so much for starting this conversation. We are in the process of figuring out some infrastructure improvements to better reach respondents for the survey as well as working to reduce and refine questions to make the survey more useful to everyone involved. In September, I will be posting on Diff and other movement communication outlets to share out about some temporary changes to the Community Insights plan and schedule for the 2021-22 fiscal year while we initiate planning and discussion on the long-term changes needed to our surveying practices. With that blog and communications step, we will include a call to action for interested community members to get involved in helping us with the planning of these revisions to our systems and processes. I am happy to also ping you at that time, but in the meantime, please feel welcome to continue the discussion here or on my talk page, I am very much appreciative of any feedback and am happy to discuss how certain things work as far as I'm able to.
To answer directly regarding questions for local projects and regions: it would depend on what kinds of questions they are. One big thing we have to think about is the length of the survey, its cross-project active editor audience focus, and how much of a burden it places on the person responding -- the longer it is, the more likely target participants are to give up and not answer the whole set of questions. This concern about survey length can be seen in the helpful conversation [“Confusion in questions”] above. For this reason, depending on the level of specificity needed by a project, it may make more sense to run a community-led survey on that project to make it representative of the project. Still, our team is always interested in learning about the survey measurement interests from editors and organizers as we work collectively to understand our movement landscape.
To answer about raw data: it depends. We are governed by the privacy settings outlined in the survey’s Privacy Statement. This year, we made data for the Collaboration, Diversity & Inclusion supplement available on google data studio, which can be downloaded as well. But again, it is not raw data as we receive it. As you noted, the issue is privacy, and the more detail available, the more risk there is to the privacy of respondents. This is especially the case if we get more in depth on regions and projects, as some projects are very small and we get very few responses from their contributors. It's a real quandary, because often the information we want to make data useful to our specific projects is exactly the information that makes privacy an issue. In all cases, privacy and safety should always be the first priority. If there are specific data you are wondering about, please reply to let me know. We would like to better understand broader interests in the data so we can look into if there is more we might do within the bounds of privacy, and are open to trying to accommodate more interests than our own so long as it is feasible within our current work scope and flows.
I hope this helps some, and I look forward to continuing the conversation! -TAndic (WMF) (talk) 08:09, 27 August 2021 (UTC)[]
Return to "Community Insights" page.