Talk:Community Insights

Costs for this survey

Hello,

how many money does this survey cost. And why do you use a external company for this survey. I think the most questions of this survey are multiple choice questions and there it does not take a long time to know how many persons have chosen what for a answer and how were the questions translated. I think there have been translated some things into German where it doesnt make sense. For example Village pumps was translated with Dorfpumpen. This is a translation but does not exist in the German Wikipedia as a Diskussion site. Is it possible to get the text of the questions in German so that I can check the questions and tell you when I think that another translation of a word were good. -- Hogü-456 (talk) 18:30, 10 September 2019 (UTC)

As Hogü said, the translations are terrible. For example there is a question about how I deal with new users and one option is "I post a message on their talk page, using a barnstar or a predefined message" while the german translation is "Ich poste eine Nachricht auf ihrer Talkseite mit einem Barnstar oder einer vorformulierten Nachricht." The translation is very close to the original, however there is one problem: Barnstars don't exist in the German Wikipedia. Another example: Ihe part where I am asked where I am from I can select a country from a list which is sorted alphabetically in english - but obviously is a mess when viewed in german. "Deutschland" can be found under "G", "Österreich" can be found under "A". It works in english but not if every entry in the list is translated individually. It's obvious that this survey is made by enwiki users for enwiki users. I hope that the surveys taken are not all grouped together but are instead analyzed in seperate groups depending on the language the survey-taker used. On another note: The final part asking about demographic data is pretty weird. For example the question about gender has the options "male", "female", "transgender", "non-binary". Sooo apparently trans people are neither man nor woman? And when asked about age the minimum available is 18. Are people below 18 not allowed to participate on Wikimedia projects? Both of these don't affect me (I am neither trans nor below 18) but I wanted to point this out anyways. ーŒ̷͠²ð·¨´´̢́̕͘³͏¯̞̗ 【🅱alk】 17:44, 21 September 2019 (UTC)

Hello Œ̷͠²ð·¨´´̢́̕͘³͏¯̞̗! We are aware that there are some issues with the translations-- they were completed by a third-party service. We had hoped to have volunteers make corrections to the survey translations, but as this is such a large request many of the languages did not get reviewed, and German is one of them. We are manually fixing errors (like the one mentioned at the top of this page) as we can, but some we are stuck with--in particular the problem of alphabetization you refer to. We intended to make the list of regions and countries alphabetical in each survey language, but it is not possible due to the survey infrastructure. Please know that this is a known issue, and we are looking into ways to fix it for next year--same goes for overall translation quality. As for the gender question, we ask respondents to select all that apply in order to make the question as inclusive as possible. For the age question, we are bound by legal restrictions on surveying people under 18 without parental consent. I appreciate your feedback! I do have one question for you-- is there another "vorformulierten Nachricht" that we might use to substitute for "Barnstar" in the German version of the survey? I'd like to fix the error you are pointing out, but I'm wondering if there is an example of a "predefined message" I can substitute that is more relevant to German Wikipedia. Thank you again for your time and feedback! --RMaung (WMF) (talk) 22:06, 27 September 2019 (UTC)

@RMaung (WMF): Thanks for your reply and sorry for my late answer. There are several templates in dewiki to greet new users, see de:WP:Begrüßung#Standardisierte Begrüßungsvorlagen. I think the phrase could be translated as: "Ich hinterlasse eine Nachricht auf seiner/ihrer Benutzerdiskussionsseite unter Verwendung einer Begrüßungsvorlage." (that would be the literal translation of "I leave a message on his/her user talk page using a greetings-template.")
Oh and about the age thing: if you are not allowed to survey people under 18, why don't you just add the following option: "I am below 18 and my parents or guardians are answering this survey for me" That way kids their parents of course know what to pick and you got your age data in a totally legal way. :P Then again adding options to questions after some people already answered them is not fair, however in the next survey this could be an option. ーŒ̷͠²ð·¨´´̢́̕͘³͏¯̞̗ 【🅱alk】 15:57, 5 October 2019 (UTC)

If the software does not allow to use different alphabetical orders for different languages you might want to use international codes in future, like "DE - Germany", "DE - Alemania" etc. Respondents should be able to find their own international code in a medium (internet) where such a code is commonly used. → «« Man77 »» ^[de] 16:38, 27 October 2019 (UTC)

Qualtrics

If you want your data to be representative, maybe you should choose another service provider. Qualtrics has sanctioned all those living in Iran. 4nn1l2 (talk) 03:56, 10 October 2019 (UTC)

@4nn1l2: I realize this response is coming incredibly late, but it is still relevant to this year's survey: Qualtrics does not block Iran, it is the result of the political relationship and resulting data embargoes between the governments in Iran and the US (and thus any survey that originates in the US, no matter which provider we use). We are hopeful that the situation will change in the future. RMaung (WMF) (talk) 15:16, 9 October 2020 (UTC)

Survey design

The failure of this survey to explain to ordinary editors what "...implementing your project or scheme..." actually means made it virtually impossible to answer one whole set of questions. Having then moved forward through the survey to try to understand the context of those questions, I found there was no back button (just before the personal info section.) It was thus impossible to go back and give proper answers. I'm now feeling I've spent 25 minutes answering a survey that's not applicable to my level of editing activity on en-wiki only. Nick Moyes (talk) 20:08, 17 October 2019 (UTC)

2019 Survey?

What is the current status of the 2019 Survey? Antrocent (talk) 06:16, 13 February 2020 (UTC)

Confusion in questions

Hi,

@RMaung (WMF): I'm not sure where to provide this feedback, so I'll be putting it here. As I'm answering this gigantic survey, it's sad to see that again there's confusion on terminology - some on rather important questions. Especially:

• "For the Wikimedia projects you participate in, how much do you agree or disagree with the following statements about user conduct policies?" It is unclear what is being referred to. I would guess community policies, but the question is asked right after a question about WMF's policies ( Code of Conduct for Wikimedia technical spaces, Trust and Safety) and right before Wikimedia Foundation’s Friendly Space Policies. It's likely that different respondents interpret this question differently, with a bias based on how much they dislike WMF's policies.
• When talking about actions towards newcomers, one line states: "I post a message on their talk page, using a barnstar or a predefined message" which is a super wide array of possible interpretations. The context suggests you imagine this to be positive, but the most common 'predefined message' would probably be templates giving them a reprimand/warning. Some people may interpret this question as such, and some may think you're only talking about positive messages.
• For the question "About how often have you done each of the following:" there's a huge gap between "A few times per year" and "once or twice per week". With such vague scales, it's likely that more modest people round down by a lot, and people in less modest cultures round up a lot. This may make a discrepancy in support seem apparent that is not there (or make a discrepancy the other way around disappear). This is especially relevant for 'rare events' such as "Mentored someone closely to help them implement their Wikimedia program, event, campaign, or partnership". If you do that once a month, that is HUGE. But it would still possibly disappear in the noise of 'a few times a year' which may be also selected by the also admirable volunteer that closely mentors someone once every two years (because it's more than never).

In general, the survey assumes that the respondee fits one single profile. That I 'lead' one project. That I contribute to one community. This is maybe true for some, but many people fill multiple roles. It is unclear to me how I should fill out questions like 'Does the community you organize or lead have established policies and/or procedures for group decision-making?'. Is that always the same community as I responded about before? Sometimes a response can be multiple communities. I imagine you could give instructions to pick one and stick with it, but I totally didn't see that kind of instruction this time.

The number of questions should really come down a *lot* :) This is just an exhaustion exercise and it's impossible for me to stay focused for this number of questions. It's definitely longer for me than the 40 minutes max that you promised (I think it may have taken about an hour). Maybe you could warn people more fairly about this, and allow them to only fill out certain sections (choose at the start) and then allow to continue to other sections at the end.

Finally, I would have appreciated a note that states the obvious: the WMF is running this survey, but no individual responses will be shared outside the survey-data-folks (even within the organization). This is a bit implied in the privacy policy that was hidden behind a link, but not explicit. This is the equivalent situation of the WMF being landlord (runs the projects), employer (runs the grants programs), supervisor (runs the movement process) and also runs the survey. I definitely presume a firewall situation, but can't realistically rely on it. Even when you remove personal identifiers, there is plenty of data points in the demographic questions to identify any community member based on their responses alone (combination of language, country of residence and start year would be plenty in my case).

I appreciate the efforts, but I wish that these surveys were much much shorter, and easier to digest to make them more accessible.

Effeietsanders (talk) 22:10, 23 September 2020 (UTC)

Yes - I agree with much of the above. About an hour in and I found myself answering questions as if I were major lead in a whole host of teams and projects, rather than a humble contributor. Maybe the fact that I did try to answer them was unhelpful? - I don't know. Yes, the timescale breakdown for things like frequency of mentoring were ridiculous and bound to produce wildly inaccurate results. Nothing asked whether the respondee helps new editors in any help fora, either. And terms were bandied around as if we all know what they are - not a hyperlink in sight (for me to work out what an affiliate is, for example, or to check whether or not I had read or remember the relevant policy being referred to.) Over all - far too long-winded. Nick Moyes (talk) 22:23, 23 September 2020 (UTC)

Hi @Effeietsanders: @Nick Moyes: Thank you for your feedback! We are working to improve the survey's length, reliability, and validity every year. The majority of respondents do take between 20 and 40 minutes to respond to the survey, but some take less and some more, depending on how a respondent participates in the movement. I apologize that you feel you saw questions that were not relevant to your participation, @Nick Moyes:, you likely saw those because you had indicated that you lead a project, event, campaign, or group. I'd love to hear more about how we might better phrase that question so you don't see irrelevant sections.

I appreciate the concerns about data interpretation-- please know that we keep this in mind when we report and interpret data. For example, the question about how often the respondent has "mentored someone closely." I agree-- just doing that once is a big deal, and it is not lost in the data. We ask 'how often' so that we can better track change over time, both in the proportion of organizers who mentor others at all, but also how frequently they do so.

For future iterations of the survey, we will add hyperlinks for those terms which might not be apparent to all. I will also work with our Legal department to make sure that section of the survey's privacy policy is even more clear. Thanks again to you both. [[User:|RMaung (WMF)]] (talk) 15:09, 9 October 2020 (UTC)

Thanks for the response. Just a sidenote: this nuance is lost in the data, when a not-appropriate scale is chosen that confuses the respondent, and people err on the side of caution, or the side of bragging (not sure how else to phrase this), depending on their cultural background or social status. I think in general it is sub-optimal to have respondents interpret what your scale means, but that is a more subjective discussion. Either way, by now it's moot to try to improve anything, even if you wanted to.

I hope for a significant improvement on both length and reliability - as these introduce biases that will then be relied upon for many strategic decisions. This is a sustained problem for quite a few years now. Effeietsanders (talk) 16:29, 9 October 2020 (UTC)

@RMaung (WMF): Darn - I drafted a long yesterday, but lost it all in an edit conflict. No time to rewrite it today, but, in essence, I'd suggest that because you guys probably think much bigger and more strategically than the majority of us mere mortals, that you create a hierarchy of questions from simple involvement to more deeply embedded in WMF work. Then the right questions only need kick in at the relevant point. So, if I've helped participate in or helped to run a local editathon/meetup/online training sessions once or twice, I don't get marked as some major project organiser, as if I ran Women in Red or a Wikimania project. Yes, hyperlinks would be good. Having terms defined or explainable as you go along is essential to respondees understanding what it is that you are asking, and for you to get the right answers. Do you have a panel of non-WMF staff from various worldwide projects on whom you trial these surveys before they're rolled out? If not, getting a few volunteers together might help you iron out problems or gaps in understanding. Regards, Nick Moyes (talk) 10:15, 10 October 2020 (UTC)

How to add a translation into Russian?

Hello! How to add a translation into Russian? I wrote it, see: Community_Insights/ru Зайва Игорь Леонидович (talk) 06:30, 26 September 2020 (UTC)

@Зайва Игорь Леонидович: Hi! what in particular are you translating? This meta page, or the survey, or something else? Thank you! RMaung (WMF) (talk) 15:11, 9 October 2020 (UTC)

@RMaung (WMF): Hi! https://meta.wikimedia.org/wiki/Community_Insights → https://meta.wikimedia.org/wiki/Community_Insights/ru --Зайва Игорь Леонидович (talk) 18:37, 9 October 2020 (UTC)

Access Denied

It's a little bit sad, given the stated aims of this research and other things coming from the WMF, that Qualtrics prohibits anyone responding to the survey, or in fact accessing any part of their site including their privacy policy and ToS, over the Tor anonymity network. And no nice message either, just a raw 403. -- zzuuzz ^(talk) 20:22, 28 September 2020 (UTC)

@Zzuuzz: Thanks for letting me know-- I'll reach out to Qualtrics to see if there are any ways around this. Qualtrics has a variety of benefits and functionality, but I acknowledge that there are some drawbacks. For this year it was the best available option for us to use. RMaung (WMF) (talk) 15:13, 9 October 2020 (UTC)

2021 survey

(Pinging User:RMaung (WMF).)

The Insights survey often has some issues which only show up after the survey is running. This year, could we perhaps have an open/public review of the survey questions, structure, procedures, and plans, in advance of the survey being run? Even better, could it be open to suggestions for additions? Local projects and regions often want to know something about their own people, and I wonder if the survey could specifically pose certain questions to those who give a specific answer on country/project/language.

Also, could the survey be built in such a way as to allow the publishing of the raw data afterward, without running into privacy issues? --Yair rand (talk) 04:45, 25 August 2021 (UTC)

Hello Yair rand! I have transitioned roles at the Foundation and I no longer project manage the Community Insights survey. Looping in TAndic (WMF) who is managing regular community surveys going forward. Best, RMaung (WMF) (talk) 16:57, 25 August 2021 (UTC)

Hello Yair rand - Thank you so much for starting this conversation. We are in the process of figuring out some infrastructure improvements to better reach respondents for the survey as well as working to reduce and refine questions to make the survey more useful to everyone involved. In September, I will be posting on Diff and other movement communication outlets to share out about some temporary changes to the Community Insights plan and schedule for the 2021-22 fiscal year while we initiate planning and discussion on the long-term changes needed to our surveying practices. With that blog and communications step, we will include a call to action for interested community members to get involved in helping us with the planning of these revisions to our systems and processes. I am happy to also ping you at that time, but in the meantime, please feel welcome to continue the discussion here or on my talk page, I am very much appreciative of any feedback and am happy to discuss how certain things work as far as I'm able to.

To answer directly regarding questions for local projects and regions: it would depend on what kinds of questions they are. One big thing we have to think about is the length of the survey, its cross-project active editor audience focus, and how much of a burden it places on the person responding -- the longer it is, the more likely target participants are to give up and not answer the whole set of questions. This concern about survey length can be seen in the helpful conversation [“Confusion in questions”] above. For this reason, depending on the level of specificity needed by a project, it may make more sense to run a community-led survey on that project to make it representative of the project. Still, our team is always interested in learning about the survey measurement interests from editors and organizers as we work collectively to understand our movement landscape.

To answer about raw data: it depends. We are governed by the privacy settings outlined in the survey’s Privacy Statement. This year, we made data for the Collaboration, Diversity & Inclusion supplement available on google data studio, which can be downloaded as well. But again, it is not raw data as we receive it. As you noted, the issue is privacy, and the more detail available, the more risk there is to the privacy of respondents. This is especially the case if we get more in depth on regions and projects, as some projects are very small and we get very few responses from their contributors. It's a real quandary, because often the information we want to make data useful to our specific projects is exactly the information that makes privacy an issue. In all cases, privacy and safety should always be the first priority. If there are specific data you are wondering about, please reply to let me know. We would like to better understand broader interests in the data so we can look into if there is more we might do within the bounds of privacy, and are open to trying to accommodate more interests than our own so long as it is feasible within our current work scope and flows.

I hope this helps some, and I look forward to continuing the conversation! -TAndic (WMF) (talk) 08:09, 27 August 2021 (UTC)

@TAndic (WMF): So, re local surveys: Many local groups have run editor surveys before (eg, the local affiliates in Ireland, Switzerland, Germany, and Ukraine; IIRC the English Wikipedia was recently considering running an admin survey as well). These are often quite costly in terms of local disruption. I'd recommend giving each country-affiliate and project a small-ish "budget" of questions to work with, which may be left unused. Also, maybe have a couple of "generic" questions for respondents in countries with affiliates, like the "...had you ever heard of [affiliate name]?", "I am confident in [affiliate]'s ability to..." questions (adapted from the equivalent WMF questions), if people would find that useful. Giving groups the ability to select certain questions may also encourage people to participate in translating the survey into more languages.

Re data: Thanks for the link to the data studio set. I hadn't seen that before. In 2020, we had a page for aggregate data, I'm not sure why the 2021 report didn't have that. Having the full text of each question is useful. Anyway, besides for that, I would recommend having, for each question, the basic stats divided by primary category (editor, organizer, developer, etc) and any "interesting" categories (which would include any categories of a type which the question was divided by in the summary; eg in the 2021 report, it was considered worthwhile to highlight data in the area of "Foundation support" for groups divided by geography, tenure, home wiki, and language (see this image, showing stats for all of them), so the data on those questions would also show each division for each of those categories, so that one could look up the stats on those questions for any other wiki, continent, etc. as well). Some columns might need to be removed or combined if the sample would otherwise be too small.

I also recommend recording which survey responses come from which source. I see in the methodology section that the survey was sent to various somewhat overlapping groups by different means. If the data from all sources is dumped into one group, wouldn't the category of "editors" have the subgroup "movement organizers who are also editors" heavily overrepresented, skewing the data? --Yair rand (talk) 21:48, 30 September 2021 (UTC)

Hello Yair rand, Sorry for my delay in seeing your post here. Thank you for these links and recommendations, we appreciate the feedback.

As promised earlier, the scope of changes for this fiscal year is now announced on Diff; the primary goal for this year is to improve our demographics questions and shorten the survey with the aim of lessening the burden on respondents and improving response rates while we work on ways to improve the data collection method itself.

Regarding why we haven’t presented descriptive data tables for each of the questions in the Insights survey this year, our team simply did not have it budgeted for in the report project plan for the overall reporting this past fiscal year. While we may not be budgeted to do this currently, I hope to have time to report the descriptive statistics next year; even if I’m unable to do so, we can at least provide the survey questions, which I agree are very helpful to have. We do aim to continue having translations reviewed by community members this year, which people interested in participating can sign up for below!

The short of it is, any change to the survey -- whether methodologically, with question wording, question additions, or most anything -- sets off another set of a chain of revisions and contingency planning for maintaining comparable data over time. We really have to be careful with big and small changes! The modest changes we’re working on this year are hoped to be a solid and focused starting point to begin our revisioning.

Regarding your question on potential over-representation of movement organizer-editors (as an example, I assume): we do keep the different ways that respondents come to the survey separated, especially as some forms of recruitment do not provide data about edits (which means we can’t do propensity score matching and weight those respondents appropriately within the larger data set). So, for example, for questions which report on all groups, we don’t include people who were recruited to the survey by means where we can’t weight their responses, so movement organizers should not be overrepresented there. They are only included in the analyses when we report specifically on the group, so for example if we say “movement organizers reported x” that means all movement organizers who answered, but when we say broadly “contributors reported x” then those who were recruited outside of the emailing process are excluded to not skew the data. I hope that makes sense - it is admittedly a bit clunky, but we hope with taking time to improve the data collection tools themselves we will one day be able to to circumvent such problems. -- TAndic (WMF) (talk) 12:25, 21 October 2021 (UTC)

Sign up here to be contacted about making changes to the Community Insights survey for 2022!

Hello! Comment and sign here if you would like to be contacted about helping to make changes to the Community Insights 2021-2022 survey. Please note whether you would like to:

• make sure demographic questions make sense for my language, community, or context (approximately October or November 2021)

• help pilot and review questions (approximately November or December 2021)

• participate in community conversations about the survey

• help review translations in my language (see the list of languages the survey is translated into on the Community Insights meta page)

• work on something else

We look forward to collaborating with you! TAndic (WMF) (talk) 17:29, 27 September 2021 (UTC)

Qualtrics to LimeSurvey

Hi all! I'm trying to help with the migration from Qualtrics (proprietary) to LimeSurvey (FLOSS). Maybe interested @4nn1l2 and RMaung (WMF): that I've seen above in a similar discussion (#Qualtrics). How can I help in this? Thaaank you :) --Valerio Bozzolan (talk) 14:21, 13 December 2021 (UTC)

Hi Valerio Bozzolan, this is really cool. I don't know what you want me to do. I'm 100% for any service which is not banned in Iran (and maybe other areas of the world) due to political reasons, including LimeSurvey. Adiós Qualtrics :) 4nn1l2 (talk) 15:41, 13 December 2021 (UTC)

Eheh @4nn1l2: are you able to reach this? https://survey.wikimedia.it/ I think yep. --Valerio Bozzolan (talk) 16:00, 13 December 2021 (UTC)

@Valerio Bozzolan:, Yes :) 4nn1l2 (talk) 16:03, 13 December 2021 (UTC)

@Valerio Bozzolan: Hello! Thank you for restarting this conversation, and thanks @4nn1l2: for confirming LimeSurvey works for you, this is very good news. I can't make any promises, but I'm beginning to go down the rabbit hole of LimeSurvey history to figure out what issues may need resolving and what next steps would be required to reassess its viability. It may take me some months to get responses on everything, just to be clear. If there is any new documentation that you have regarding privacy and security for LimeSurvey and WMF, that would be very helpful to have. Please feel welcome to write here, on my talk page, or email at surveyswikimedia.org. TAndic (WMF) (talk) 21:47, 20 December 2021 (UTC)

Thank you so much. Unfortunately I don't know why WMF hasn't adopted in the last years an enterprise assistance plan with LimeSurvey to ensure they have everything at the latest security update all the time. I hope it happens soon. If someone want to help can contribute on LimeSurvey page on Meta-wiki. Valerio Bozzolan (talk) 08:12, 8 February 2022 (UTC)

I subscribe to this. The use of Qualtrics is the reason why I am not going to answer to the Community Insights survey in 2024. - Mind Booster Noori (talk) 16:01, 23 March 2024 (UTC)

Sign-up issue

I received an email today, asking me to click a link to agree to receive the survey. I did so, and was taken to a page that shows only a never ending "Loading" icon (it's been going for over five minutes, so far). I copied and pasted the link to another browser, and got a notice that "You have either already completed the survey or your session has expired". Andy Mabbett (Pigsonthewing); Talk to Andy; Andy's edits 15:06, 14 June 2022 (UTC)

Thank you for your message @Pigsonthewing. I'm sorry about the weird bug. Did you get a second email from surveyswikimedia.org (email title: Community Insights 2022 Survey - Wikimedia Foundation) with a link to the actual survey? It can sometimes end up in spam, unfortunately. Let me know if it didn't make it, I can send another link :) Thank you once more! -TAndic (WMF) (talk) 18:04, 14 June 2022 (UTC)

I did. Thank you. Andy Mabbett (Pigsonthewing); Talk to Andy; Andy's edits 18:21, 14 June 2022 (UTC)

No email received

I can see new email notification in Wikimedia sites, but no new mail in my email inbox (neither in spam folder), could you resend it? Thanks.--S8321414 (talk) 02:01, 21 March 2024 (UTC)

Hello and thank you for your message. I have resent the email via EmailUser :) - TAndic (WMF) (talk) 15:12, 21 March 2024 (UTC)

Thank you! Received new email and completed survey. :-)--S8321414 (talk) 23:28, 21 March 2024 (UTC)