Research:Cloud Services Annual Survey/2015

The 2015 Tool Labs survey collected feedback from Tool Labs' developers and maintainers on how the services offered can be improved to help their development and maintenance needs.

Participants

edit

The participants of the survey were the members of the Wikimedia Tool Labs. 962 participants were contacted via email. The emails were harvested from Labs' LDAP system, which is available to everyone with a labs account. The list initially involved 977 participants. All those who had opted out of contacts via Wikitech were eventually excluded.

106 email recipients participated in the survey which represents 11% of those who are contacted.

Dates

edit

The 2015 survey was active between 2015-09-25 and 2015-10-08.

Questions and answers

edit

Basic demographics

edit

Responses to the basic demographic questions were divided into roughly equal sized groups to provide "cohorts" of similar Tool Labs users. These groups were then plotted against each other in the qualitative responses to determine if there were significant differences in the opinions of the sub-groups.

How many years have you used Tool Labs?

edit

Help: If you were a Toolserver user, please include the total number of years that you have used Toolserver and Tool Labs.

 
Cohort Percentage
1 Year 29.81%
2-3 Years 36.54%
4+ Years 33.65%

Many of the respondents have been using Tool Labs for 2 or fewer years. In fact, nearly 30% of the respondents are using Tool Labs only for a year. This can be the sign of many young users or that young users are the ones that are more willing to participate in surveys. The distribution has a long tail indicating there are quite a few users who have been around for a long time.

How many tools have you developed in Tool Labs?

edit

Help: We do not have a canonical definition of "tools". We are trying to estimate how much burden is on you, so please use your best judgement in assessing how many tools you have developed.

 
Cohort Percentage
0 Tools 8.57%
1 Tool 31.43%
2-3 Tools 28.57%
4+ Tools 31.43%

40% of respondents have developed 0-1 tools. This may correlate with the amount of time that they have been active on Tool Labs. The distribution of the 4+ Tools cohort again has a long tail indicating that some tool developers are highly prolific.

How many tools do you actively maintain in Tool Labs?

edit

Help: We do not have a canonical definition of "tools". We are trying to estimate how much burden is on you, so please use your best judgement in assessing how many tools you maintain.

 
Cohort Percentage
0 Maintain 16.19%
1 Maintain 27.62%
2 Maintain 26.67%
3+ Maintain 29.52%

The maintenance numbers roughly correlate with the number of Tools developed although the raw numbers are skewed downward slightly. This seems to suggest that very few respondents have become maintainers of tools that they didn't build themselves and also possibly indicates that some tools have been abandoned or replaced over time.

How many hours per week do you spend developing or maintaining tools on Tool Labs?

edit
 
Cohort Percentage
0 Hours 24.51%
1 Hour 34.31%
2-8 Hours 29.41%
9+ Hours 11.76%

Over 50% of respondents spend an hour or less per week on average working on their tools.

Were you a developer and/or maintainer in the Toolserver as well?

edit
 
Cohort Percentage
No Toolserver 59.43%
Toolserver 40.57%

The majority of survey responses were received from users with no history using Toolserver. It is currently unknown how this distribution correlates with the full number of active Tool Labs users.

Additional demographics

edit

What programming languages do you prefer to program in?

edit

For this question, the participants had the following choices: Perl, PHP, Python 2, Python 3, Ruby, or they could enter the language of their choice. The following table shows the ranked list of languages in terms of the percentage of time they were chosen as the language of choice. Note that participants could report more than one language.

Language %
PHP 58.49%
Python3 37.74%
Python2 36.79%
JavaScript 9.43%
Perl 8.49%
Ruby 7.55%
Other 7.55%
Java 6.60%
C# 4.72%

High usage of PHP and Python were not unexpected results. MediaWiki itself is written in PHP and Python is both widely used for data analysis tasks and well supported by the Pywikibot community. Freeform responses were normalized to make this analysis easier. The JavaScript numbers include responses of "JavaScript", "node", and "nodejs". The "Other" grouping includes several common scripting tools and a few very esoteric languages.

When you develop a tool, what percentage of your work is done locally (as opposed to remotely on Tool Labs)?

edit

Help: For example, enter 30 if 30% of your development work is done locally.

 

Answers were rounded up to the nearest 5% for analysis.

Answer Percentage reported
0 8.65%
5 3.85%
10 9.62%
20 2.88%
25 0.96%
30 3.85%
50 9.62%
60 3.85%
70 10.58%
75 2.88%
80 16.35%
85 1.92%
90 9.62%
95 5.77%
100 9.62%

Do you prefer using source control to manage your tool's source code?

edit
 
Yes No
73% 26%

Qualitative questions

edit

A series of qualitative questions were asked in order to gauge general satisfaction with Tool Labs. Since this was the first survey of this type given to the Tool Labs community these results will form a historical baseline to compare the results future surveys against.

Services have high reliability (up time)

edit
 
All 1 Year 2-3 Years 4+ Years 0 Tools 1 Tool 2-3 Tools 4+ Tools 0 Maintain 1 Maintain 2 Maintain 3+ Maintain 0 Hours 1 Hour 2-8 Hours 9+ Hours No toolserver Toolserver
Agree/Strongly Agree 63.81% 67.74% 54.05% 70.27% 70.00% 65.63% 76.67% 48.48% 66.67% 65.52% 70.37% 54.84% 68.97% 57.14% 63.33% 72.73% 68.25% 57.14%
Disagree/Strongly Disagree 34.29% 29.03% 43.24% 29.73% 20.00% 31.25% 23.33% 51.52% 22.22% 34.48% 29.63% 45.16% 24.14% 42.86% 36.67% 27.27% 28.57% 42.86%

It is easy to write code and have it running on Tool Labs

edit
 
All 1 Year 2-3 Years 4+ Years 0 Tools 1 Tool 2-3 Tools 4+ Tools 0 Maintain 1 Maintain 2 Maintain 3+ Maintain 0 Hours 1 Hour 2-8 Hours 9+ Hours No toolserver Toolserver
Agree/Strongly Agree 68.57% 61.29% 67.57% 75.68% 50.00% 53.13% 73.33% 84.85% 55.56% 58.62% 74.07% 80.65% 51.72% 77.14% 66.67% 90.91% 66.67% 71.43%
Disagree/Strongly Disagree 24.76% 25.81% 27.03% 21.62% 0.00% 40.63% 26.67% 15.15% 16.67% 34.48% 25.93% 19.35% 31.03% 22.86% 30.00% 0.00% 23.81% 26.19%

I feel I am supported by the Tool Labs team when I contact them via labs-l, wikimedia-labs IRC channel, or phabricator.

edit
 
All 1 Year 2-3 Years 4+ Years 0 Tools 1 Tool 2-3 Tools 4+ Tools 0 Maintain 1 Maintain 2 Maintain 3+ Maintain 0 Hours 1 Hour 2-8 Hours 9+ Hours No toolserver Toolserver
Agree/Strongly Agree 66.67% 54.84% 72.97% 70.27% 60.00% 53.13% 73.33% 75.76% 50.00% 55.17% 81.48% 74.19% 55.17% 68.57% 70.00% 81.82% 65.08% 69.05%
Disagree/Strongly Disagree 10.48% 16.13% 5.41% 10.81% 0.00% 18.75% 13.33% 3.03% 11.11% 20.69% 3.70% 6.45% 17.24% 2.86% 16.67% 0.00% 7.94% 14.29%

I receive useful information via labs-announce / labs-l mailing list.

edit
 
All 1 Year 2-3 Years 4+ Years 0 Tools 1 Tool 2-3 Tools 4+ Tools 0 Maintain 1 Maintain 2 Maintain 3+ Maintain 0 Hours 1 Hour 2-8 Hours 9+ Hours No toolserver Toolserver
Agree/Strongly Agree 52.88% 40.00% 54.05% 62.16% 20.00% 45.16% 56.67% 66.67% 17.65% 44.83% 74.07% 61.29% 32.14% 48.57% 63.33% 90.91% 49.21% 58.54%
Disagree/Strongly Disagree 16.35% 6.67% 18.92% 21.62% 10.00% 9.68% 13.33% 27.27% 11.76% 13.79% 11.11% 25.81% 10.71% 17.14% 26.67% 0.00% 11.11% 24.39%

Documentation is easy-to-find

edit
 
All 1 Year 2-3 Years 4+ Years 0 Tools 1 Tool 2-3 Tools 4+ Tools 0 Maintain 1 Maintain 2 Maintain 3+ Maintain 0 Hours 1 Hour 2-8 Hours 9+ Hours No toolserver Toolserver
Agree/Strongly Agree 45.71% 58.06% 43.24% 37.84% 50.00% 43.75% 43.33% 48.48% 44.44% 51.72% 40.74% 45.16% 55.17% 45.71% 26.67% 72.73% 50.79% 38.10%
Disagree/Strongly Disagree 51.43% 35.48% 56.76% 59.46% 40.00% 53.13% 56.67% 48.48% 50.00% 44.83% 55.56% 54.84% 41.38% 54.29% 66.67% 27.27% 44.44% 61.90%

How do you compare the support you received when using Toolserver with the support you receive when using Tool Labs?

edit

Only the participants who had indicated previous Toolserver use were shown this question. Therefore, the responses in the following table are based on this subset of participants. This was an optional question, and 4% of the participants chose not to respond.

 
All 1 Year 2-3 Years 4+ Years 0 Tools 1 Tool 2-3 Tools 4+ Tools 0 Maintain 1 Maintain 2 Maintain 3+ Maintain 0 Hours 1 Hour 2-8 Hours 9+ Hours No toolserver Toolserver
Better 34.15% 33.33% 0.00% 41.94% 0.00% 12.50% 50.00% 31.58% 33.33% 28.57% 46.15% 27.78% 50.00% 31.25% 35.71% 0.00% 0.00% 34.15%
As good 36.59% 0.00% 42.86% 38.71% 0.00% 25.00% 35.71% 42.11% 33.33% 14.29% 23.08% 55.56% 12.50% 37.50% 50.00% 33.33% 0.00% 36.59%
Worse 29.27% 66.67% 57.14% 19.35% 0.00% 62.50% 14.29% 26.32% 33.33% 57.14% 30.77% 16.67% 37.50% 31.25% 14.29% 66.67% 0.00% 29.27%

Tool Labs documentation is comprehensive.

edit
 
All 1 Year 2-3 Years 4+ Years 0 Tools 1 Tool 2-3 Tools 4+ Tools 0 Maintain 1 Maintain 2 Maintain 3+ Maintain 0 Hours 1 Hour 2-8 Hours 9+ Hours No toolserver Toolserver
Agree/Strongly Agree 46.67% 54.84% 43.24% 43.24% 60.00% 40.63% 46.67% 48.48% 50.00% 44.83% 40.74% 51.61% 51.72% 51.43% 33.33% 54.55% 49.21% 42.86%
Disagree/Strongly Disagree 49.52% 41.94% 54.05% 51.35% 20.00% 56.25% 53.33% 48.48% 38.89% 51.72% 55.56% 48.39% 37.93% 48.57% 63.33% 45.45% 46.03% 54.76%

Tool Labs documentation is clear.

edit
 
All 1 Year 2-3 Years 4+ Years 0 Tools 1 Tool 2-3 Tools 4+ Tools 0 Maintain 1 Maintain 2 Maintain 3+ Maintain 0 Hours 1 Hour 2-8 Hours 9+ Hours No toolserver Toolserver
Agree/Strongly Agree 49.52% 58.06% 37.84% 54.05% 50.00% 46.88% 43.33% 57.58% 38.89% 48.28% 51.85% 54.84% 58.62% 45.71% 43.33% 54.55% 49.21% 50.00%
Disagree/Strongly Disagree 47.62% 38.71% 59.46% 43.24% 30.00% 53.13% 56.67% 39.39% 50.00% 51.72% 48.15% 41.94% 34.48% 51.43% 56.67% 45.45% 47.62% 47.62%

Service usage

edit

A section of questions on the frequency of use of various services offered by Tool Labs was included in the survey. The responses to these questions may be used to prioritize support and/or new service development by the Tool Labs administrators. These questions will not necessarily appear in future surveys.

How often have you used LabsDB access?

edit
 
All 1 Year 2-3 Years 4+ Years 0 Tools 1 Tool 2-3 Tools 4+ Tools 0 Maintain 1 Maintain 2 Maintain 3+ Maintain 0 Hours 1 Hour 2-8 Hours 9+ Hours No toolserver Toolserver
Often 44.90% 29.63% 47.06% 54.05% 11.11% 44.44% 43.33% 56.25% 12.50% 44.00% 51.85% 56.67% 28.00% 38.24% 67.86% 45.45% 39.66% 52.50%
Seldom/Sometimes 32.65% 25.93% 35.29% 35.14% 44.44% 18.52% 33.33% 40.63% 43.75% 32.00% 29.63% 30.00% 40.00% 35.29% 25.00% 27.27% 32.76% 32.50%
Never 18.37% 37.04% 11.76% 10.81% 22.22% 33.33% 20.00% 3.13% 25.00% 24.00% 18.52% 10.00% 20.00% 23.53% 7.14% 27.27% 24.14% 10.00%

How often have you used Webservices?

edit
 
All 1 Year 2-3 Years 4+ Years 0 Tools 1 Tool 2-3 Tools 4+ Tools 0 Maintain 1 Maintain 2 Maintain 3+ Maintain 0 Hours 1 Hour 2-8 Hours 9+ Hours No toolserver Toolserver
Often 59.60% 46.43% 67.65% 62.16% 11.11% 59.26% 53.33% 78.79% 35.29% 64.00% 57.69% 70.97% 52.00% 64.71% 62.07% 54.55% 55.17% 65.85%
Seldom/Sometimes 27.27% 32.14% 20.59% 29.73% 55.56% 18.52% 33.33% 21.21% 35.29% 12.00% 38.46% 25.81% 28.00% 26.47% 31.03% 18.18% 31.03% 21.95%
Never 11.11% 14.29% 11.76% 8.11% 11.11% 22.22% 13.33% 0.00% 17.65% 24.00% 3.85% 3.23% 12.00% 8.82% 6.90% 27.27% 10.34% 12.20%

How often have you used Cronjobs?

edit
 
All 1 Year 2-3 Years 4+ Years 0 Tools 1 Tool 2-3 Tools 4+ Tools 0 Maintain 1 Maintain 2 Maintain 3+ Maintain 0 Hours 1 Hour 2-8 Hours 9+ Hours No toolserver Toolserver
Often 53.61% 53.57% 54.55% 52.78% 11.11% 40.74% 73.33% 58.06% 11.76% 52.00% 73.08% 62.07% 20.83% 54.55% 72.41% 72.73% 56.14% 50.00%
Seldom/Sometimes 25.77% 25.00% 21.21% 30.56% 22.22% 25.93% 20.00% 32.26% 29.41% 20.00% 23.08% 31.03% 29.17% 30.30% 24.14% 9.09% 22.81% 30.00%
Never 18.56% 14.29% 24.24% 16.67% 44.44% 33.33% 6.67% 9.68% 47.06% 28.00% 3.85% 6.90% 41.67% 15.15% 3.45% 18.18% 17.54% 20.00%

How often have you used Redis?

edit
 
All 1 Year 2-3 Years 4+ Years 0 Tools 1 Tool 2-3 Tools 4+ Tools 0 Maintain 1 Maintain 2 Maintain 3+ Maintain 0 Hours 1 Hour 2-8 Hours 9+ Hours No toolserver Toolserver
Often 8.42% 8.00% 12.12% 5.41% 0.00% 4.00% 10.00% 12.90% 0.00% 8.33% 7.69% 13.79% 0.00% 5.88% 14.29% 18.18% 10.91% 5.00%
Seldom/Sometimes 12.63% 8.00% 12.12% 16.22% 11.11% 4.00% 10.00% 22.58% 6.25% 4.17% 15.38% 20.69% 4.55% 17.65% 14.29% 9.09% 10.91% 15.00%
Never 74.74% 76.00% 75.76% 72.97% 66.67% 92.00% 80.00% 58.06% 75.00% 87.50% 76.92% 62.07% 81.82% 76.47% 67.86% 72.73% 74.55% 75.00%

How often have you used GridEngine Continuous jobs?

edit
 
All 1 Year 2-3 Years 4+ Years 0 Tools 1 Tool 2-3 Tools 4+ Tools 0 Maintain 1 Maintain 2 Maintain 3+ Maintain 0 Hours 1 Hour 2-8 Hours 9+ Hours No toolserver Toolserver
Often 35.05% 51.85% 27.27% 29.73% 11.11% 30.77% 43.33% 37.50% 12.50% 32.00% 50.00% 36.67% 16.67% 29.41% 46.43% 63.64% 36.84% 32.50%
Seldom/Sometimes 21.65% 18.52% 24.24% 21.62% 22.22% 11.54% 20.00% 31.25% 18.75% 4.00% 26.92% 33.33% 16.67% 23.53% 25.00% 18.18% 21.05% 22.50%
Never 41.24% 22.22% 48.48% 48.65% 44.44% 57.69% 36.67% 31.25% 56.25% 64.00% 23.08% 30.00% 58.33% 47.06% 28.57% 18.18% 38.60% 45.00%

Free form responses

edit

The survey included several free form response sections. Survey participants were told that we would only publicly share their responses or survey results in aggregate or anonymized form. The free form responses include comments broadly falling into these categories:

  • Documentation (33 comments)
  • Stability and performance (18 comments)
  • Version control and Deployment (14 comments)
  • Logs, Metrics, and Monitoring (12 comments)
  • Package management (10 comments)
  • SGE (8 comments)
  • Database (7 comments)
  • Account/Tool creation (5 comments)
  • SSH (5 comments)
  • Other (11 comments)

Documentation

edit

In the responses mentioning documentation either directly or indirectly, generally the respondents were looking for more detailed how-to, getting started, and tutorial content. The need for more documentation for database contents was mentioned several times. Multiple respondents felt they did not know enough or have enough free time to contribute to the on-wiki documentation themselves.

Stability and performance

edit

Stability and performance concerns often mentioned NFS. Some respondents are hoping for more proactive notification of planned and emergency downtime.

Version control and Deployment

edit

Deployment comments mentioned the lack of a clear best practices and easy workflows for getting tools and their dependencies deployed to the servers. The version control comments were very similar in nature.

Logs, Metrics, and Monitoring

edit

Monitoring for job failures and errors in user developed applications topped the comments related to logs, metrics and monitoring. Some facility for automated health checks and email alerts was mentioned several times.

Package Management

edit

Package management comments either asked for specific system wide packages to be installed or lamented the difficulty in managing libraries needed for particular tools.

Sun Grid Engine comments generally mentioned the unexpected need to restart jobs and difficulty understanding how to pick the correct options for running jobs.

Database

edit

Database comments ranged from complaints about using MySQL Workbench, to the naming of replica.my.cnf, to lack of EXPLAIN command support, and less open data than Toolserver provided.

Account/Tool creation

edit

Account creation comments centered on the difficulty of registering for a Tool Labs account, largely mirroring the documentation concerns.

Comments on ssh mentioned the complexity of using ssh in general and frustration with bastions and tunneling.