Research:Cloud Services Annual Survey/2015

The 2015 Tool Labs survey collected feedback from Tool Labs' developers and maintainers on how the services offered can be improved to help their development and maintenance needs.

Participants

The participants of the survey were the members of the Wikimedia Tool Labs. 962 participants were contacted via email. The emails were harvested from Labs' LDAP system, which is available to everyone with a labs account. The list initially involved 977 participants. All those who had opted out of contacts via Wikitech were eventually excluded.

106 email recipients participated in the survey which represents 11% of those who are contacted.

Dates

The 2015 survey was active between 2015-09-25 and 2015-10-08.

Questions and answers

Basic demographics

Responses to the basic demographic questions were divided into roughly equal sized groups to provide "cohorts" of similar Tool Labs users. These groups were then plotted against each other in the qualitative responses to determine if there were significant differences in the opinions of the sub-groups.

How many years have you used Tool Labs?

Help: If you were a Toolserver user, please include the total number of years that you have used Toolserver and Tool Labs.

Cohort	Percentage
1 Year	29.81%
2-3 Years	36.54%
4+ Years	33.65%

Many of the respondents have been using Tool Labs for 2 or fewer years. In fact, nearly 30% of the respondents are using Tool Labs only for a year. This can be the sign of many young users or that young users are the ones that are more willing to participate in surveys. The distribution has a long tail indicating there are quite a few users who have been around for a long time.

How many tools have you developed in Tool Labs?

Help: We do not have a canonical definition of "tools". We are trying to estimate how much burden is on you, so please use your best judgement in assessing how many tools you have developed.

Cohort	Percentage
0 Tools	8.57%
1 Tool	31.43%
2-3 Tools	28.57%
4+ Tools	31.43%

40% of respondents have developed 0-1 tools. This may correlate with the amount of time that they have been active on Tool Labs. The distribution of the 4+ Tools cohort again has a long tail indicating that some tool developers are highly prolific.

How many tools do you actively maintain in Tool Labs?

Help: We do not have a canonical definition of "tools". We are trying to estimate how much burden is on you, so please use your best judgement in assessing how many tools you maintain.

Cohort	Percentage
0 Maintain	16.19%
1 Maintain	27.62%
2 Maintain	26.67%
3+ Maintain	29.52%

The maintenance numbers roughly correlate with the number of Tools developed although the raw numbers are skewed downward slightly. This seems to suggest that very few respondents have become maintainers of tools that they didn't build themselves and also possibly indicates that some tools have been abandoned or replaced over time.

How many hours per week do you spend developing or maintaining tools on Tool Labs?

Cohort	Percentage
0 Hours	24.51%
1 Hour	34.31%
2-8 Hours	29.41%
9+ Hours	11.76%

Over 50% of respondents spend an hour or less per week on average working on their tools.

Were you a developer and/or maintainer in the Toolserver as well?

Cohort	Percentage
No Toolserver	59.43%
Toolserver	40.57%

The majority of survey responses were received from users with no history using Toolserver. It is currently unknown how this distribution correlates with the full number of active Tool Labs users.

Additional demographics

What programming languages do you prefer to program in?

For this question, the participants had the following choices: Perl, PHP, Python 2, Python 3, Ruby, or they could enter the language of their choice. The following table shows the ranked list of languages in terms of the percentage of time they were chosen as the language of choice. Note that participants could report more than one language.

Language	%
PHP	58.49%
Python3	37.74%
Python2	36.79%
JavaScript	9.43%
Perl	8.49%
Ruby	7.55%
Other	7.55%
Java	6.60%
C#	4.72%

High usage of PHP and Python were not unexpected results. MediaWiki itself is written in PHP and Python is both widely used for data analysis tasks and well supported by the Pywikibot community. Freeform responses were normalized to make this analysis easier. The JavaScript numbers include responses of "JavaScript", "node", and "nodejs". The "Other" grouping includes several common scripting tools and a few very esoteric languages.

When you develop a tool, what percentage of your work is done locally (as opposed to remotely on Tool Labs)?

Help: For example, enter 30 if 30% of your development work is done locally.

Answers were rounded up to the nearest 5% for analysis.

Answer	Percentage reported
0	8.65%
5	3.85%
10	9.62%
20	2.88%
25	0.96%
30	3.85%
50	9.62%
60	3.85%
70	10.58%
75	2.88%
80	16.35%
85	1.92%
90	9.62%
95	5.77%
100	9.62%

Do you prefer using source control to manage your tool's source code?

Yes	No
73%	26%

Qualitative questions

A series of qualitative questions were asked in order to gauge general satisfaction with Tool Labs. Since this was the first survey of this type given to the Tool Labs community these results will form a historical baseline to compare the results future surveys against.

Services have high reliability (up time)

	All	1 Year	2-3 Years	4+ Years	0 Tools	1 Tool	2-3 Tools	4+ Tools	0 Maintain	1 Maintain	2 Maintain	3+ Maintain	0 Hours	1 Hour	2-8 Hours	9+ Hours	No toolserver	Toolserver
Agree/Strongly Agree	63.81%	67.74%	54.05%	70.27%	70.00%	65.63%	76.67%	48.48%	66.67%	65.52%	70.37%	54.84%	68.97%	57.14%	63.33%	72.73%	68.25%	57.14%
Disagree/Strongly Disagree	34.29%	29.03%	43.24%	29.73%	20.00%	31.25%	23.33%	51.52%	22.22%	34.48%	29.63%	45.16%	24.14%	42.86%	36.67%	27.27%	28.57%	42.86%

It is easy to write code and have it running on Tool Labs

	All	1 Year	2-3 Years	4+ Years	0 Tools	1 Tool	2-3 Tools	4+ Tools	0 Maintain	1 Maintain	2 Maintain	3+ Maintain	0 Hours	1 Hour	2-8 Hours	9+ Hours	No toolserver	Toolserver
Agree/Strongly Agree	68.57%	61.29%	67.57%	75.68%	50.00%	53.13%	73.33%	84.85%	55.56%	58.62%	74.07%	80.65%	51.72%	77.14%	66.67%	90.91%	66.67%	71.43%
Disagree/Strongly Disagree	24.76%	25.81%	27.03%	21.62%	0.00%	40.63%	26.67%	15.15%	16.67%	34.48%	25.93%	19.35%	31.03%	22.86%	30.00%	0.00%	23.81%	26.19%

I feel I am supported by the Tool Labs team when I contact them via labs-l, wikimedia-labs IRC channel, or phabricator.

	All	1 Year	2-3 Years	4+ Years	0 Tools	1 Tool	2-3 Tools	4+ Tools	0 Maintain	1 Maintain	2 Maintain	3+ Maintain	0 Hours	1 Hour	2-8 Hours	9+ Hours	No toolserver	Toolserver
Agree/Strongly Agree	66.67%	54.84%	72.97%	70.27%	60.00%	53.13%	73.33%	75.76%	50.00%	55.17%	81.48%	74.19%	55.17%	68.57%	70.00%	81.82%	65.08%	69.05%
Disagree/Strongly Disagree	10.48%	16.13%	5.41%	10.81%	0.00%	18.75%	13.33%	3.03%	11.11%	20.69%	3.70%	6.45%	17.24%	2.86%	16.67%	0.00%	7.94%	14.29%

I receive useful information via labs-announce / labs-l mailing list.

	All	1 Year	2-3 Years	4+ Years	0 Tools	1 Tool	2-3 Tools	4+ Tools	0 Maintain	1 Maintain	2 Maintain	3+ Maintain	0 Hours	1 Hour	2-8 Hours	9+ Hours	No toolserver	Toolserver
Agree/Strongly Agree	52.88%	40.00%	54.05%	62.16%	20.00%	45.16%	56.67%	66.67%	17.65%	44.83%	74.07%	61.29%	32.14%	48.57%	63.33%	90.91%	49.21%	58.54%
Disagree/Strongly Disagree	16.35%	6.67%	18.92%	21.62%	10.00%	9.68%	13.33%	27.27%	11.76%	13.79%	11.11%	25.81%	10.71%	17.14%	26.67%	0.00%	11.11%	24.39%

Documentation is easy-to-find

	All	1 Year	2-3 Years	4+ Years	0 Tools	1 Tool	2-3 Tools	4+ Tools	0 Maintain	1 Maintain	2 Maintain	3+ Maintain	0 Hours	1 Hour	2-8 Hours	9+ Hours	No toolserver	Toolserver
Agree/Strongly Agree	45.71%	58.06%	43.24%	37.84%	50.00%	43.75%	43.33%	48.48%	44.44%	51.72%	40.74%	45.16%	55.17%	45.71%	26.67%	72.73%	50.79%	38.10%
Disagree/Strongly Disagree	51.43%	35.48%	56.76%	59.46%	40.00%	53.13%	56.67%	48.48%	50.00%	44.83%	55.56%	54.84%	41.38%	54.29%	66.67%	27.27%	44.44%	61.90%

How do you compare the support you received when using Toolserver with the support you receive when using Tool Labs?

Only the participants who had indicated previous Toolserver use were shown this question. Therefore, the responses in the following table are based on this subset of participants. This was an optional question, and 4% of the participants chose not to respond.

	All	1 Year	2-3 Years	4+ Years	0 Tools	1 Tool	2-3 Tools	4+ Tools	0 Maintain	1 Maintain	2 Maintain	3+ Maintain	0 Hours	1 Hour	2-8 Hours	9+ Hours	No toolserver	Toolserver
Better	34.15%	33.33%	0.00%	41.94%	0.00%	12.50%	50.00%	31.58%	33.33%	28.57%	46.15%	27.78%	50.00%	31.25%	35.71%	0.00%	0.00%	34.15%
As good	36.59%	0.00%	42.86%	38.71%	0.00%	25.00%	35.71%	42.11%	33.33%	14.29%	23.08%	55.56%	12.50%	37.50%	50.00%	33.33%	0.00%	36.59%
Worse	29.27%	66.67%	57.14%	19.35%	0.00%	62.50%	14.29%	26.32%	33.33%	57.14%	30.77%	16.67%	37.50%	31.25%	14.29%	66.67%	0.00%	29.27%

Tool Labs documentation is comprehensive.

	All	1 Year	2-3 Years	4+ Years	0 Tools	1 Tool	2-3 Tools	4+ Tools	0 Maintain	1 Maintain	2 Maintain	3+ Maintain	0 Hours	1 Hour	2-8 Hours	9+ Hours	No toolserver	Toolserver
Agree/Strongly Agree	46.67%	54.84%	43.24%	43.24%	60.00%	40.63%	46.67%	48.48%	50.00%	44.83%	40.74%	51.61%	51.72%	51.43%	33.33%	54.55%	49.21%	42.86%
Disagree/Strongly Disagree	49.52%	41.94%	54.05%	51.35%	20.00%	56.25%	53.33%	48.48%	38.89%	51.72%	55.56%	48.39%	37.93%	48.57%	63.33%	45.45%	46.03%	54.76%

Tool Labs documentation is clear.

	All	1 Year	2-3 Years	4+ Years	0 Tools	1 Tool	2-3 Tools	4+ Tools	0 Maintain	1 Maintain	2 Maintain	3+ Maintain	0 Hours	1 Hour	2-8 Hours	9+ Hours	No toolserver	Toolserver
Agree/Strongly Agree	49.52%	58.06%	37.84%	54.05%	50.00%	46.88%	43.33%	57.58%	38.89%	48.28%	51.85%	54.84%	58.62%	45.71%	43.33%	54.55%	49.21%	50.00%
Disagree/Strongly Disagree	47.62%	38.71%	59.46%	43.24%	30.00%	53.13%	56.67%	39.39%	50.00%	51.72%	48.15%	41.94%	34.48%	51.43%	56.67%	45.45%	47.62%	47.62%

Service usage

A section of questions on the frequency of use of various services offered by Tool Labs was included in the survey. The responses to these questions may be used to prioritize support and/or new service development by the Tool Labs administrators. These questions will not necessarily appear in future surveys.

How often have you used LabsDB access?

	All	1 Year	2-3 Years	4+ Years	0 Tools	1 Tool	2-3 Tools	4+ Tools	0 Maintain	1 Maintain	2 Maintain	3+ Maintain	0 Hours	1 Hour	2-8 Hours	9+ Hours	No toolserver	Toolserver
Often	44.90%	29.63%	47.06%	54.05%	11.11%	44.44%	43.33%	56.25%	12.50%	44.00%	51.85%	56.67%	28.00%	38.24%	67.86%	45.45%	39.66%	52.50%
Seldom/Sometimes	32.65%	25.93%	35.29%	35.14%	44.44%	18.52%	33.33%	40.63%	43.75%	32.00%	29.63%	30.00%	40.00%	35.29%	25.00%	27.27%	32.76%	32.50%
Never	18.37%	37.04%	11.76%	10.81%	22.22%	33.33%	20.00%	3.13%	25.00%	24.00%	18.52%	10.00%	20.00%	23.53%	7.14%	27.27%	24.14%	10.00%

How often have you used Webservices?

	All	1 Year	2-3 Years	4+ Years	0 Tools	1 Tool	2-3 Tools	4+ Tools	0 Maintain	1 Maintain	2 Maintain	3+ Maintain	0 Hours	1 Hour	2-8 Hours	9+ Hours	No toolserver	Toolserver
Often	59.60%	46.43%	67.65%	62.16%	11.11%	59.26%	53.33%	78.79%	35.29%	64.00%	57.69%	70.97%	52.00%	64.71%	62.07%	54.55%	55.17%	65.85%
Seldom/Sometimes	27.27%	32.14%	20.59%	29.73%	55.56%	18.52%	33.33%	21.21%	35.29%	12.00%	38.46%	25.81%	28.00%	26.47%	31.03%	18.18%	31.03%	21.95%
Never	11.11%	14.29%	11.76%	8.11%	11.11%	22.22%	13.33%	0.00%	17.65%	24.00%	3.85%	3.23%	12.00%	8.82%	6.90%	27.27%	10.34%	12.20%

How often have you used Cronjobs?

	All	1 Year	2-3 Years	4+ Years	0 Tools	1 Tool	2-3 Tools	4+ Tools	0 Maintain	1 Maintain	2 Maintain	3+ Maintain	0 Hours	1 Hour	2-8 Hours	9+ Hours	No toolserver	Toolserver
Often	53.61%	53.57%	54.55%	52.78%	11.11%	40.74%	73.33%	58.06%	11.76%	52.00%	73.08%	62.07%	20.83%	54.55%	72.41%	72.73%	56.14%	50.00%
Seldom/Sometimes	25.77%	25.00%	21.21%	30.56%	22.22%	25.93%	20.00%	32.26%	29.41%	20.00%	23.08%	31.03%	29.17%	30.30%	24.14%	9.09%	22.81%	30.00%
Never	18.56%	14.29%	24.24%	16.67%	44.44%	33.33%	6.67%	9.68%	47.06%	28.00%	3.85%	6.90%	41.67%	15.15%	3.45%	18.18%	17.54%	20.00%

How often have you used Redis?

	All	1 Year	2-3 Years	4+ Years	0 Tools	1 Tool	2-3 Tools	4+ Tools	0 Maintain	1 Maintain	2 Maintain	3+ Maintain	0 Hours	1 Hour	2-8 Hours	9+ Hours	No toolserver	Toolserver
Often	8.42%	8.00%	12.12%	5.41%	0.00%	4.00%	10.00%	12.90%	0.00%	8.33%	7.69%	13.79%	0.00%	5.88%	14.29%	18.18%	10.91%	5.00%
Seldom/Sometimes	12.63%	8.00%	12.12%	16.22%	11.11%	4.00%	10.00%	22.58%	6.25%	4.17%	15.38%	20.69%	4.55%	17.65%	14.29%	9.09%	10.91%	15.00%
Never	74.74%	76.00%	75.76%	72.97%	66.67%	92.00%	80.00%	58.06%	75.00%	87.50%	76.92%	62.07%	81.82%	76.47%	67.86%	72.73%	74.55%	75.00%

How often have you used GridEngine Continuous jobs?

	All	1 Year	2-3 Years	4+ Years	0 Tools	1 Tool	2-3 Tools	4+ Tools	0 Maintain	1 Maintain	2 Maintain	3+ Maintain	0 Hours	1 Hour	2-8 Hours	9+ Hours	No toolserver	Toolserver
Often	35.05%	51.85%	27.27%	29.73%	11.11%	30.77%	43.33%	37.50%	12.50%	32.00%	50.00%	36.67%	16.67%	29.41%	46.43%	63.64%	36.84%	32.50%
Seldom/Sometimes	21.65%	18.52%	24.24%	21.62%	22.22%	11.54%	20.00%	31.25%	18.75%	4.00%	26.92%	33.33%	16.67%	23.53%	25.00%	18.18%	21.05%	22.50%
Never	41.24%	22.22%	48.48%	48.65%	44.44%	57.69%	36.67%	31.25%	56.25%	64.00%	23.08%	30.00%	58.33%	47.06%	28.57%	18.18%	38.60%	45.00%

Free form responses

The survey included several free form response sections. Survey participants were told that we would only publicly share their responses or survey results in aggregate or anonymized form. The free form responses include comments broadly falling into these categories:

Documentation (33 comments)
Stability and performance (18 comments)
Version control and Deployment (14 comments)
Logs, Metrics, and Monitoring (12 comments)
Package management (10 comments)
SGE (8 comments)
Database (7 comments)
Account/Tool creation (5 comments)
SSH (5 comments)
Other (11 comments)

Documentation

In the responses mentioning documentation either directly or indirectly, generally the respondents were looking for more detailed how-to, getting started, and tutorial content. The need for more documentation for database contents was mentioned several times. Multiple respondents felt they did not know enough or have enough free time to contribute to the on-wiki documentation themselves.

Stability and performance

Stability and performance concerns often mentioned NFS. Some respondents are hoping for more proactive notification of planned and emergency downtime.

Version control and Deployment

Deployment comments mentioned the lack of a clear best practices and easy workflows for getting tools and their dependencies deployed to the servers. The version control comments were very similar in nature.

Logs, Metrics, and Monitoring

Monitoring for job failures and errors in user developed applications topped the comments related to logs, metrics and monitoring. Some facility for automated health checks and email alerts was mentioned several times.

Package Management

Package management comments either asked for specific system wide packages to be installed or lamented the difficulty in managing libraries needed for particular tools.

SGE

Sun Grid Engine comments generally mentioned the unexpected need to restart jobs and difficulty understanding how to pick the correct options for running jobs.

Database

Database comments ranged from complaints about using MySQL Workbench, to the naming of replica.my.cnf, to lack of EXPLAIN command support, and less open data than Toolserver provided.

Account/Tool creation

Account creation comments centered on the difficulty of registering for a Tool Labs account, largely mirroring the documentation concerns.

SSH

Comments on ssh mentioned the complexity of using ssh in general and frustration with bastions and tunneling.