Grants talk:IEG/Lua libs for behavior-driven development

"Coverage" of docs and tests

A very coarse check of coverage of docs and tests for some language versions of Wikipedia. It is not a very thorough check, it is just a check on existence of pages.

		dawiki		dewiki		enwiki		nowiki		nnwiki		svwiki		all
		Num	Rel	Num	Rel	Num	Rel	Num	Rel	Num	Rel	Num	Rel	Comment
1	/	121	56 %	172	63 %	2248	80 %	2073	94 %	20	37 %	123	55 %	Should find all subpages
2	/do[ck][^/]*\$	69	32 %	107	39 %	661	24 %	1000	45 %	5	9 %	45	20 %
3	^[^/]*/do[ck][^/]*\$	53	25 %	63	23 %	348	12 %	102	5 %	4	7 %	36	16 %
4	/test[^/]*/	0	0 %	3	1 %	9	0 %	0	0 %	1	2 %	1	0 %	Noisy, contains /doc
5	/test[^/]*\$	3	1 %	7	3 %	165	6 %	23	1 %	4	7 %	14	6 %
6	/[ck]onf[^/]*/	2	1 %	3	1 %	21	1 %	19	1 %	0	0 %	0	0 %	Noisy, contains /doc
7	/[ck]onf[^/]*\$	8	4 %	4	1 %	23	1 %	12	1 %	1	2 %	1	0 %
8	/[^/]*dat[ae]n?/	2	1 %	1	1 %	620	22 %	1746	79 %	0	0 %	0	0 %	Noisy, contains /doc
9	/[^/]*dat[ae]n?\$	5	2 %	1	1 %	52	2 %	8	0 %	0	0 %	2	1 %
10	All Module-pages	216		272		2796		2207		54		223
11	Real modules	95	44 %	100	37 %	548	20 %	134	1%	34	63 %	100	45 %
12	Modules with tests		3 %		7 %		30 %		17 %		12 %		14 %
13	Modules with doc		56 %		63 %		64 %		76 %		12 %		36 %

Nearly all the columns called "Rel" have relative numbers normalized to row 10, "All Module-pages", except for row 12. The relative numbers can be said to be the amount of pages of a specific type within the namespace. When dawiki has 56 % in row 1, then 56 % of the pages in the Module namespace is a subpage. Some projects use a lot of subpages to configure modules, which is visible in row 1 and 8 for enwiki and nowiki.

An interesting row is row 11 "Real modules", which is the difference between all module-pages and all subpages in that namespace, and row 12 "Modules with tests", which is row 5 normalized against row 11. Ideally the numbers in row 11 should be 100 %. Numbers below 100 % means that we have modules without tests. At some projects there are almost no tests at all. At enwiki there are an impressive number of tests, even if it should be better there too. Other projects lags considerably behind.

The row 13 "Modules with doc" is made like row 12, but the relative numbers comes from row 3.

There are a lot of spurious hits in the rows 4, 6, and 8, but none of those hits are removed. It is the titles ending in a forward slash (/) that are wrong, as a lot of the pages are documentation subpages. They are retained here as it is a pretty obvious question what happen in those subspaces.

An ultimate goal of the project would be to move from the present situation in row 12 to something more like the present situation in row 13. Tests should come naturally as part of the development of the code. — Jeblad 19:46, 12 April 2016 (UTC)

Showcases/documentation in project plan?

Under "Risks" you acknowledge that no matter how good the result turns out, there's a risk that very few users would actually use it. Lots of wikipedia users (most?) writing Lua code are not pro developers, and may not familiar with BDD. So I think good documentation and showcases could be essential for this to be a success. You note that "it could be an idea to make some showcases that directly compares different ways to write tests", but haven't included this in the project plan. Would it be an idea to do so? Danmichaelo (talk) 17:43, 14 April 2016 (UTC)

I did not include showcases as this is only a three month project. There will be sufficient documentation to write tests (a reference manual), possibly as LuaDoc, but only a very limited text on how to write them (a programmers manual) on the usual /doc page. If the code will be in an extension (it probably will be), then the doc page must be a separate help page. It will probably contain references to text books providing best practices, but not giving the rationale for why something are considered best practices.

We don't have a good solution for how to update on-wiki reference manuals from off-wiki texts. Only solution is to add external links. This is more or less the same problem as with the rest of the code for Mediawiki, but a lot more visible here as the Lua-code is for the general community and not for the pro developers. — Jeblad 07:46, 15 April 2016 (UTC)

On the title of the proposal

The name is not quite accurate. We want to support behavior-driven development but to do that we provide the tools to do spec-style unit testing. See RSpec for an example. We can't really make the developers do behavior-driven development, but we can expose tested modules and make the community expect tests to be in place. — Jeblad 08:24, 15 April 2016 (UTC)

Eligibility confirmed

This Individual Engagement Grant proposal is under review!

We've confirmed your proposal is eligible for review and scoring. Please feel free to ask questions and make changes to this proposal as discussions continue during this community comments period (through 2 May 2016).

The committee's formal review begins on 3 May 2016, and grants will be announced 17 June 2016. See the round 1 2016 schedule for more details.

Questions? Contact us at iegrantswikimedia · org .

--Marti (WMF) (talk) 05:11, 28 April 2016 (UTC)

Scribunto extension changes?

Hello!

Do you know if changes to mw:Extension:Scribunto are required to be written and deployed in order for this proposal to be completed successfully? If so, have you reached out to the maintainers of that extension to discuss it? Thanks! YuviPanda (talk) 03:27, 16 May 2016 (UTC)

@YuviPanda: No changes to Scribunto are necessary for this extension. The extension is strictly for testing code from, and within, the module namespace. The "extension" in this case is nothing more than a Lua lib, localization, and a few gadgets. And no, I'm not using Busted. — Jeblad 22:03, 16 May 2016 (UTC)

If this is meant to be used by many projects, perhaps it would make sense to include it in the scribunto standard libraries list (e.g. Like what happened with mw.html library). Although, if so, it would probably be better to wait and see how popular it is. Bawolff (talk) 11:03, 21 May 2016 (UTC)

It needs localization, so it is more than just a standard library. It also adds a Javascript gadget to make it possible to rerun tests. — Jeblad 13:48, 21 May 2016 (UTC)

Goals and Measurable Results

Check against Grants:Evaluation/Program Goals and Measurable Results. — Jeblad 16:00, 29 May 2016 (UTC)

Aggregated feedback from the committee for Lua libs for behavior-driven development

Scoring rubric	Score
(A) Impact potential Does it have the potential to increase gender diversity in Wikimedia projects, either in terms of content, contributors, or both? Does it have the potential for online impact? Can it be sustained, scaled, or adapted elsewhere after the grant ends?	7.1
(B) Community engagement Does it have a specific target community and plan to engage it often? Does it have community support?	6.4
(C) Ability to execute Can the scope be accomplished in the proposed timeframe? Is the budget realistic/efficient ? Do the participants have the necessary skills/experience?	7.8
(D) Measures of success Are there both quantitative and qualitative measures of success? Are they realistic? Can they be measured?	6.3
Additional comments from the Committee: Yes, we need more use of Lua and more Lua developers. Personally, I consider myself a techie and will not go near Lua with a ten-foot-pole. In the Wikiverse we are continually "testing in production" and it is time we have a workspace for testing, especially when it comes to complicated templates using Lua. The impact when making mistakes can be too big for some ideas to ever be implemented otherwise. The proposal does not present a compelling case for how the lack of a standardized testing infrastructure for Lua modules adversely impacts the development or use of such modules in practice, or how developing such an infrastructure would result in any measurable impact to Wikipedia content. The idea is interesting and there is a need to develop something more friendly. Increasing the quality of code that is written on wiki is very important. Templates/Lua are some of the defining features of MediaWiki, since it puts a massive amount of power in the hands of editors. Proper software development practices haven't really impacted on-wiki development as prominently yet. This looks like a good start that can be reused across wikis. However, developing only the framework isn't enough--some actual conversion needs to happen. I'd highly recommend adding tests to at least one popular Lua module be part of the proposal’s plan. Fairly MediaWiki Lua module extension case. It helps deliver a more robust Lua module. I agree that the properties are not linked with an online impact. It's very likely to be used if Lua module author wants so. But WMF might want to consider the cost of integrating the plugin. I doubt anyone on English Wikipedia would be willing to go for it, but lots of Norwegians have endorsed and I respect Jeblad's expertise if Norwegian Wikipedia is the testing ground. I would support it if successful there. The proposed measures of success are largely focused on the success of the technical implementation, but do not effectively measure the adoption of the new tool by the community or its impact on the quality of Wikipedia content. I have concerns that something could help for an interpreter language. Easy to measure number of tests / coverage of modules :) Great addition. Though Lua, internal CI, coding editor all in Mediawiki sounds much wrong. But that's the decision from MediaWiki. I don't doubt its impact. The proposed schedule and budget appear reasonable for a development effort of this scale. This project has already been developed. Nice to see the endorsements, and I agree that Jeblad has a good track record. There is minimal evidence of engagement with the existing Lua developer community; this lack of engagement / support is highlighted as a risk in the proposal, but no effective mitigation strategy is proposed. I think it is important for the proposal to include targeting a specific community or a specific, highly-used Lua module, and to make writing tests. I don't see mention of that, but the proposal could be modified to take that into account. I understand the problem, but I am unsure if the project should be funded, as the scope is too limited for users who create templates with Lua. This would not affect the majority of Wikipedia users, but only those who create modules. This proposal seems like a plunge in the dark, but worth the risk in my opinion. Interesting if there is a need in the community, but I am unclear if there is one. I would know if the community really asks for a tool like this if it's a niche product. It needs preliminary analysis (considering the workload and the cost). I am concerned about costs/benefits because the benefits are more individual and not supported by a community need. I like the idea of providing tools to improve the development of Lua code for Wikipedia, and the idea of supporting someone who is already doing this. I would support this if the proposal was modified to add: 'write tests (with community acceptance) for at least one high profile Lua module in a wiki project providing a % of test coverage.' Creating Lua is especially difficult to accomplish the moment, but we need this. I will also ask for a better breakdown of the budget, i.e. how many hours/week. The 50% implies part time but it isn't clear how many hours that would be.

-- MJue (WMF) (talk) 00:44, 3 June 2016 (UTC) on behalf of the IEG Committee.

Some answer, I'm not sure about all the statements/questions.

• I am one of those crazy dudes that has used both Scheme and Forth. Lua is weird, and that makes it a bit fun. Developing in Lua is a continuous back and fort between the source, the console, testing in templates, checking if something breaks, and then often saving just to see that 5623 transcluded templates did break. So you start over again. And again. And again.
• Not many dare to touch the template mess, or fix it with Lua, because it is so extremely fragile. You touch something with that ten-foot-pole and something explode. The real problem is all those involved templates, but to fix them we need to write some modules, but then to do that we need better and more efficient testing. We try to fix ten years of complete mess with a toothpick and some floss. Perhaps we need that ten-foot-pole to drive out the garbage.
  I guess it is also about how and what you define as content. Templates are pretty much about presentation of content, and fixing them would not make the content any better, even if the presentation would be better and more efficient.
• I'm not sure if I would call it more friendly. My idea when I wrote the proof of concept was clean tests for limited set of cases, and to make the cycles rapid. I wanted to get out of the situation where I repeatedly posted the same tests in the console.
• I was afraid the "one popular module" as it can easily spiral down into fights. I have listed a few modules that can be used to test some core functionality, and those should be pretty safe. Further demos could be done at the end of the project to avoid impact of disagreements over the color of the dogs house.
  I have an idea about a follow-up, its about user case testing from the general editor population. That would be really interesting, allowing editors to easily probe modules as part of their discussion posts.
  This project is strictly about the on-wiki developers own testing.
• Not quite sure about this point. Visibility of testable code might impact the willingness to write tests. Yes, there will be a cost of integrating the plugin, but I believe it will be fairly low. Server load is the major problem if people starts to write very heavy tests.
• I guess some testing will be done at Norwegian Wikipedia, and in fact it seems like we are somewhat more inclined to do testing than other major wikis. That in itself is a bit interesting.
• I have proposed a way to measure the success of the adoption of the new tool, but there are no good solution to measure the impact on the content itself. One indirect success indicator could be the adoption of testable modules compared to untested ones, ie the impact on the rendered pages, but we don't have any baseline and no clear way to normalize the data.
• Not sure about this.
• Number of tests are easy, coverage is not easy. It is possible to figure out if tests hits public interfaces (members) by monkey-patching, but not internal coverage. It is possible to do monkey-patching internally in a module, but it creates a bit dirty code.
• It is about fixing a problem, ... an interpreted language in an server running an interpreted language, ...
• I hope to get more focus on tests, not just "it seems to work" or "it did work when I wrote it".
• :)
• I wrote some code to demo that it was in fact doable, ie a proof of concept. It is not completed.
• I was a bit amazed by the endorsements. :)
• Lua development is strange. Most of the development is copy-pasting existing code from other projects. What I really misses is interaction with Lua developers outside the Wikimedia-umbrella. There are extremely good libraries out there, and we don't use them.
• I'm a bit afraid of starting to write tests for a highly visible module early on, that can easily derail the whole project. There should be sufficient examples to write more tests, but those examples should be easy to understand and adapt.
• Problems with modules impacts all users.
• :)
• I believe the need is evident from the lack of good testable modules. I would not call the existing testing regime on most wikis functional, but that is me. :)
• :)
• Note that real coverage in Lua as it is used in Wikimedia-projects is a hard problem.
• A normal week in Norway is 37.5 hours.

There are additional info on mw:Help:Spec. — Jeblad 01:02, 19 June 2016 (UTC)

Round 1 2016 decision

Congratulations! Your proposal has been selected for an Individual Engagement Grant.

The committee has recommended this proposal and WMF has approved funding for the full amount of your request, $12,000

Comments regarding this decision:
The committee is pleased to support the development of an extension for testing of Lua-modules. We appreciate the expertise you bring as both a Wikimedia volunteer and a participant in the broader developer community. We look forward to seeing how you will combine that expertise to ensure your tool will achieve impact among the existing community of Wikimedia volunteers currently using Lua.

Next steps:

1. You will be contacted to sign a grant agreement and setup a monthly check-in schedule.
2. Review the information for grantees.
3. Use the new buttons on your original proposal to create your project pages.
4. Start work on your project!

Questions? Contact us.

Extension request, new completion date

Request for a change to the completion date and start date for our grant

I am requesting an extension of the project completion date for "Lua libs for behavior-driven development".

1. The new proposed completion date is 30. June 2019. Current development, mostly cleanup, started 22. Dec. 2018.
2. The extended delay is all on me me, getting tired of the project that grew way out of size. It was a combination of several factors. One was a slight adjustment of scope, that I should have rejected because it changed the project from the initial quite narrow focus. Another was that I had to do more documentation than expected, which also extended the scope from a purely technical project. In short I ran out of time, but still with a lot left to do. I failed to see the consequences of small feature creeps (also my own due to generalizations) and then backed away.
3. The project is still viable, and the code is mostly done and in good shape. For now I'm cleaning up the technical doc, and expect to start on the missing parts shortly. I am not going to promise any fast closure of the project, as I assume there are at least two months full-time work before its finished.

Note that the help page is at mw:Help:Pickle, and the extension page is at mw:Extension:Pickle. Repository is at Github: jeblad/Pickle, with project page at [1]. Technical documents are organized in the on-site wiki, with formatting that can be reused in a mediawiki instance if necessary. If it works out there should be a Vagrant role available soonish.

Sorry for the delay. — Jeblad 13:54, 3 January 2019 (UTC)

@Jeblad: Thanks for the explanation, Jeblad. I'm sorry that the project grew beyond the scope you expected it to, and I understand why that would be overwhelming and make it difficult to complete the work. This extension for 30 June 2019 has been approved. I also appreciate your summary of the completed work so far. Thanks, I JethroBT (WMF) (talk) 18:43, 4 January 2019 (UTC)

Please note that the new final report due date is 30 July 2019. Thank you. -- JTud (WMF), Grants Administrator (talk) 23:06, 4 January 2019 (UTC)