Wikimedia CH/Software Development Guidelines
This is a work in progress. It is NOT an official document. This is an early stub.
Thank you for any early contribution to this stub ❤️ |
Software Development Guidelines edit
- For Wikimedia CH
The Wikimedia movement consist in millions of software.
Wikimedia CH contributors can help, and this document can be useful to learn how.
Note: this is a stub, a work in progress. This is not an official document. Thank you for your contributions! ❤️
The Software Development Guidelines helps technical contributors to have successful software development on a Wikimedia project. It is especially important for Wikimedia CH technical contractors, including Innovative Projects.
Preamble edit
While Wikimedia CH is not a software company, it co-exist with Wikipedia that has billions of visits every month, and it is just a piece of the global Wikimedia infrastructure.
Your help will support Wikimedia volunteers. It can happen that the project leader is actually a volunteer. So, thank you for being even more proactive in communicating your progresses.
If a difficult bug stops you, don't worry! Share your challenges in the public. It's easier to fix a problem if somebody—in a public place—tried to describe and analyze part of it.
For the purpose of this document, some common words have specific meanings, as per RFC2119:
|
Onboarding edit
At the beginning of your work, to contribute as a Wikimedia developer, you will need to create some new accounts.
Disclaimer: You are responsible for the security of your devices and the robustness of your passwords. Don't register into a service re-using passwords from other services. Note that every credential is personal. If you think your account is compromised, report to your supervisor immediately. |
You will need a total of two accounts to be used in four separate websites:
- Wikimedia account
- Meta-wiki: global Wikimedia account (account registration)
This registration provides a valid identity to login in meta.wikimedia.org, mediawiki.org, wikipedia.org, phabricator.wikimedia.org and additional collaborative websites for the general community. - Wikimedia Phabricator: bug tracking (do your first login)
Visit the page and use the button on the left to do the login (the one called "Log in or Register MediaWiki")
If you are redirected to a login page in "mediawiki.org", just enter the credentials related to your global Wikimedia account.
- Meta-wiki: global Wikimedia account (account registration)
- Developer account
- Wikitech: dedicated technical wiki (account registration)
This separate registration provides access to important separated services, like wikitech.wikimedia.org, gitlab.wikimedia.org and other resources for the technical community. - Wikimedia GitLab: collaborate on source code (do your first login)
Just enter your credentials of Wikitech
- Wikitech: dedicated technical wiki (account registration)
Don't panic. It is a total of two credentials.
Take your time to customize your account preferences and become more familiar with these different powerful platforms.
Team edit
Once created the accounts as mentioned above, it is good practice to customize your own user pages in a way which is meaningful and helpful for others.
A good public user page is short, nice, useful and updated.
The first principle is that your public user page is not intended for self-promotional purposes, but to communicate your background, your purposes, and to declare your conflict of interests, your role, your paid assignments. The purpose of this is to help others understand where you can contribute on what area and why.
Here's a handy checklist to follow when starting one's assignment or when a role changes:
- https://meta.wikimedia.org/
- Login. Visit your user page (usually it's mentioned on the top-right corner)
- add info about your role, in English
- Example: "Hello! In YEAR I'm paid to work on project ABC for Wikimedia CH. My specific role is: developer"
- mention your nickname of Phabricator
- Example: "On Phabricator you can find me as @supermario-wmch"
- mention your nickname of Wikitech
- Example: "On Wikitech I'm this user: https://wikitech.wikimedia.org/wiki/User:SuperMario-WMCH"
- https://wikitech.wikimedia.org/
- Login. Visit your user page
- add a link pointing to your Meta-wiki user page
- https://gitlab.wikimedia.org/
- Login. Select your profile picture > Edit profile.
- add a link to your Meta-wiki user page
You have success if your user page in Meta-wiki helps others in finding you in all other services, and if people from somewhere-else (for example Wikitech) can easily find your Meta-wiki account.
Please help your coworkers in doing the same.
Bonus point: on your Meta-wiki user page you can mention each other, to help others find your coworkers and better understand your team divisions. Creating a template for your team would be an easy-manageable solution to keep everyone connected without the need of reviewing a lot of user pages for any change in the team.
Where do I start? edit
Now that you have the necessary accounts to act, where should you start contributing?
The first entry point to Wikimedia CH software should be Phabricator: this tool can be used for track projects, act on software and to foster communication among members.
There are several projects affiliated to WMCH:
By visiting the links above you can figure out what the current projects are and how the community looks like.
Please don't start to contribute just yet, complete the reading of the remainder of this document.
Software Guidelines edit
The Wikimedia technical community maintains hundreds of thousands of software projects.[1][2] Your help is important to follow basic best practices.
Operating System edit
Software SHOULD be designed in a way that it executes successfully on Debian GNU/Linux stable.[3][4]
The reason is, among others, Debian stable is the only operating system supported by Wikimedia Cloud Services. This is true for both OpenStack and Kubernetes.[3]
To install your dependencies you SHOULD adopt these sources and in this order of preference:
- Debian apt with repository "main" or "security"
- Debian apt with repository "backports"
- Flatpak / snap
- Docker
To install your dependencies your SHOULD NOT use these sources:
- manual download of ".deb" packages from the web
- download of generic software from the web
- execution of installation scripts like "wget" in pipe at sudo
Things that MUST NOT be used:
- adoption of the Debian apt repository "non-free"
- download of any software that is not released under a Free/Libre and Open Source license
Supported Software edit
Example of solutions that MUST NOT be used:
- adoption of proprietary dependencies (both in content and in software)
- adoption of proprietary user trackers (such as Google Analytics etc.) - instead you can adopt WMCH Matomo
- adoption of external web resources (such as Google Font etc.) - instead you can serve them as local resources
Example of solutions that SHOULD NOT be used:
- adoption of unstable / nightly software as base
- adoption of esoteric programming languages
Since the operating system adopted in the Wikimedia movement is Debian stable, there are well-known stable versions that SHOULD be supported.
Your server-side software SHOULD NOT be incompatible with these versions:
- Docker 1.5
- MariaDB 10.5
- Node.JS 12
- PHP 7.4
- Python 2.9
- Ruby 2.7
Your client-side software SHOULD NOT be incompatible with these versions:
- Chromium 108
- Firefox 91
In addition to the above software versions, please note that further software versions, officially supported at least by Wikimedia Toolforge, are listed in the following page:
If these versions are too outdated for you, as already mentioned, check the next section for some acceptable workarounds.
Editors edit
In order to efficiently develop and write code, tools like the basic code editors or the full-fledged IDEs are an absolute necessity. There are hundreds of tools out there, but we can suggest a couple as industry-standard suggestions:
- Eclipse is a widely used IDE that can effectively assist in develop PHP code
- VS Codium is a freely-licensed binary distribution version of the most used code editor in the world.
Software Versions Workarounds edit
The indicated versions could be obsolete for your needs. In that case you can follow one of these accepted workarounds:
- PHP 8+ - you may need to adopt the reliable and compatible repository https://sury.org/
- Node.JS 13+ - you may need to adopt the official Docker image (https://hub.docker.com/_/node)
- Ruby 2.8+ - you may need to adopt the official Docker image (https://hub.docker.com/_/ruby)
These and similar workarounds allow flexibility with Debian stable compatibility, still with sources that are trusted and has security updates.
Infrastructure edit
Quick summary of the main available infrastructures to be know:
Wikimedia Foundation Infrastructure edit
Wikimedia Foundation already provides a structured, modern infrastructure, without costs to the end-users, able to minimize fragmentation, costs, complexity, gatekeeping. This infrastructure also provides effective user management, so, more efficient collaboration.
Here is an overview of the two separate types of access, and what they provide:
Meta-wiki provides a global account to access collaborative public access tools, including:
- Wikipedia
- the Free encyclopedia
- Wikidata
- The largest collaborative database of structured data
- MediaWiki.org
- website documenting MediaWiki (the software running Wikipedia)
- Wikimedia Phabricator
- collaborative bug tracker and more
- and more
Wikitech provides a separate account, to access restricted technical tools, including:
- Wikimedia GitLab
- code hosting with git
- Wikimedia Cloud services
- virtual private servers (VPS) on OpenStack (PaaS)
- Wikimedia Horizon
- OpenStack administration panel
- Wikimedia Toolforge
- shared hosting based on Kubernetes (IaaS)
- and more
Note that all of these services are hosted on machines physically controlled by the Wikimedia Foundation. It is not normally required in any way to adopt third party services like SaaSS. For example, when "GitLab" is mentioned, we normally mean https://gitlab.wikimedia.org/ and not https://gitlab.com/ etc.
Wikimedia CH Infrastructure edit
The infrastructure of Wikimedia CH is Switzerland-based and it's completely separated from the one of Wikimedia Foundation.
Infrastructure Comparison edit
This table helps in finding the infrastructure best suited for your project. In short:
#Wikimedia Foundation Infrastructure | #Wikimedia CH Infrastructure | |
---|---|---|
Easy to add and remove users and set access roles | Yes | No |
Available to the entire Tech community | Yes | No |
OpenStack and Kubernetes | Yes | No |
May host data to be protected | No | Yes |
Supports different GNU/Linux distros than Debian | No | Yes |
Primary Data Center | United States | Switzerland |
Hosting edit
This table summarizes suitable hosting solutions according to your need:
Need | Implemented with | Platform to be adopted | Platform owner |
---|---|---|---|
shared hosting | Kubernetes | Wikimedia Toolforge | #Wikimedia Foundation Infrastructure |
VPS | OpenStack | Wikimedia Cloud | |
VPS | hypervisor | WMCH internal cloud | #Wikimedia CH Infrastructure |
To have a more comprehensive overview, check these resources:
- wikitech:Help:Cloud Services introduction
- wikitech:Help:Cloud Services introduction#Which service is right for you?
Here some use-cases:
- hosting a bot operating in read/write mode on wiki contents
- hosting a web tool useful for Wikimedia purposes with stable versions of PHP/Python/Ruby on Debian:
- hosting a web tool useful for Wikimedia purposes with recent software versions on Debian:
- hosting a tool with custom software on custom GNU/Linux:
- → WMCH internal cloud
Coding Conventions edit
Any new project SHOULD mention its Coding Conventions to help newcomers in the onboarding. In this way, contributors don't need to inspect random portions of code to have a guess.
For MediaWiki extensions and gadgets, the project SHOULD adopt the following Coding Conventions:
- mw:Manual:Coding conventions
- mw:Manual:Coding conventions/PHP
- mw:Manual:Coding conventions/JavaScript
- mw:Manual:Coding conventions/Database
- ...
If your project it's based on another software that already has different Coding Conventions, your project SHOULD adopt these.
A new project SHOULD adopt simple and libre tools to help newcomers in adopting the Coding Conventions successfully and without frictions. For example adopting PHP CodeSniffer etc.[5]
Documentation edit
People in the development team are the ultimate experts in their own creation. This is why we need your help to take care of the documentation.
The purpose of documentation is not to write it but to read it.
A good documentation helps in avoiding to abandon a project, or rewrite it from scratch, due to lack of shared knowledge.
The documentation must be released under a free license. Suggestions:
Sysadmin Documentation edit
We need your help to create a Sysadmin Documentation. This documentation will be useful to future GNU/Linux system engineers handling your service. The sysadmin documentation should be in English.
The goal is understanding how to (re)create a testing and a production environment and how to update, backup and restore.
Usually useful information to mention:
- software dependencies (packages to be installed in a new minimal Debian GNU/Linux stable)
- installation instructions (what apt command, etc.)
- configuration instructions (which system configuration files should be changed, etc.)
- references to other applications, repositories and documentation related to or useful for the deploy
- log file paths (those relevant for investigating application issues)
- Unix users in play (perhaps the app uses system users such as www-data or has custom users)
- system daemons related to the application (e.g. own custom systemd units, mention other services in use such as mariadb, apache2, nginx, etc.)
- listening ports (TCP/UDP) and from which app component
- security instructions
- which not paths should be externally exposed (example: ./my-temp etc.)
- hardening instruction
- paths that must be writable by the app (and thus assigned to a possible my-app user) during its normal operation (example: ./my-upload, ./my-tmp, etc.)
- paths that not should be writable by the app (and therefore assignable to root) (example: all executable files, ./my-conf, etc.).
- update notes
- application update procedure
- backup and restore procedure
- where on the filesystem are the user data to be preserved (example: ./my-upload etc.)
- which databases should be preserved
- how to enable any maintenance mode
Not all of these points may be relevant. If you feel something is missing, better add it.
The documentation must not contain any secret or password.
The Sysadmin Documentation can be a section in the README file in the project repository.
Development Documentation edit
A short documentation in English describing the structure of the project is useful to help other technical people to approach, orient and contribute to the software.
Information that should be covered:
- purpose of the project/challenges encountered
- structure of the project (to navigate the directories)
- how to test the code locally
- how to configure the application (testing / production)
- where to find application logs, how to examine
A section called "Development Documentation" in your README file of your project can be a good starting point to help other developers.
User Documentation edit
Any good software has good User Documentation. The User Documentation allows end-users to master the software.
Tips:
- start the draft in English, especially if you plan to make it multi-language in the future
- it is also okay to write it just in the language of your main destination community
- start the draft without paying too much attention to formatting (example: OK an Etherpad, a wiki, a README, ...)
- avoid proprietary tools from the beginning (example: avoid Google Docs)
Note: the user documentation is usually improved by non-technical users. So, it's probably better to adopt a wiki, than a README on git.
You can omit obvious details. For example you can omit steps already covered by in-application wizards, etc.
Server Inventory edit
Any server that is not inventoried sufficiently may risk elimination by the Wikimedia Foundation[6]
- verify that adopted servers are well-known and mentioned in the documentation
- check that any custom domain is mentioned as well
- check if the documentation mentions the "The Cloud VPS Instance lifecycle" sufficiently
Maintenance edit
Until the end of their assignment your team should take care of the application maintenance. Examples:
- take care of the initial setup
- perform routine maintenance to keep it running
- apply security updates on application dependencies
- apply security operating system updates (when applicable)
- verify that the #Backup procedures are working
Insights:
- wikitech:Cloud VPS Server Admin Log - SAL
- the official tool for reporting what was done during one's work on a server.
If the application is already in production:
- schedule and communicate the intervention windows required to perform maintenance activities that may cause downtime (polite 6-hour notice)
Backup edit
At the beginning of the assignment, the team sets up a simple automatic backup plan, first of all storing a copy on the same server where the application is located, to implement a first on-site backup.
Purpose of on-site backup: to allow developers to restore data from a point in time before a (their?) mistake, independently and quickly.
Minimum and recommended on-site backup parameters:
- Data to be saved: the minimum data needed to do a project restore
- Examples you can include: databases, private application configurations, user uploads, application logs of the day, ...
- Examples you can exclude: operating system files, files already published elsewhere (git), caches, logs already old, ...
- Frequency: every night
- Time of day: the night between 01:00 and 04:00 in UTC+1 (Switzerland, Geneve time)
- Data retention: 24 hours (one copy only, each new backup overwrites the oldest)
Once implemented, the backup should be supervised at least until the end of the assignment.
Tip: it it's useful, a simple on-site backup can be done thanks to a simple crontab
line, using simple tools like mysqldump
and/or rsync
etc.
The backup directory must not be accessible to the public or to users that are not trusted.
Suggested destination for your on-site backup in your VPS:
/var/backups/wmch/$HOSTNAME/daily/files...
Suggested permissions: chown app:app
with chmod o= ...
.
An on-site backup is not sufficient. It is just the first step to quickly setup an off-site backup.
Verify that both the backup procedure and its restore are well-defined in the #Sysadmin Documentation.
Code of Conduct edit
You (and your team) accept the following code of conducts:
- Universal Code of Conduct (applying to any project)
- mw:Code of Conduct (applying to Wikimedia technical spaces)
- Wikimedia CH/Respectful behavior space policy (applying to Wikimedia CH)
In short: be nice with others.
Terms of Service edit
You (and your team) accept the Wikimedia Foundation Terms of Service:
Software License edit
Any new software created for Wikimedia CH and to be used in Wikimedia projects must be released with a Free/Libre and Open Source license.
If you work in a company, be sure that the person in charge of your company allows you to release such software. Get it written down. Details:
Offboarding edit
Prior to the conclusion of a team member's assignment, that person follow this checklist:
- update the Team Documentation to reflect the role change
- communicate the accounts that need to be deactivated
- communicate the list of personal information that should be removed (not guaranteed to be removed)
- contribute to the relevant beautiful #Documentation
Communication edit
A good Communication helps users to be aware of development directions. An optimal Communication helps the team in avoiding design mistakes.
Let's start by saying that this is not that easy, since some volunteers could create a controversy if they are not involved in early phases, while some other people just want to choose to be not involved.
A Communication compromise is necessary since the community is big. Some volunteers are conservative, since the Wikimedia platforms they use are almost assimilated as a working desk, and any change can waste their time and create frustration.
The development team, on the other hand, should be able to have creative and positive space to deliver something new.
Communicate Early edit
Don't wait the final project conclusion to communicate progresses. This is important also because many projects will be probably never declared as completed. You might be surprised how long your software might last (as example, let's mention Wikipedia).
Do you have a new project challenge? do you have a new progress? do you have a new important bug? That can be a good moment to propose a date for a quick meeting to show that. It is not required to plan a one-hour presentation each day or each week. However, it would be a mistake not to dedicate five-minute to share your screen sometime.
Online meetings should be open to the public, in order to allow some people at least to join and listen. For this reason, the video conferencing platform should be libre (example: Jitsi, BigBlueButton, etc.) otherwise, technical contributors may be excluded and there wouldn't be much benefit.
Contact / Questions edit
If something is not clear, please share your opinion in the talk page or contact us:
Note edit
- ↑ https://phabricator.wikimedia.org/diffusion/
- ↑ https://gitlab.wikimedia.org/repos/?sort=stars_asc
- ↑ a b wikitech:Debian
- ↑ https://wiki.debian.org/DebianStable
- ↑ https://github.com/squizlabs/PHP_CodeSniffer
- ↑ (English) m:wikitech:Help:Cloud VPS Instances#The Cloud VPS Instance lifecycle but also from Wikimedia CH.
- "Instances will be removed for projects that have been determined inactive."