Grants:IdeaLab/Application scoring system

status: idea
Application scoring system
idea creator:
project contact:
Build a scoring system that allows multiple reviewers to score IEG proposals against a set of predefined criteria. System should be flexible enough to be reused for other types of applications (event scholarships, etc) later on.
created on: 20:46, 14 March 2014

Project idea


What is the problem you're trying to solve?


Individual Engagement Grants review process currently uses an ad hoc scoring system involving Google Forms to collect input from reviewers. It is unwieldy and won't scale well as we have more proposals to review.

Findings from initial investigations into alternative solutions:

  • The movement has demonstrated a need for scoring tools - many other programs across the Wikimedia movement are also using Google Forms to score applications for things like scholarships.
  • Outside of our movement, other grant programs that use group review processes on large numbers of applications often do so in-person - applications are physically scored and marked.
  • Wikimania scholarships rely on a scoring system that is hosted by WMF. This system appears to have functionality that could meet needs for IEG and other programs where scoring is needed, but the current application is not flexible enough to be extended for uses beyond Wikimania scholarships. A new application could be built off of this existing code base, if necessary.

What is your solution?


Build a flexible scoring system to meet our needs.

Project goals

  • Develop a scoring solution to meet the needs of IEG review by end of September 2014.
  • Make this solution available for other programs where scoring is needed.



Refine requirements


Analysis of the problem would start from a few core user stories:

As an administrator of the grant review system
I want to create new grant campaigns (defining reviewers, review criteria, data to be reviewed, and data grouping)
So that reviewers can provide feedback on the applications in the campaign.
As a grant administrator or reviewer
I want to mark grant applications as eligible or ineligible
So that ineligible grant applications can be excluded from further review.
As a grant reviewer
I want to score a grant application on multiple dimensions (defined by the admin)
So that the quality of the grant application can be compared to others in the same campaign.
As a grant provider or administrator
I want to view reports on the aggregate reviewer scores for grant applications in a campaign
So I can share results with others outside of the review system and select applications to fund.

The core user stories would be fleshed out with additional details based on interviews with the IEG team and other interested parties within the Wikimedia movement to produce a list of core requirements. (for example, an IEG committee member has already started a rev on a spec, which could be discussed and incorporated)

Decide on a solution


The second phase would seek to make an analysis of existing FOSS applications that could satisfy these core requirements. This analysis would be used to make a "build versus buy" decision on whether to create a solution based on existing software or undertake the creation of a brand new project to satisfy the requirements.

Implement solution


The third phase will be the implementation of the build versus buy decision. The activities needed here will obviously vary based on which avenue is chosen, but will likely require some software development in either case. If an existing application is found to be a close match the third phase would involve configuring the application to satisfy the requirements and be deployable to the Wikimedia Foundation production cluster. Implementation of an existing product would quite likely also involve fixing small to medium complexity upstream bugs and adding small to medium size features. If instead the decision is to build a new solution the third phase would involve creating a new project that leverages as much existing code as is reasonable from the Wikimania Scholarships application.

Get involved