Grants talk:Programs/Wikimedia Community Fund/WikiJournal 2023

Review follow-up questions

Hello, @Mikael Häggström, @Thomas Shafee, and @Evolution and evolvability. Thanks for your application. Will you please answer the following questions by Monday October 31, 2022, 5pm EDT to help the committee to review of your application? Thank you!

1) Please provide more current readership statistics, from 2019-2022 (most current public stats are 2018).

Readership comes through a few different avenues (in particiular, some articles are present in copies on both wikiversity and wikipedia). Here are the 2022 readership numbers via different platforms.

• Readership from 2019-01-01 to 2022-10-29 on Wikivesity (combined across all journals)
• Readership from 2019-01-01 to 2022-10-29 on Wikipedia (by journal) for the subset of articles also copied to Wikipedia:
  • WikiJournal of Medicine
  • WikiJournal of Science
  • WikiJournal of Humanities
• In addition, people may access directly through DOI links, which is a useful proxy for what proportion of readers came via academic channels. Referrals from 2021 to 2022 that came through crossref by people clicking DOIs:

Months	2022-09	2022-08	2022-07	2022-06	2022-05	2022-04	2022-03	2022-02	2022-01	2021-12	2021-11	2021-10	2021-09	2021-08	2021-07	2021-06	2021-05	2021-04	2021-03	Monthly Average
Resolution Attempts	13,263	10,582	8,711	11,147	17,737	24,315	17,339	11,087	9,276	10,579	14,729	22,311	13,026	10,994	10,487	14,824	16,561	8,819	13,254	13,273

Compared with referrals from 2019:

Months	2019-11	2019-10	2019-09	2019-08	2019-07	2019-06	2019-05	2019-04	2019-03	2019-02	2019-01	2018-12	2018-11	Monthly Average
Resolution Attempts	8,043	6,965	3,518	24,093	15,926	32,060	11,395	6,881	6,895	7,494	10,669	12,600	12,270	12,216

Note: reports from Crossref were not received during 2020.

• Eventually, citation metrics (particularly in the form of the notourious impact factor) will be also useful from an academic readhership point of view. We are still working on implementation of Extension:WikiSEO which would improve the accuracy of Google Scholar estimates (since it would ensure embedded metadata within the DOI-targetted pages themselves). Note that impact factor is a contentious metric in many acadmic circles, and other stats like readership and altmetrics are a highly significant point of difference for the WikiJournals as compared to other academic journals. Additionally, although impact factor is simple to calculate, it is a proprietary metric and we are not yet officially calculated for us by clarivate. In the meantiome, the provisional manual calculations can be seen here.

2) Please share a job description for the CEO and a more explicit org chart/explain better to whom this position reports (looks like all 80 board members?). Please share job descriptions for the Administrative Assistant & the Project Coordinators. How many Project Coordinators are there?

Job descriptions and org chart have been added as a staffing plan:

• CEO
• Administrative Assistant
• Project Coordinators
• Technical Editors (role implemented since 2022)

The administrative board at the top of the org chart is a smaller board consisting of the elected Editor in Chief of each journal (to ensure that the interests of all member journals are included), as well as other elected members interested in the administrative/organisation aspects of the project.

3) You mention in your 2021 application having "a strong interest in having editorial boards that represents geographic areas equally". What is the geographic make-up of your editorial boards, and how has this changed since 2021?

Editorial board and tech editor geographic diversity

The only 2022 editorial board additions were one from the USA and one from S. Korea. We also hired 9 technical editors in 2022. We began diversifying the location of tech editors. 1 is located in Middle East and Africa region (Sub-Saharan Africa), 2 belong to Northern and Western Europe. Payroll across regions remains an ongoing challenge to further diversification.

4) How many articles have been published in 2021 and 2022? What is the breakdown between the different journals? Please provide article publications, and not just submissions, as a report of last year and as a metric for this cycle.

Publication numbers by year by journal. It should be noted that submissions, whether published or rejected, consume time and labour as well as generating value in the form of the assessments on their discussion pages (see e.g. declined articles). Some submissions were abandoned by the authors partway and would not be reflected in the publication numbers despite time input so far by volunteer editors, volunteer peer reviewers, and paid technical editors (assisting in arranging peer reviews).

5) Re: Budget - please provide a more detailed line-item breakdown for marketing expenses and Wikimania expenses. What venues will you use for marketing purposes? How do you arrive at the $10k figure for marketing expenses?

a) Detailed line-item breakdown

• Ads (work with an ad agency or online platform) - $3500
• Graphic design - $1250 (logo, journal design, social media)
• Social media
  • Linkedin boosts (min = $10 a day) - $3650
  • Creation of social media posts $600
• Videos
  • Creating promotional videos/webinars to promote WikiJournal - $1000

b) Venues for marketing purposes?

1. Linkedin, Twitter, Facebook and Instagram
2. ResearchGate and Academia.edu
3. Podcasts and other online platforms

c) How do you arrive at the $10k figure for marketing expenses?

The above venues are the only ones with line item expenses. Additional Marketing through academic peer networks and incidentally attended symposia and conferences not included as costs but will be significant marketing routes outside of purely online modes

d) What audiences will you be targeting for marketing?

We will be targeting both the general public and academics. The academic target will aim to expand the authorship and visibility of the journal, particularly in psychology, psychiatry and behavior science where some editorial board members expressed an interest in expanding the WikiJournal publications in psychology-related areas. The general public will be targeted to increase readership thus increasing metrics and general visibility.

How will you know if your marketing is successful? That is, how will you assess this particular goal?

The primary goals are awareness (currently the project is not well known in academic circles) and engagement (especially article subsmission, peer reviewer response rates, readership, and citation)

1. Direct metrics (e.g. views and clicks) where possible for online material
2. Indirect metrics (e.g. submissions, readership, citations) through these will not be linkable to any single specific marketing facet. Note: Peer reviewer response rate is unlikely to be significantly affected by marketing in the near future since those filling this role are primarily cold-contacted by editors requesting reviews.
3. Surveying (authors can be surveyed as part of existing processed; readers likely to have lower response rates)

6) In 2021, the WMF funded $13,000 for back end development work, and it looks like the task list for 2022 is the same. What work was completed in 2021 with the $13,000 from that funding cycle?

We did not find a suitable developer during 2021, so the amount was rolled forwards to 2022, where it was then applied to technical development for automating inclusion of published articles journal issues

7) Did you receive more than one quote for your insurance coverage? Please share the quotes on which you are basing the $40,000 figure.

The insurance coverage was marketed to 17 carriers. Due to the relationship between WikiJournal and Wikipedia (although tangential) the carriers were hesitant to provide coverage as the “wiki” in our name opened Wikijournal up to unique exposures. This has been outlined in previous budget proposals. Unfortunately we are still in a “hard market” when it comes to the coverage we require (more information on relevant market hardness in the bi-annual industry benchmark report" "The State of the Market").

List of carriers marketed: Old Republic; Scottsdale; Great; American; USLI; Navigators; Aspen; AIG; CFC; Kinsale; RSUI; Hudson; Intact; Ironshore; Hiscox; Amtrust; ANV

Hudson is the current carrier that we use. Our 2022-07-08 to 2023-07-08 premium is $35,365. Premium increases are in direct relation to gross revenue. In addition, we are currently in a hard market and there is also an inflation rate of ~7% in the United States where we procure our coverage. We wanted to account for these factors when asking for grant money in 2023. Year over year, the 2021 to 2022 rates increased by a factor of 11.7%. We are not sure what the rates will be from 2023-07-2024-07, but expect a similar increase - especially with inflation.

8) Who is using the proposed phone plan?

The phone number would be for external contact of WikiJournal. This would likely ring through to the administrative assistant, but no decisions have been made as of yet. In the meantime, it could be manned by Mikael and/or Sarah divvying up the duties for answering the phone and forwarding messages to the relevant parties. This would enable the WikiJournals to be contactable via a central phone number so that any author, interested potential board member or editor, etc. would have a direct way of reaching the journal. Currently the only methods are text based (i.e. email, wiki accounts of members, or other accounts such as LinkedIn when people google board members).

Emjackson42 (talk) 14:48, 24 October 2022 (UTC)

Some follow-up questions

@Mikael Häggström, Thomas Shafee, and Evolution and evolvability: Thanks for responding so far. After the above responses, I have a few follow-up questions:

• you mention that DOI clicks are a proxy for academic readership. This might have been true at some point, but in your current layout (eg on [1] ) it is actually very tempting to click on the DOI link in that design, rather than 'full text', so i anticipate some amount of false positives. Have you looked at the referrers for those clicks by chance and excluded wikimedia projects from those numbers, as well as perhaps automated traffic? What would be a benchmark from other journals for the number of visits to a DOI?
  • A good point - though definitely hard to test, as crossref doesn't provide a detailed breakdown of where the click comes from. It might even be easier to work from the opposite end and try to work out what portion of all pageviews are referred from within wikiversity and which from without - though I don't know if any existing tools that are set up for that. T.Shafee(Evo﹠Evo)^talk 06:42, 4 November 2022 (UTC)

• You correctly point out that submissions also cost time when they are rejected. How many submissions have received serious attention in the past years, and were still rejected (i.e. were not desk-rejected)?
  • The best way to pull these is probably from the time taken via viewing this query - some example reviews of decline papers from the recent years: [2] [3] [4] [5] (a couple of older expeptions with faster rejection after reviewer comments are [6] and [7]). One of the challenges is that authors often ghost after receiving significant reviewer criticism. However, other authors who go silent for long periods have come back much later to sufficiently update an article for the reviewers to be satisfied, so have been very conservative so far about finalising rejections before we're certain that authors will not adress them. This does skew some of the long tail of turnaround time stats though. T.Shafee(Evo﹠Evo)^talk 06:42, 4 November 2022 (UTC)

• Can you be a bit more specific as to what backend work you expect to perform this year? Given that you elected to spend the 2021 funds on a specific aspect, I would be curious to know if you can point out what the priorities are from that list this time. Are you expecting to execute the entire list?
  • The key one is probably integration with Open Journal Systems. This has been on the todo list for a while but the members of the group principally responsible for identifyinga recruiting a suitable developer have been unable to put in adequte time over the last couple of years. This has been a real bottleneck, as the fiddling with tracking artilce progress (largely via google forms and spreadsheets) has affected A) retention of active editors due to fiddliness and ease of losing track of an article's processing stage and B) ensuring timely article processing which affects attracting new authors. T.Shafee(Evo﹠Evo)^talk 06:42, 4 November 2022 (UTC)

Thanks! Effeietsanders (talk) 20:34, 3 November 2022 (UTC)

Additional questions from regional committee member

Hello @Mikael Häggström, @Thomas Shafee, and @Evolution and evolvability. Thanks again for your application. I have a few more questions to add to @Emjackson42's - You can wrap your responses into one if that works as my queries are related.

I am mostly interested in your metrics.

9) Can you provide data related to last year's cycle around metrics? Specifically the number of article publications previously (organized by year and by journal)? Submission number is helpful but doesn't give us a sense of what's being published.

We've added these stats to the relevant question 4 above to keep the info all together

10) Related to this, can you provide a target number of published articles as a goal for this proposal cycle's metrics?

We believe approximately 20 articles is a realistic number considering that a lot of administrative work will be put on electing and onboarding the new positions. Once this is done, we can focus on shortening the backlog of existing submitted articles. Attracting further submissions after that is believed to be the relatively straightforward part - reaching out to both existing wikimedians (e.g. via WP:JAN) and more imporantly, non-wikimedians to .

This number would be significantly impacted by whether the proposed fourth journal in the group (wikiJPPB, see proposal here) is able to be launched during 2023. The main proposers had been waiting until the technical editors had become practiced before starting soliciting articles since they had been worries that the processes prior to 2021 might not cope wit significant increases or spikes in submissions, and were keen to avoid long turnaround and processing times. Whether the journal starts in 2023 will depend on their assessment on how well the technical editor programme has fared and the availability of key members in Q1 and Q2 of 2023.

11) What audiences will you be targeting for marketing for this year's proposal? How will you know if your marketing is successful? That is, how will you assess this particular goal?

We've included these as subsections of the related question 5 above

Thanks! Matthewvetter (talk) 13:10, 25 October 2022 (UTC)

Response deadline

@Mikael Häggström, @Thomas Shafee, and @Evolution and evolvability, we have just learned that there is more time for responses based on our decision timeline! A response by Nov. 3rd would be helpful for the committee. If that's not possible, we can accept a response by Nov. 6th. Thank you. Emjackson42 (talk) 14:49, 25 October 2022 (UTC)

I've started to fill in the already drafted answers now (drawing stats from wikidata and wikimedia tools where possible). We have also been drafting a few additional statistics on this page. We'll aim to get the responses filled in by 3rd of November. T.Shafee(Evo﹠Evo)^talk 11:30, 30 October 2022 (UTC)

I've now added responses to all the the questions above based on a combined draft by @Mikael Häggström, @OhanaUnited, @SHaggstrom and myself with help from others in the editorial boards as well as input from current technical editors. T.Shafee(Evo﹠Evo)^talk 13:44, 3 November 2022 (UTC)

For the additional questions added on the 4th of November (above and below), I'm answering these without corrsponding first with other group members to have timely responses. We'd definitely be happy to discuss in more detail as a group, since many of these touch on broader strategy but I appreciate that you'll likey be needing to make decisions before it's practical to organise a larger meeting. T.Shafee(Evo﹠Evo)^talk 06:46, 4 November 2022 (UTC)

Additional questions from committee member

@Mikael Häggström, Thomas Shafee, and Evolution and evolvability: Thanks for the submission and your responses so far. I have a few additional questions as well, apologies for the lateness:

• You describe that you are struggling to maintain the same model with volunteers as you're growing. How are you imagining further growth? Would you become even more dependent on funds and professionals? What would the role of community involvement be, and how would this long term be different from traditional academic publishers (even in open access)?
  • I think the main areas of growth of volunteer would be:
    • Growing editors in the editorial boards. Reaching out to new potentially interested parties has stalled quite a bit as we've ended up looking inwards and improving tools and processes to try to fix the slow turnaround ussues. Deeply involved editorial board members are slow to cultivate and require volunteers reaching out with confidence that the technical and process side of the involvement (e.g. keeping track of articles), won't overwhelm the editorial side (organising peer review).
    • Growing authors. This requires some more systematic outreach, probably best achieved by someone with some dedicated time in e.g. a CEO role. It can also be done by A) volunteer editors when there is less pressure on back-office tasks and by other authors when they have good experiences (turnaround time, support, followup). This scales roughly with submission number as submissions have 1-10 authors (varies wildly between fields).
    • Growing peer reviewers. This more directly correlates with submission number, since all submissions get 2-3 reviewers. The main metric to optimise is actually probably response rate, since some articles currently take >50 email requests to secure two reviewers.
    • Note, there is some volunteer activity that we currently don't capture, e.g. assistance in pretifying diagrams by the Wikimedia Commons Graphic Lab, and wikiversity community members who 'wikignome' reference formatting and similar but who have not requested to be mmembers of the edituroal board or assoc editor parts of the user group.
  • Growth of non-volunteers:
    • The main issues that non-volunteer growth would address are the key, time sensitive activities which are difficult, and more importantly unreliable, to leave to volunteers. These tasks include those referred to in question 2. Timeliness would give opportunities for reputation building, reduce volunteer burnout on certain low-enjoyment tasks, and ensure that we better maintain the more complex tasks required for project management and technical development.
    • We expect that expanding our core paid labour foce will support and encourage larger growth in contribution by volunteers. This is because of the following initiatives:
      • The step-up proposed in this budget is intended to redress a hump in the growth curve, so that volunteer growth can be better nurtured directly (i.e. more systematic reaching out to potential partner academic organisations as sources of volunteers)
      • Fully organic growth can proceed as volunteers have smoother experiences with improved workflow and support, and so recommend the project to contacts they know to be interested and compatible with our ideals and processes.
  • The long-term role of community involvement will be simimlar in some ways to existing publishers and differnt in others:
    • Similarities: authors and peer reviewers are typically volunteer for almost all academic journal publishers (and most academic book publishers). Similarly editorial boards are often mostly volunteers for academic journals.
    • Differences (business model): frequent open-to-all meetings, transparent minutes even for non-open meetings, voting rights of authors, editors, and reviewers as user group members are all extremely different to most academic publishers.
    • Differences (cost): Fee-free open access is still uncommon for most academic fields, so the omission of article processing fees will always be a point of difference, especially given the unique value-add of tapping into the wikimedia ecosystem.
    • Differences (format): although not all articles are intended for wikipedia integration, those that are remain a highly unique format and one of the most impactful in terms of reach and getting WP contributions from expert communities and the subsequent evolution of those articles by the existing WP community.

T.Shafee(Evo﹠Evo)^talk 13:51, 4 November 2022 (UTC)

• What would be some benchmark journals that you're aiming to be comparable to in 3 years, and what would their equivalent metrics be like? I would also like to understand what your more traditional targeted metrics would be like: number of articles published, number of citations per year per article and maybe an impact factor? This is primarily to get a good understanding of your ambition, and what is needed to get there.
  • The long-term goal would be for significant, compounding volunteer growth as the WikiJournal model becomes A) known about in the first place and B) respected as a robust and useful format. The rate that this is possible would depend on the sort of funding scale up that the WMF is interested in committing longer-term. We've aimed to be viable with low-scale funding but with the ability to scle up if there were to be support to support a larger scale publishing house. The ideal small publisher comparators would be probably something like:
    • PeerJ (probably the most community-run current journal model) which started with one multidisciplinary journal, and now publishes 7 journals (10 years after its founding). It started off in 2011 with one of the lower budgets for a publishing group of $1M per annum so is more comparable in terms of potential growth. After funding, it published 30 articles by 2013 years, but published its 1000th in 2015 and had an IF of 2.5. With a lower budget, we'd be expecting lower grouth, but this gives an example of how fast size can increase. At the end of 2018, it had published 8k articles, 22k peer reviews, 39k authors, 1900 pageviews total per article. With only part time paid members, we'd expect to go for a lower-growth model, but likely significantly higher pageviews.
    • F1000 Research grew via a different model of just increasing the size of its single journal but partitionaing 'gateways' as effectively subjournals. It stated in 2012 (undislosed funding) and by 2015 was publishing >300 articles per year, an IF of just above 1.1 and per article pageviews of <1000. It was dissapointingly sold to one of the megapublishers in 2020, something that we've been deveoping clauses in out bylaws to prevent being possible.
    • I suspect we may end up with a somewhat higher IF because it is very skewed by review articles (of which we tend to publish a greater than normal proportion).
  • There are a few smaller publishing houses, but these are genrally supported within a university and typically have university staff time devoted to them, so it is quite hard to judge the exact resource input they actually have as it's not separated from the university (or university's library) as a whole. For example, ANU Press has 6 staff (though it's unclear if they are full-time), was started in 2002 and now publishes 10 journals each with 4-10 articles per annum in a fee-free open acccess model supported by the university.
  • A Small-Medium publisher like PLOS staretd with $9M funding in 2002 and by 2005 had 5 journals and was publishing 700 articles per year.

T.Shafee(Evo﹠Evo)^talk 13:51, 4 November 2022 (UTC)

• Your most-cited article in 2022 ([8] with 9 citations so far) has been almost identically been published in another journal ([9]). How common is this? Could you provide a bit more context?
  • In this case, the article in WikiJMed is an expansion on the points made in the Bipolar Disord (and the authors were keen on generating something open access where they didnt' have to hand over copyright to the publisher). This actually isn't particularly common generally (and the only example in the WikiJournals). However, this is one of a few similar collaborative formats that we'd be interested to explore if we have more capacity:
    • Approach authors to publish articles like the one you noted - a companion/followup to a paper or book chapter in a subscription or otherwise non-OA publication. STEM academic book chapters are particularly relevant because they're often extremely similar to WP pages in their topic breadth, but very rarely actually read beacuse the paywalls are often 10x higher than for journals.
    • Approach authors similar to the above, but >5-years after their original publication if in an OA journal and propose publishing an updated version - rather like an updated textbook.
    • Approach journals to co-publish single articles or possibly entire special issues (example paper co-published in two places here + here). This could be particularly good with those academic society journals whose area of interest is poorly covered in WP.

T.Shafee(Evo﹠Evo)^talk 13:51, 4 November 2022 (UTC)

Thank you for taking the time to respond! Effeietsanders (talk) 20:34, 3 November 2022 (UTC)

General Support Fund proposal approved in the amount of 116,900 USD

@Mikael Häggström and Evolution and evolvability:Congratulations! Your grant is approved in the amount of 116,900 USD with a grant term starting 1 January 2023 and ending 31 December 2023.

The committee supported continuing your programmatic work from last year to increase opportunities for underrepresented communities to publish academic works, as well as for readers to generally access high quality research and publications across numerous domains. The committee is broadly supportive of WikiJournal User Group’s purpose and broader goals in these areas, which they recognize as important gaps in the academic world that impact our movement and our ability to support knowledge equity. The needs for better geographical and socioeconomic representation on your editorial boards to support equitable decisionmaking is particularly well-suited to parts of Movement Strategy, such as building capacity to improve high-impact content in underrepresented communities and investing in experimental projects that address barriers to knowledge equity.

The committee also voiced important areas of concern and needs for clarity in the proposal, and has some recommendations for how to address these needs:

• Budgetary increases of this proportion from the previous year (almost 200% in this case) will generally not be possible to support in future requests. This is because the general budget available for funding increases is limited, even when there is a compelling rationale supporting this increased investment.

• WikiJournal User Group’s plan for long-term growth in its publications and organization through increasing submissions and technical contractors and other staff needed to process them is not an approach to growth we can sustainably support. While it’s common for affiliates to be unable to grow their work based on voluntary work alone, the committee is concerned that with increasing submissions, your organization would generally not be able to rely on volunteers and would quickly require staff to support this kind of growth. To address this concern, committee had several suggestions for your consideration:
  • to seek out other sources of funding to support this kind of staffing growth,
  • to exploring possible partnerships with academic associations or affiliates that could support some volunteer work in your organization, and
  • Revisiting your 2021-2024 organizational strategy to provide greater detail into your long-term priorities and plans, as well as to consider other priorities or improvements that depend less on supporting an increasing number of submissions across your publications. The committee is willing to provide some additional funding to support this strategic planning (e.g. for an external consultant) if a budget and brief methodology for this work can be shared with your program officer.

• While it’s helpful to understand the circumstances around your insurance needs and experiences, the committee did not support the requested insurance budget and would like it to be reduced in future proposals for the following reasons:
  • the kinds of insurance being requested were not clearly specified (possibly D&O and EPL based on last year’s request)
  • the rationale for why this insurance coverage is important for your operations is also not clearly specified, and
  • more generally, even with a strong rationale, the budget for insurance exceeds what our funding programs can realistically support year to year, especially if these rates are expected to increase. The amount of funding here (~40,000 USD) is substantial, roughly equivalent to a small- or mid-sized affiliate's annual budget in the region. The committee also notes that the rates specifically for D&O insurance are significantly lower for other Wikimedia affiliates where we have funded these needs (for example, see Art+Feminism’s 2023 budget, where the D&O rate is 1,350.00 USD).

The committee is appreciative your diligence in seeking out quotes from several providers around this area. It may be more helpful to reach out to other affiliates in the region to assess providers that could support a more reasonable rate. Your program officer can help connect you to some organizations around this need and discuss these comments further.

• The committee would like to see improvements in your evaluation plan and metrics for this year’s proposal. In particular, more specific learning questions should be provided rather than more general statements of what work you will be broadly learning from. At least one of these learning questions should include some indication of the impact of works published through WikiJournals.

These learning questions should then directly inform the metrics you collect. For example, the committee would have liked to have seen some indication of citations or how published works in WikiJournals are being used. Where possible, it would be helpful to see benchmarks with comparable publications. However, if benchmarks with other journals is not viable, the committee would still want to better understand the benefits or impact that WikiJournals have in the communities you support so that relative changes in your impact year over year can be better understood.

• Also related to impact, the committee would like to see better documentation of the ways in which the WikiJournal User Group is using Wikimedia projects and tools to invite readers and contributors to its publications. This documentation should be publicly and easily accessible, especially for academics, journal editors, or others who may be new to the Wikimedia movement and are exploring Wikimedia projects or open access opportunities to publish their works.

The committee looks forward to WikiJournal’s work over the next year, and is pleased to continue supporting your efforts to support better access to academic work and publication opportunities.

On behalf of the Regional Committee, I JethroBT (WMF) (talk) 20:47, 18 November 2022 (UTC)

So that the links are easier to find:

• initial: 2023 Budget plan.docx
• updated: 2023 Budget plan

T.Shafee(Evo﹠Evo)^talk 02:52, 20 September 2023 (UTC)