New Readers/Wikipedia Offline Medical Pilot

New Readers
Research and findings
	Outcome #1: Access
	Outcome #2: Awareness
	Outcome #3: Discovery
	Outcome #4: Retention
	Outcome #5: Syndication
Updates
FAQ and all pages

Objective

Through this pilot, healthcare providers at 20 medical clinics in Nigeria will have a Wiki Medicine hotspot and training that will allow them to use Wikipedia and other knowledge in their medical practice. The objective is to understand the health, technological, and informational needs in Nigeria and determine if the offline Wiki Medicine hotspot would be a useful tool.

The offline medical Wikipedia pilot study will determine the efficacy of medical knowledge in a box. It has the potential to have short-term successes that can provide a pathway to future rollouts of the offline Wiki Medicine hotspot across Africa, Asia, and South America. It will also serve as an additional tool to support efforts to improve healthcare provider performance, as well as a model to make other Wikipedia content accessible to people who are offline.

Why this pilot?

This project supports both of the strategic pillars of the Wikimedia movement. In the pursuit of knowledge equity, we will focus our efforts on the knowledge and communities that have been left out by structures of power and privilege - including those without reliable internet. We will support knowledge as a service by delivering open knowledge through new interfaces in a practical, impactful environment - medical clinics.

Additionally, this pilot supports foundational strength by developing the capacity to work with external partners to deliver impact outside of our immediate scope of work. We will partner with The Africa Center for implementation as well as Internet in a Box and the Wikiproject Medicine group to support this project, expanding our reach well beyond what the Foundation is poised to do on its own.

The pilot will be implemented across Nigeria, a New Readers focus country. Nigeria is the most populated country in Africa with over 170M people, where the official language is English. Roughly half of Nigerians have internet access, so it serves as an ideal context for this pilot.

Project summary

This pilot will deploy wifi hotspots to 20 medical clinics in Nigeria. These hotspots are raspberry pi devices with medical content from a number of free sources, primarily from the Wikimedicine Project.

The project would allow the Wikimedia Foundation to:

• Develop partnerships in Nigeria (government, universities, private organizations)
• Build methods of data collection
• Collect data on:
  • Usage (ease of use/# of times, etc.)
  • Relevance of content (do you need video/audio/content frequently consumed)
  • Suitability of hotspot device in setting/context
  • Methods of use
  • Demographics of use
  • Interactivity and information sharing
  • Methods of rollout

Primary measure of success: healthcare providers regularly access the Wiki Medicine information, supplemented by the qualitative and quantitative data received from surveys and usage data that will provide information on what to be tweaked prior to further implementation.

Project details

See full budget and project details below

• Where: 20 medical clinics in Nigeria, in peri-urban communities
• When: starting October 2018 for 12 months
• What: Raspberry Pi devices preloaded with Wikiproject Medicine and other freely licensed medical content
• Who:
  • Project execution: Wikimedia Foundation & Africa Center (Uzo Iwaela)
  • Supported by: Federal Ministry of Health, CDC Nigeria office, Health Systems Consult Limited (HSCL)
  • Materials by: Kiwix, Internet in a Box, and Wikiproject Medicine

Data collection

• Regular monitoring visits by the in-country Project Manager to solicit user feedback and stories, and other project communications materials from the field.
• Usage data will be collected from all the hotspots will be collected at the completion of the pilot.
• Interview one healthcare provider per state for a longer survey at the end of the pilot, related to their digital needs, their thoughts on the offline Wiki Medicine hotspot, and the project approach and management.
• Completion interviews with representatives of the partner organizations will also be conducted to ascertain successes and area for improvement on working together

Scaling this pilot

After proof of concept and success with the offline Wiki Medicine hotspots, use the successes and findings to apply for further funding from large organizations such as the Bill and Melinda Gates Foundation, the World Bank, or the Omidyar Network. These foundations have a focus funding towards the UN Sustainable Development goals, which align directly with this project.

This will allow for implementation of offline Wiki Medicine hotspots loaded with Wikipedia and other content to be distributed across the continent, providing critical medical information in many areas that don’t have reliable internet access. After the initial pilot, there are also opportunities for additional partnerships with top medical and public health schools across the globe.

Final report - 2020

This is the final report to share the outcomes, impacts, and learnings from the WOMP project. The material below condenses the multiple project reports from the duration of the project into a single summary.

Project plan

20 devices were distributed to 20 health-care sites across the six geo-political zones in Nigeria.

Some of the potential sites required an in-person visit before accepting any form of proposal, which necessitated contacting additional sites.

Three hospitals decided not to participate for the following reasons:

1. Wiki is not a medical authority.
2. Wiki cannot be cited as an academic source.
3. Medical information on Wiki has not been peer-reviewed or gone through a rigorous vetting process.
4. Most of the facilities were not fully convinced that this is absolutely free and thought there may be a hidden agenda e.g. money or some great gain for later.
5. For much bigger facilities, e.g. Federal Hospitals with over 300 medical staff, there was the concern that a single device would not be enough.
6. The cyber security of the device and the specific concern about if it can be reformatted, or hacked into by a hacker etc.

The final selection of sites included 8 public hospitals and 12 private hospitals, including 1 teaching hospital. The sites were chosen based on this criteria:

1. A legally registered clinic, hospital, university teaching hospital or pharmacy in Nigeria that

is either privately owned or state-owned.

1. Electricity and/or a generator that functions at least 50% of the time.
2. A functional computer or laptop that is available to and used by staff.
3. At least 60% of staff who are literate and own a smart phone.
4. Situated in peri-urban areas of the country.
5. A patient load that is average or above average according to Ministry of Health guidelines.

The selected sites were all initially provided with an in-person training session that gave the medical practitioners information about how to use the devices, the information within the devices, and to answer any questions.

The plan included Focus Group discussions and questionnaires at 10 of the sites, but some of the sites visited were not expecting the WOMP team despite prior notification via email and phone calls, so the questionnaires were established as the baseline survey. The practitioners were also more inclined to fill out questionnaires than to participate in focus groups due to their tight schedules.

After the site-selection, the WOMP Nigeria team visited each of the sites three times during the pilot, sent out monthly questionnaires to the practitioners via email, and also conducted bi-weekly follow-up calls with the site point-person. The bi-weekly calls asked the following questions:

• Is the device working well?
• Have medical staff been using the device?
• Are your colleagues satisfied using the device?
• Is the information on the device meeting your medical information needs?
• Is the information message style easy for you all to understand?
• Did you need to check this device while attending to a patient or a medical issue?
• Is there any issue with the device?
• Are there any questions concerning the device?

During a mid-point review, three of the sites were found to not be using their devices, so those devices were redistributed to alternative sites.

At the end of the project, the WOMP Nigeria team visited the sites and conducted 10 focus group discussions (with up to 15 practitioners in one meeting), or individual interviews at sites where there were not enough respondents available to participate in a group discussion due to work pressure or absence. Respondents varied from doctors, doctors (lab/clinical microbiology), auxiliary nurses, staff nurses, laboratory technicians, optometrists, and dentists.

Challenges, feedback, and key learnings

Technical

Over the duration of the project, some of the WOMP Devices or practitioners had a number of technical challenges, including: an inability to connect, devices not being compatible with their type of phones or network browser (e.g. Opera), IIAB (Batch B device) being too slow.

The final reports detailed these grouped comments.

1. The range of the device (20m) is too short and it should be increased.
2. The device memory of 16 GB or 32 GB is too small for the type of information it contains.
3. Device cannot update itself and outdated information is a major challenge for medicine which is a constantly evolving field.
4. The device is too small for the facility and more devices should be deployed within larger facilities.
5. The steps to log-in to the device are somewhat cumbersome.
6. Logging-in is sometimes slow.

Social

Some of the medical practitioners were difficult to contact, or expressed apathy. These issues were partially related to the heavy demands of their jobs, and partially due to the technical challenges from the devices. These challenges were responded to by halting the efforts to get device usage reports via phone or email, and instead gathering this information at the mid-term, in-person meetings with sites. The team also sent out monthly questionnaires via email, and encouraged usage through weekly phone text messages sent to 55 of the health workers.

Content

The survey questionnaires highlighted some feedback from the medical practitioners in these themes:

• It needs a wider coverage. It should contain more information outside of the medical information. It is limited.
• There should be more on medicine and drug dosage as seen on Medscape.
• Journals should be available on the device.
• A translation dictionary should be included on the devise so that words can be translated into different languages.
• The content on it is very elementary. There needs to be more depth and coverage of topics.

The final report provided these grouped comments:

1. This present device is currently too basic for advanced doctors, rather it is more suitable for clinicians who recently graduated medical school. Nurses also found the device very helpful. They would take screen shots and study later at home.
2. The information on the device uses a ‘one size fits all’ approach, but that limits engagement from more advanced health workers.
3. The device lacks higher level and specialist specific content (e.g. radiology, neurology, in-depth content on laboratory technology science, forensic medicine/pathology, patho-physiology, optometry-glaucoma, acute periodontitis, clinical microbiology, etc.)
4. Device needs content from authoritative bodies in the field of medicine.
5. Medical journals should be available on the device.
6. Range of content needs to be wider as content is limited.
7. There should be more content on medicine and drug dosage as seen on Medscape.

An additional point highlighted the impact of the device on two live medical situations:

• 4 respondents - nurses - said that the device was useful and within reach when they had a medical situation that they were not familiar with. One of these situations was a patient with an ectopic pregnancy, and another situation was a patient with vitiligo.

Recommendations

The reports gave these points as items to consider:

Future projects of a similar nature

1. How to accommodate rural areas where there is no electricity.
2. There needs to be more awareness about the device in the respective health and publicity is key in terms of health care workers using the device.
3. Some respondents felt that the acceptability of the device by health workers in general could be an issue to look out for as well.
4. It may be harmful if the device becomes public and easily accessible to everyone because the content could be misused by quack doctors and unqualified medical practitioners. Perhaps because the device is from Wikipedia, it could be an issue.
5. In the absence of computers, some health workers may not have smart phones and this limits who is able to use the device.
6. Some health practitioners would still not engage with device. This could be due to poor reading and research habits by health workers, even if the device is available and given free.
7. More effective monitoring by the organization in charge of the device to encourage health workers to use it.

Technology

1. The storage memory of the device should be 64gb or 128gb for the size of information it should contain.
2. With an increase in storage memory, the Processor should be increased as well, so the device does not become slow.
3. Provide a mechanism for the device to update itself- especially when a knowledge partner updates their information, it can update itself to accommodate that.
4. Suggested range of device should be at least 200m.
5. The device should have built in power or batteries for settings that do not have electricity.
6. In device placement and location, the device should not be placed in places where there are specialized machinery because that can interfere with the signal.
7. Battery life should be similar to that of a bank’s Token device.
8. Device should have a security system in place if it does not already.
9. Could download content from device on an App.

Content

1. Device should be made audience specific- for specialists level, for medical school and freshmen doctors, for laymen, i.e. one for each level.
2. Specialized medicine is non negotiable. More treatments (like drug dosage) should be included.
3. Content should be tailored more towards ‘academic’ content, i.e. in the form of journals.
4. Partnering with some major platforms that are medical information authorities whose content can be cited.
5. The information included in the device needs to be broader. It should contain more information outside medical information.
6. A dictionary should be placed on the device to help translate English words to the other languages, so they can use the languages as well.

Additional sources as Knowledge partners for the device:

1. WebMD
2. MEDScape
3. PUBMED
4. Mayo Clinic
5. British journal of Medicine
6. Journal of Hematology
7. NCPI database (USA)
8. AJOM – African Journal of Medicine
9. Hinari
10. The Nature
11. Medical microbiology
12. Materials from WHO Centre for Disease Control
13. National Centre for Biotechnology Information (NCBI)
14. Slide share App
15. EMDEX, DMX
16. Encyclopedia of Nursing and Allied Health
17. UpToDate
18. NICE Guidelines

Key questions during the final meetings included:

• Who are the target audiences?
• Who does WOMP want to serve?
• What does each specific audience need?

The practitioners specifically suggested:

1. Other medical authorities have something they are known for and WOMP should have something unique to offer. For example - Medscape is good with drug dosage and intervention. What does WOMP want to be known for?
2. Information on the device was not organized. During the meeting, a reference was made to Medscape and the level of organization of their content. WOMP could adopt that method.
3. Another important point was that content should focus more on the context of Nigeria, e.g. tropical medicine and its peculiar cases. It would be very helpful if the WOMP would have “content creators” instead of, or in addition to, knowledge content partners. These content creators could be experienced, long serving medical practitioners that have trained doctors and who have garnered a wealth of knowledge and experience over the course of their career. The intention would be for them to share their findings, treatments, and resource materials that they have put together from working on peculiar cases, on the WOMP platform/device, so others can have access to these resources. These contents could be from acknowledged contributors, and this exchange need not be monetary. E.g. A book on pediatrics called ‘77’ used this method. (NOTE: WOMP team was not able to find additional information on the book)

Future Distribution

1. The device would be useful for universities, academics, libraries - public or school libraries, and medical student hostels
2. Consider more rural areas who do not have access to the internet – but availability of power in such places is a major issue, so a back-up battery/power plan must be in place.
3. Useful for local maternity homes owned by nurses, because it would help them in their practice.
4. Partner with governments for efficiency and more effectiveness.
5. Device could be used at libraries through fiber so that more people have access.

The practitioners also suggested that with the challenge of electrical power supply and limited smart phones available in rural settings, the devices could be placed in Primary Healthcare Centers (PHCs). PHCs use grass roots and community-based medical approaches to provide healthcare to communities. The health workers in these centers could potentially find the device very useful, as they may not be as advanced as their colleagues in other, higher level, health facilities.

Motivation

Towards the conclusion of the project, the WOMP Nigeria team suggested gifting an "appreciation package" to the health facilities and practitioners that participated.

The team locally produced a range of Wikimedia Merchandise, including t-shirts, water bottles, mugs, notepads, pens, key-rings, and carrier bags. Certificates were also presented to a representative of the management of each Hospital.

The Wiki merchandise had an undeniable effect, as the recipients were greatly encouraged and motivated to improve participation with the device, but unfortunately the WOMP IIAB devices had to be retrieved in this stage as the pilot was over. Most Medical Directors were surprised by the package and certificate. They stated that they did not know the project was this important.

The team recommends that motivations like this should be done from project commencement, in order to get the desired results.

Usage analytics

Overview

In aggregate across all the devices the following statistics stood out:

• 8,000+ Search Term Completions
• 500+ Searches
• 2,500+ Total Non-search Page Views
• 256 Videos Viewed
• 12,000 server requests, and 4.6 gigabytes of data was transmitted to the users

Of the 30 most viewed pages listed below, none are specifically aimed at males, whereas 7 or 8 pertain principally to females.

Breakdown by Source

The Medical Wikipedia completely dominates the individual, non-search, page-views over the WikiEM web site and the Practical Action collection of materials, accounting for 95% of the server requests.

Use of videos

Altogether there were 256 instances where videos from the Medical Wikipedia were viewed. These included titles such as:

• Obsessive Compulsive Disorder
• Pregnancy
• Malaria

Other titles are included in the list below, as several of the videos were among the top 30 pages viewed.

Localized content

The Wikipedia in Hausa, Igbo, and Yoruba were included in content supplied with the Internet in a Box servers for this project. They were not much used, however, with fewer than 10 total accesses across all locations over the life of the project and most of those to the Wikipedia home page. All other content was in English.

Key analytic learnings

• It would be helpful to have keywords or categories by which to summarize usage. For example the underlying pages viewed could be searched during analysis for the occurrence of these terms.
• The tools to aggregate content access need to provide summaries by source, type, etc. as well as summaries over all devices.
• Search Term Autocompletion dominates the server response, constituting perhaps 70% or more of the total, so this needs to be isolated.
• We need to provide a means for tracking which SD cards came from which locations. SD cards have a unique serial number and we need to provide a user interface for assigning a name.
• The Unicode strings stored the awstats files need decoding. This is especially the case for content not in English.
• We should incorporate the batch analysis tools into the Admin Console so that the user could simply download a spreadsheet of statistics directly from the server.

Top 30 Pages Viewed

• MedWiki-Book:Women's_health.html
• MedWiki-Book:Infectious_disease.html
• MedWiki-Book:General_surgery.html
• MedWiki-Appendicitis.webm.160p.webm
• MedWiki-Menstrual_cycle.webm.160p.webm
• MedWiki-Malaria.webm.160p.webm
• MedWiki-Preeclampsia_&_eclampsia.webm.160p.webm
• MedWiki-Pregnancy.webm.160p.webm
• MedWiki-Malaria.html
• MedWiki-Pregnancy.html
• MedWiki-Book:Medications.html
• MedWiki-Book:Ophthalmology.html
• MedWiki-Removal_Implanon.webm.160p.webm
• MedWiki-Syphilis.webm.160p.webm
• MedWiki-Appendicitis.html
• MedWiki-Book:Psychiatry.html
• MedWiki-Erythropoietin.html
• WikiEM-Upper_gastrointestinal_bleeding.html
• MedWiki-Book:Cancer.html
• MedWiki-Bowel_obstruction.html
• MedWiki-Book:Children's_health.html
• MedWiki-Candidiasis.html
• MedWiki-Endoscopy.html
• MedWiki-Family_planning.html
• MedWiki-Indigestion.html
• MedWiki-Malaria.html
• MedWiki-Tonsil.html
• MedWiki-Celiac_disease.webm.160p.webm
• MedWiki-Contraception_How_to_Prevent_Unwanted_Pregnancy.webm.160p.webm
• MedWiki-Eczema.webm.160p.webm