User:VHargyono (WMF)/Test

• Home
• About
• Resources
• Community
• Newsletter
• Community Support
• Reading Wikipedia in the Classroom

Commonly Used Data Collection Methods

This section will present eight of the commonly used data collection methods that people usually use in evaluation. Each section contains basic information of each method; its strengths and limitations; overview on how to do it; and further resources.

Document Review

▼

What is it? And how do you do it?

Desk based literature review. The documents can include internal documents within a program or organization, interview notes/summary, academic literature, journal articles, transcripts, meeting notes, policy papers, grey literature, newsletters, personal documents, etc.

How?

• Create a scope, timescope, geographic, languages etc.
• Organized around central themes/questions
• Methodologies, scoping, systematic, exhaustive

Caption

Strengths	Limitations
Good practices/experience from other or similar projects In*depth information of the project Inexpensive/cost efficient Unobtrusive Bring up issues that are not noticed by other means Background information Help refine research questions Easy to access documents	Time consuming Labour intensive May be biased/navigating bias Information may be incorrect or incomplete/gaps in research/research bound by what already exists Information may be irrelevant, disorganised, inapplicable

Review/Analysis of Statistical Data

▼

What is it? And how do you do it?

Review/analysis of statistical data is the process of examining quantitative data using statistical methods for us to describe, assess, compile, and compare the data that you have collected. Quantitative data are information in numeric form. There are two commonly used methods on doing this:

1. Quantitative (secondary data), statistical records (for example: number of students, schools records)
2. Secondary analysis based on official records/databases

How?

• Assess baseline data
• Depends on the context/organization culture whether we can use the existing data

Caption

Strengths	Limitations
Present the broader community/range of information Save money, time, and human resources Can be used as baseline data Can be used as multipurpose Legitimate data Quick	Generalization of data Lack of detailed analysis Doesn’t capture behavioral and social patterns Cannot check the validity Misinterpreted Need to be updated Issue of relevance Bias of selection Data availability Incompatible data from the firsthand studies Implementation issues

Participatory Learning and Action

▼

What is it? And how do you do it?

It originally aims to include rural people's knowledge and opinions into the project's planning and management. This method is using a qualitative approach which i used to get a more thorough understanding of situation in the community or about the community itself. It is usually used in collaborative project that involves community. The main goal of this method is to help assist the community to analyze their own situation in a project, create learnings, and to make sure we can transform the learning into action in the future.

How?

• Participative and community-led. Project implementers are facilitators
• Project' goals and objectives are formulated based on community's needs and agreed by the community
• Tools used, approaches, and data analysis are participatory
• Common tools and techniques used include: observation, semi-structured interviews, focus group discussions, photography, videos, case studies, charts and diagrams, maps

Caption

Strengths

Limitations

Relevant, given that it comes from real priorities of people in the community Participatory, which create sense of ownership among participants Transparent, which comes from openness in the process of the project activity Sustainability, due to community's involvement in the planning and activities execution Use of local resources Community-led Empowering, where community can identify and solve their own problem Capacity building

Time consuming to reach consensus Power relation within the community can be a challenge Capacity of facilitators Language/cultural barrier Costly and requires a lot of resources

Pictures or Images

▼

What is it? And how do you do it?

It's an activity of filming or photographing certain activities or conditions. It can also include other visual or arts methods, such as painting or sculpture. The purpose is to present information in a more engaging way and increase understanding of a project or program implementation' results. This method can also help in visualizing different kinds of information, in many cases it helps to reveal hidden meaning that may not be possible through a survey approach.

How?

• Storyboard
• Community mapping
• Photography which represents their experience on the issue (for example: gender inequality)
• Photo Voice (also a participatory method)

Caption

Strengths	Limitations
Can accommodate specific groups (children, young students, illiterate participants) and it gives voice to the community Better at presenting sensitive issues Offer different perspective through pictures Able to reach target audience Good for representing	Costly equipment: camera, printer, etc Can be time consuming Highly subjective Wrong interpretations Technical limitations Possible manipulation Cannot be used alone, needs explanation to explain the images May not be permitted to take pictures or film (cultural sites, children, etc)

Focus Group Discussions

▼

What is it? And how do you do it?

A focus group is one of the qualitative approaches that can help us obtain a more in-depth understanding of problems that our program is trying to solve. Since it is a qualitative approach, the data collected is descriptive. It is done by conducting a group interview with people who are impacted, have shared experiences, interests, and/or characteristics, and the discussion will focus on a particular topic. Usually it is conducted in a small discussion group from 4 up to 10 people participating. This group is purposefully selected based on certain criterias. The discussion is led by a facilitator, whose role is very important. The facilitator must be able to build a safe environment that will encourage participants to express their responses, perceptions, and point of view genuinely. The facilitator uses a set of questions or topics to guide the conversation. The questions and topics are predetermined based on the key evaluation questions.

How?

• Develop an interview guide with 7-10 questions based on your key evaluation questions

Prepare a consent form to be signed by each participant

• When introducing the interview to participants, make sure to provide clear information about the objectives of the Focus Group Discussion, reason why the participant was selected, and don’t forget to introduce the interviewer and organization as well
• Can be conducted face-to-face with participants or via online video conferencing. Be mindful when using teleconferencing, facilitator must make sure that everyone participates in the discussion
• The discussion is usually conducted for 60 to 90 minutes using a semi-structured method. Facilitator can help stimulate the discussion by writing down participants’ answers on a whiteboard or virtual board, or on sticky notes
• Start comparing results with other interviews conducted to other group of participants

Caption

Strengths

Limitations

Quick and easy to set up Imitate natural social interactions Cost effective and time efficient to conduct Can obtain more information that was not captured by individual interview due to the group dynamic Can be designed to be more inclusive and culturally sensitive, as it is participatory and more careful consideration is given to local context. This can give space for marginalized/minority group to voice their perspectives/opinions Investigate phenomenon past statistical data due to open ended method, which gives more opportunity to explore the phenomenon more in-depth Can create positive interactions between participants and members of the project

The dynamic of the discussion is dependent on facilitator's skills Prone to facilitator's bias Biased selection of participants Tendency in a group to maintain harmony or compliance, or known as "groupthink". Peer pressure may also affect how the discussion is going and some participants might dominate the discussion Does not capture information at individual level, but also not representative of larger population Facilitator must make sure to provide a safe space to maintain genuine participation

Observation

▼

What is it? And how do you do it?

• Systematic data collection approach to examine the natural behaviour of individuals
  

How?

• Can be direct observation: Observing project activities in action without interaction, or Participant observation: the observer also joins in activities and interacts with others
• Qualitative and quantitative

Caption

Strengths	Limitations
It does not rely on people’s willingness or ability to provide information (aside from consent to be observed) Can gain both qualitative and quantitative data Acquire accurate information from observing what they are doing rather than what they say they did Easy Realistic/evidence based Cheap Natural setting Informal and doesn’t require extensive training	Time consuming Subjective: depends on the researcher Ethical principles can be involved: have to get consent of participants Observations only tell one part of the story Participants might be overly aware of the observer and change their behaviour when observed Needs to be frequent and consistent, usually over a period of time, to get an accurate picture, rather than once*off

Semi-Structured Interview

▼

What is it? And how do you do it?

How?

• Based on an ‘interview guide’ but able to follow topical trajectories in the discussion that may stray from the guide, where appropriate
• Includes open-ended questions: best to tape record and transcript tapes for analysis (as discussions may stray from the guide and go deeper)
• Interviewer and respondents engage in a formal interview
• Able to bring up new ideas from the conversations

Caption

Strengths

Limitations

Flexibility to explore/explain/change the questions depending on the specific interview, and allows interviewer to pursue new topics and cover more topics Freedom for the interviewers to express their view in their own terms More interactions between the interviewer and interviewees Can provide in*depth quality data Broader perspective of results/impact

Time consuming It requires specific/strong skills to interview/engage interviewees Difficult to generalise Interviewee bias: they may need to say/act a certain way in order to fit with the researcher’s perception Interviewees may feel exposed by questions on sensitive topics Strain on resources when there are a lot of respondents More training for interviewers For data analysis: hard to reconcile different perspectives

Structured Interviews/Surveys

▼

What is it? And how do you do it?

How?

• Structured interviews
• Quantitative
• Measurable
• Capacity building

Caption

Strengths

Limitations

Easy to administer Easy to analyse Able to gather information from larger group/range Allows statistical analysis More reliable data to be provide to donors (potentially less bias) Consistent Time/cost efficient Real time data Well planned and structured

Not really useful for small projects/samples More rigid: tends to be limiting (e.g. individual*level, organisational level, not accommodating the unexpected results) Limitations of questions Not allowing room for why or questions and only answering the what. Exclusion of other factors Need interpreters Face to face bias May not give the correct answers Participants can be bored if there are more/lots of questions Privacy and sensitivity Lack of trust Language barriers

Data Collection Method Recommendation for Evaluating Wikimedia Programs

Many Wikimedia project activities, particularly related to education, involve young students. However, not only students can participate in the project itself, we can also engage them into evaluating the program. Eventually, the students are the ones who are supposed to benefit from the program, it will be interesting to see more of their perspectives in all aspects of the project. First, let’s understand what youth engagement means. According to ACT For Youth, youth engagement is the outcome of when we involve young people in creating positive social change. Youth can be involved from the planning stage up, decision making, and until evaluation stage. So not only having youth as participants, but also having them involved in the beginning/planning stage and the implementation phases, including evaluation, as well.

Method Recommendation 1: Photovoice

▼

What is it?

A photograph can be a powerful tool to tell a story and to advocate on the issues or represent changes that were made. Just like text document or video footage, a photograph can hold a large amount of information but quicker. This way, a photograph does not depend on the audience to spend their time trying to digest the information by reading and watching the document and videos.

Photovoice can be a powerful tool to enable participants, particularly students, to express their perspectives and points of view as well as representing their communities. It is done by youth taking photographs of scenes that are related to topics or themes that you are trying to evaluate. For example: students can take photos of their activities, record interviews among their peers about how they think the activities, or record themselves with their opinion about the activities conducted in the Wikicamp. After capturing the photographs, groups of youth can discuss and/or interpret what the photos highlight a particular theme.

Tips: use digital photos for more instant results. Participants and project staff can discuss about the results right as soon as the photos are taken.

Who is involved

Individual student or divide individuals into groups, program staffs

What do we need

• Digital cameras and/or cell phones with cameras
• Voice recorders
• Whiteboard/papers and markers

Application example

One of your evaluation objectives is to understand what the activities and experience means to students as participants. To measure this, you can ask students to capture photos of the activities within the program, for example, during 2 days of camping. When you wrap up the camping activities, you can divide students into small groups or a large group (depending on the number of students participating). Ask students to discuss the best photos they captured. You can prepare some topics or question prompts to encourage students to voice their stories/opinions. The prompts can cover some basic questions, such as why they took a particular photo, what does the photo mean to them, what aspects of the photo that they like/dislike, etc. Or you can also frame the question prompts based on your predetermined key evaluation questions. The discussions can be audio recorded. Then you can use the photos and recording as data for your program evaluation. Then later, you can use the photos to highlight the program’s experiences, which can be displayed online by uploading and creating albums of the photographs in Wikicommons. You can also ask students to write down their perception of the photos in the albums and include them in Wikicommons.

Method Recommendation 2: Story Board Activity

▼

What is it?

Storyboard activity is a very participative evaluation method/tool so people involved in the project can be engaged in telling their stories. This tool helps people understand that evaluation is something that everyone can do. Making the evaluation process fun and engaging to stakeholders, will make it easier for us to capture rich and meaningful data. It also helps co-creation of evaluation frameworks that can represent our organization/project’s work and impacts. The Storyboard activities particularly will be most enjoyable to engage between youth and adults, such as in a Wikicamp project. Project participants will be asked to tell their experiences in the form of stories.

How do you do it

Tools Needed

Participants can use any tools, materials, and images they like to create their own storyboards. You can provide, crayons, papers, printed, cardboards, stickers, glitters, pictures, photos, anything that will help participants to tell their stories!

Indicators Recommendation

Caption

Outcomes	Indicators
Participant's Increased self-esteem / Increased Participation	Participants take initiatives working on x project Participants always have activities to do Participants take ownership to tasks given Participants are making key decisions independently Participants actively share ideas about things that matter to them Participants help their peers in learning new skills Participants feel excited to be involved in the project
Participants's Sense of Belonging in the Community	Participants's willingness to have a leadership roles or responsibilities among their peers or in the community Participants are able to access information that is related to decision making Participants shows that they are interested in helping their peers and/or others on their community Participants has a desire to mentor their peer(s) Participants are actively involved in discussions and activities in their community Participants feel proud of their community Participants, particularly students, do not hesitant to ask for help or advice from adults in the community Participants feel safe to and able to give ideas on how to improve their community Participants also joins or involved in other community-based activities/projects Participants express a sense of belonging toward their community Participants display interest and/or concern about community change
Participants Retention	Participants recommend the program/activities to their peers Participants lead most of the activities/projects Project's staffs and /or volunteers have the skills to be mentors Participants, particularly students, feel comfortable being around adults in a work setting (vice versa) Participants are able to make reflections and decisions based on their own experiences Some participants participate and involved in the project for at least a year When involving students, when older youth "graduated" the program, there are younger peers that replaced them Participants also use their experience in the program as a socializing opportunities A system of recognition for participants' accomplishments during the activities is established Participants feel passionate about the issues addressed through this project Participants feel challenged and motivated to improve and do their best in the activities
Increased Inclusion and Accessibility