Pilot Survey

Guidelines on why, when, and how to pilot your survey instrument. Covers the iterative process of piloting, including different phases, sample selection, timeline planning, and using pilot results to refine survey content, improve flow, and test protocols for high-quality data collection.

Key Takeaways
  • Piloting is essential for refining survey content, improving flow, and testing protocols to ensure high-quality data collection.
  • It is an iterative process that saves time and resources by identifying and addressing issues before the main survey launch.

Why Pilot?

As one of IPA’s Required Research Protocols – or MMDs – piloting is an essential step for which there are no shortcuts. Teams can use piloting for different purposes and it has numerous benefits:

  1. Designing a New Survey or Important Module:
    • Acquire knowledge of unknown phenomena/concepts and answer questions about the overall survey design.
  2. Refining the Content of the Survey and Improving Survey Flow:
    • Adjust question wording and order.
    • Identify non-essential questions to drop from the final version.
    • Identify common responses for pre-encoding.
    • Identify translation mistakes for correction.
    • Pay attention to how experienced surveyors ask questions and incorporate their “conversation” into the question text for a smooth flow.
    • Verify that respondents maintain interest.
    • Verify that interviewers and respondents feel comfortable with the interview flow.
    • Identify potential hitches for the interviewer, such as needing to repeat questions, correct misinterpretations, or record volunteered information.
  3. Testing Your Data Flow:
    • Verify your survey programming works well using High-Frequency Checks.
    • Verify your preloading from a baseline survey works well.
    • Collect metadata.
    • Check the completeness of your pilot dataset.
    • Test your back-check strategy and dataset using the Back Check Manual.
  4. Testing Your Field Protocols:
    • Verify the planned timing of interviews fits respondents.
    • Check the existing infrastructure, such as access to electricity, internet, cell phone networks, and other resources.
    • Ensure your sampling, replacement protocols, and tracking strategy work well in the field.
    • Verify your existing sampling frame is up to date or identify the time needed for making a listing.
    • Identify the duration and accuracy of capturing GPS locations with your devices.

Pilot Sample

When selecting your piloting sample, ensure that:

  • The piloting area is outside your sample area. Pilot the survey with respondents who are outside your sample but in similar neighborhoods/backgrounds, such as people from the same village but in different households or people from neighboring villages.
  • Every question in every section gets piloted several times. Pilot with at least 30 households if the survey contains many new questions or you administer it in a new context, or 15 households if you used it before in a similar context.

Pilot Phases

Piloting is an iterative process, meaning teams should pilot surveys each time they undergo significant changes. The complete piloting process consists of three phases:

  1. Pre-pilot Phase:
    • Conducted when designing a new survey or important sections.
    • Start with focus groups and qualitative interviews.
    • This phase should happen early in the piloting process—a few months before data collection—and helps with survey design and demystifying important phenomena.
  2. 1st Pilot Phase:
    • Conducted before translating the survey.
    • Pilot the survey instrument yourself with a few experienced surveyors who speak the local language or dialect.
    • This phase improves the content of the survey.
  3. 2nd Pilot Phase:
    • Conducted after finalizing, programming, and translating the survey.
    • Pilot the final translated instrument with a larger group of surveyors.
    • This phase should happen before enumerator training and survey launch and checks the survey flow, programming, data flow, and data quality checks.

At a minimum, every survey must go through the second phase of piloting before its launch, but not all surveys have to go through all three phases. This table shows simple rules for piloting:

Piloting Steps by Survey Type
Survey Type Pre-pilot 1st Pilot 2nd Pilot
Brand new survey / important module + + +
Existing well-designed survey in a new location/language + +
Existing well-designed survey in the same location/language, such as endline with small changes +

Pilot Timeline

The amount of time needed to pilot the survey depends on how long and how new the survey is. The newer the survey, the more piloting it requires.

  • If designing your survey from scratch, start pre-piloting five to six months before the survey launch.
  • If adapting an existing survey to a new context/language, start the first pilot three to four months before the launch.
  • If making small changes to an existing survey for the same context/language, start the second phase four to five weeks before the launch.

Pilot Length

Piloting is an iterative process, and there is no best practice regarding the required number of iterations. Piloting is over when:

  • The survey flows well.
  • No significant changes remain.
  • Respondents can understand all the questions.

However, consider that some questions, such as hypotheticals, will always require respondents to struggle, even if the question is clearly understood.

Pilot Results

To use pilot results:

  • Take detailed notes during piloting.
  • Use these notes to make changes to your questionnaire, translation, or programming.
  • Discuss important points during enumerator training and include them in the survey manual.

On the Content of the Survey

  • Reword or re-translate questions that are mistakenly interpreted by respondents.
  • Drop or rephrase questions if responses show no variation.
  • Change question order if respondents can guess the general hypothesis, as this may indicate biased or leading questions.

On the Flow of the Survey

  • Shorten the survey by dropping non-essential questions if respondents or interviewers show signs of fatigue.
  • Move interesting modules, such as games, to the middle of the survey to maintain engagement.

On the Data Flow and Field Protocols

  • Correct programming mistakes, such as adjusting relevance fields or constraints.
  • Use pilot data to refine survey constraints based on reasonable responses.
  • Use pilot metadata, such as duration, to adjust survey and field plans.
  • Adjust interview timing or provide additional equipment if teams identify infrastructure issues such as lack of electricity.

Pilot vs. False Launch

There are two main differences between a survey pilot and a false launch:

  1. Timing: The pilot starts earlier – often during survey design – finalizes the survey’s content, translation, and programming, and checks data flow and field protocols.
  2. Awareness: Your team knows about the pilot. In contrast, teams implement a false launch with a final translated instrument, and the field team believes it is the first day of real data collection.

While it is possible to organize the second phase of the pilot as a false launch, this approach is not ideal. Piloting usually requires multiple iterations with debriefing sessions and revisions.

Pilot vs. Soft Launch

There are two main differences between a survey pilot and a soft launch:

  1. Timing: The pilot starts earlier – often during survey design – and finalizes the survey’s content, translation, and programming, and checks data flow and field protocols.
  2. Location: Teams implement the pilot outside your intended sample area. In contrast, teams always implement a soft launch with a final translated instrument and in a small area within the intended sample area, that is, part of the real data collection.

Piloting and Time Constraints

Remember, teams do not have time to skip the pilot! Piloting requires effort, but it is an upfront investment that will save time and money during data collection, cleaning, and analysis. Mistakes cost least when caught during the pilot. Moreover, piloted surveys are much more likely to result in high-quality data than non-piloted surveys. Always pilot your survey!

Pilot IRB Approval

Whether your pilot requires IRB approval depends on whether the activity constitutes human subjects research. Research is a systematic evaluation designed to produce generalizable knowledge.

  • If the pilot refines a tool – such as evaluating text delivery, testing question length, finding programming errors – 4and teams use the results to improve data quality rather than as part of outcome measures, it does not require IRB approval.
  • If the pilot collects identifiable data that teams will use for the study, it requires IRB approval.

If you are unsure whether your pilot activity needs IRB approval, email humansubjects@poverty-action.org.

If you are unsure whether your pilot activity needs IRB approval, email humansubjects@poverty-action.org.

Back to top