Skip to content
All articles

Statistics

Questionnaire Design and Validation: Likert Scales, Cronbach's Alpha and Pilot Testing

A questionnaire is only as good as its questions. Good design starts by listing exactly what you need to measure, then writing short, neutral, single-idea questions in the participant's own language, in a logical order.

Resora Research Team · 15 Nov 2026 · 6 min read

A questionnaire is only as good as its questions. Good design starts by listing exactly what you need to measure, then writing short, neutral, single-idea questions in the participant's own language, in a logical order. Likert scales (for example strongly disagree to strongly agree) measure attitudes; give them 5 points with clear labels. Before use, a questionnaire must be shown to be valid (does it measure what it claims?) and reliable (does it give consistent results?). Check validity through expert review (content validity index) and factor analysis, and reliability with Cronbach's alpha (0.70 or above is usually acceptable) or test–retest. Always pilot-test on a small sample, revise, and report how you developed and validated the tool. A borrowed questionnaire needs re-checking in your setting and language.

Should you use an existing questionnaire?

First, search for a validated instrument (for example WHO-5 for wellbeing, PHQ-9 for depression, GPAQ for physical activity, Kuppuswamy or the Modified BG Prasad scale for socio-economic status in India). Using one saves time and lets you compare your results with others. Obtain permission where required, cite the source, and check whether it was validated in a population and language like yours. Build a new one only when nothing suitable exists.

Step 1: Plan before you write

  1. List your objectives and variables. For each, write what data you need and how it will be analysed.
  2. Include only what you will analyse. Every extra question reduces completion and quality.
  3. Choose the mode: interviewer-administered (better for low literacy), self-administered (cheaper, more private), telephone, or online.
  4. Decide the sections: introduction and consent, demographics, main topic sections, closing.

Step 2: Write good questions

RulePoorBetter
One idea per question (no double-barrelled)Do you smoke or drink?Do you smoke? (separate) Do you drink alcohol?
Be specificDo you exercise regularly?In the past 7 days, on how many days did you walk for at least 30 minutes?
Use simple wordsDo you consume tobacco products?Do you chew or smoke tobacco?
Avoid leading wordsDon't you agree that vaccines are safe?How safe do you think vaccines are?
Avoid negativesDo you not disagree…?Do you agree…?
Give a clear time frameHow often do you visit a doctor?How many times did you visit a doctor in the last 6 months?
Cover all optionsAge: 10–20, 20–30Age: 10–19, 20–29 (categories must not overlap)
Offer "don't know" or "prefer not to say" where appropriateForced answer on a sensitive topicAdd the option

Sensitive questions (income, sexual behaviour, substance use) should come later, in a neutral tone, with a reassurance of confidentiality, and sometimes with ranges rather than exact values.

Step 3: Choose the response format

  • Open-ended: free text. Rich but hard to analyse; use sparingly and for exploration.
  • Closed-ended: fixed choices, easy to analyse. Make options mutually exclusive and exhaustive (add "Other (specify)").
  • Dichotomous: yes/no.
  • Multiple choice: single or multi-select.
  • Rating scales: Likert, numeric (0–10 pain score) or visual analogue scale.

Likert scales

A Likert scale asks how far a person agrees with a statement:

12345
Strongly disagreeDisagreeNeutralAgreeStrongly agree
  • Use 5 or 7 points, with every point labelled and balanced on both sides.
  • Include a neutral midpoint unless you want to force a direction.
  • Mix positively and negatively worded items to reduce "yes-saying" (acquiescence bias), then reverse-score the negative items before totalling.
  • A Likert item is ordinal. A summed scale is often analysed as continuous if it has several items, but report medians and use non-parametric tests for single items.

Step 4: Layout, order and language

  • Start with easy, interesting questions; leave demographics and sensitive items for later (or the start if the questions are neutral).
  • Group by topic and move from general to specific.
  • Number the questions, add clear skip instructions ("If No, go to Q8"), and keep it short, ideally completed in 10–15 minutes.
  • Translate properly. Use forward–backward translation: one translator converts to the local language, another translates back blind, and a panel resolves differences. Direct word-for-word translation can change the meaning.
  • Add a participant information sheet and consent at the start.

Validity: does it measure what it claims to?

TypeQuestionHow to check
Face validityDoes it look appropriate?Informal review by experts and a few respondents
Content validityDo the items cover the whole topic?Expert panel rates each item; compute the content validity index (CVI)
Construct validityDoes it behave as the theory predicts?Factor analysis; compare known groups; correlate with related measures
Criterion validityDoes it agree with a gold standard?Correlate with an established tool; sensitivity and specificity against a diagnostic standard

Content validity index. Ask 5–10 experts to rate each item's relevance on a 4-point scale (1 = not relevant to 4 = highly relevant). The item-CVI (I-CVI) is the proportion of experts scoring 3 or 4; 0.78 or higher is usually accepted when there are six or more experts. The scale-level CVI (S-CVI) is the average, with 0.90 or higher considered excellent.

Reliability: does it give consistent results?

TypeWhat it checksStatistic
Internal consistencyDo the items of a scale measure the same thing?Cronbach's alpha
Test–retestAre results stable over time (usually 1–2 weeks)?Intraclass correlation (ICC) for continuous scores; kappa for categories
Inter-raterDo two interviewers get the same answer?Cohen's kappa or ICC

Cronbach's alpha

\alpha = \frac{k}{k-1}\left(1 - \frac{\sum \sigma_i^2}{\sigma_T^2}\right)

where k is the number of items, σᵢ² the variance of each item, and σₜ² the variance of the total score. It rises when items correlate with each other, and when there are more items.

AlphaInterpretation
Below 0.50Unacceptable
0.50–0.69Poor to questionable
0.70–0.79Acceptable
0.80–0.89Good
0.90 or aboveExcellent (above 0.95 can mean redundant items)

Cautions. Alpha applies to one scale measuring one concept, not to a whole questionnaire with different sections, so calculate it separately for each scale (knowledge, attitude, practice). It does not prove validity. A low alpha can come from a few bad items: check "alpha if item deleted" and each item's correlation with the total.

Knowledge tests with right/wrong items use KR-20, the equivalent for dichotomous items.

Pilot testing

A pilot is a dress rehearsal on about 10–30 people similar to your sample, who are not included in the final study.

Look for:

  • Questions that people misunderstand, skip or answer in unexpected ways
  • Time taken and fatigue
  • Awkward wording, missing options, and unclear skip patterns
  • Interviewer problems
  • Data entry issues

Use cognitive interviewing with a few respondents: ask them to explain, in their own words, what each question means. Revise, then test again if the changes are large. Report the pilot in your methods.

Sample size for validation studies

A common rule of thumb for factor analysis is 5–10 respondents per item, with a minimum of about 100–200. For a 20-item scale, aim for roughly 200 participants. Use the same sample to calculate Cronbach's alpha.

Common mistakes

  1. Writing questions before listing the objectives.
  2. Double-barrelled, leading or vague questions.
  3. Using a tool in another country or language without checking validity.
  4. No pilot test.
  5. Reporting Cronbach's alpha for the whole questionnaire, rather than for each scale.
  6. Treating a high alpha as proof of validity.
  7. Overlapping or missing answer options.
  8. Not reverse-scoring negatively worded items, which can cause a falsely low alpha and wrong scores.
  9. Too long a questionnaire, causing tired or incomplete answers.
  10. Analysing a Likert item as if it were an exact measurement.

How to write it in your methods

A semi-structured questionnaire was developed after a literature review. Content validity was assessed by seven experts (S-CVI = 0.92). It was translated into Hindi by forward–backward translation and pilot-tested on 20 participants, who were excluded from the final study. Internal consistency of the knowledge scale was acceptable (KR-20 = 0.78) and of the attitude scale was good (Cronbach's alpha = 0.84).

Keywords

questionnaire design, questionnaire validation, Likert scale, Cronbach's alpha, how to design a questionnaire, validity and reliability, content validity index, pilot testing questionnaire, KAP questionnaire, survey questions, forward backward translation, reverse scoring, internal consistency, questionnaire for thesis, survey design

References

  • Boynton PM, Greenhalgh T. Selecting, designing, and developing your questionnaire. BMJ. 2004;328:1312–1315.
  • Tavakol M, Dennick R. Making sense of Cronbach's alpha. International Journal of Medical Education. 2011;2:53–55.
  • Polit DF, Beck CT. The content validity index: are you sure you know what's being reported? Research in Nursing & Health. 2006;29:489–497.
  • Likert R. A technique for the measurement of attitudes. Archives of Psychology. 1932;140:1–55.
  • Streiner DL, Norman GR, Cairney J. Health Measurement Scales. 5th ed. Oxford University Press; 2015.