Choosing Data Collection Methods for a Research Question

Data collection methods determine what your research can observe, what it can miss and which conclusions are defensible. A questionnaire is convenient for collecting the same responses from many people. It may be a poor way to discover why a work process fails when participants cannot explain the problem using the options provided.

This guide compares seven approaches and applies them to an illustrative training evaluation. The examples are planning exercises, not findings from an actual study. The aim is to help you justify a method, design useful measurements and avoid collecting more personal information than the question requires.

1. Turn the research topic into a measurable question

“Employee learning” is a topic. “Can employees correctly identify the escalation route in a new complaint scenario after training?” is a question with an observable outcome. It tells you more about the required evidence than a broad request to run a survey.

Specify the population, unit of observation, outcome and timing. A population might be newly assigned service agents. The unit could be one agent’s assessment attempt or one customer case. Those units are different: ten cases from one agent are not equivalent to ten independently sampled agents.

Make a short evidence map before writing questions. In one column, place the conclusion you hope to draw. In the next, describe the observation needed to support it. If the conclusion concerns demonstrated skill, a confidence rating alone leaves an important evidence gap.

2. Compare seven data collection approaches

These categories can overlap. An experiment is a study design and may collect outcomes using questionnaires, observations or logs. A focus group is not simply a faster way to conduct several individual interviews: interaction between participants is part of the evidence it produces.

3. Use surveys when comparable responses are needed

A survey can ask participants the same question using the same response options. This makes comparisons easier, but only when the question is understood as intended. “Was the course useful and easy to follow?” combines two judgments. A participant might find the content useful while struggling with navigation.

Separate the judgments and state a clear reference period where one is needed. Instead of “Do you often use the new procedure?”, ask about a defined period and an action people can reasonably recall. Include an appropriate option for people who had no opportunity to use the procedure.

The Government Analysis Function questionnaire guidance discusses testing, respondent burden and inclusive design. Use such guidance to inform a pilot, rather than treating an untested online form as a validated instrument.

4. Choose interviews and groups for explanation

An interview is useful when you need to understand a participant’s reasoning. Start with a concrete episode: “Tell me about the last case where you considered escalation.” Then ask what happened next, what information was available and what made the choice difficult. This usually gives a clearer account than asking whether the participant values good customer service.

Prepare a consistent topic guide while leaving room to follow relevant details. Separate what the participant actually said from the researcher’s interpretation. Keep a record of how codes or themes were developed so another person can understand the analysis process.

In a focus group, consider power relationships. A junior employee may not criticize a procedure in front of a supervisor. Establish discussion rules, avoid promising confidentiality that other participants cannot guarantee and consider whether individual interviews would be more suitable.

5. Observe behavior with a clear recording framework

Observation can show what people do during a task, including actions they forget to mention in an interview. Define the setting and the events to record. “Handles complaints professionally” is too vague for consistent scoring. “Checks the exception rule before promising a resolution” is easier to observe.

Practise using the recording framework on sample cases. When observers disagree, discuss which definition caused the disagreement and revise it. Do not quietly average incompatible judgments and assume the problem has disappeared.

Observation can also change behavior when people know they are being watched. State the conditions in the report and consider how closely the observed setting resembles ordinary work. Follow the consent and authorization requirements that apply to the study.

6. Examine records and logs before collecting new data

Existing records may answer part of the question with less participant burden. Check their original purpose. A field called “resolved” might mean an agent closed a case, a supervisor approved it or a customer confirmed satisfaction. Those definitions produce very different measures.

System logs need the same scrutiny. A course launch is not necessarily an engaged learner. Time on a page might include an idle browser tab. A missing event could reflect an integration failure rather than missing behavior. Write down event definitions and compare a few records with the process that generated them.

Keep an untouched source copy and document cleaning decisions. If you plan to work in R, our R and RStudio setup guide explains how to begin with a saved script and a reproducible project folder.

7. Pilot the entire collection process

A pilot should test more than spelling. Can eligible participants receive the invitation? Do they understand the purpose and instructions? Does a mobile form retain answers? Can a researcher export the data without losing categories or introducing duplicate records?

For an interview, rehearse the opening explanation and recording process. For observation, test the scoring definitions. For a survey, ask a few suitable people to explain how they understood important questions. A technically working form can still measure the wrong thing.

Worked example: evaluating a complaint-handling module

Suppose a team wants to know whether employees can apply an updated escalation policy. A small planning matrix might include an assessment case to examine learning, authorized case review to examine later application and interviews to understand barriers. Each source answers a different question.

Record how employees were selected, who did not participate, which cases were eligible and when collection occurred. Employees who volunteer for an interview may differ from those who do not. Reporting that limitation is more useful than describing the findings as representative without a defensible sampling basis.

For a full design example, see employee training modules. If the resulting question concerns association between two numeric variables, read the correlation in R guide before choosing a test.

Quality and privacy checks before fieldwork

• Define the observation and link it to a research question.
• Document recruitment, exclusions and expected missing information.
• Pilot wording, recording, export and coding.
• Explain participation and obtain the approvals required for the setting.
• Collect only the identifiers needed; restrict access and plan retention.
• Keep a data dictionary and a record of changes.
• Plan how uncertainty and limitations will be communicated.

No single method removes every source of bias. Combining methods can reveal useful differences, but it also creates more interpretation work. Explain why the sources belong together and examine disagreements instead of presenting agreement as automatic proof of accuracy.

Editorial update: Oxford Training and Research. Prepared with AI-assisted drafting; the original author record is retained. Worked examples are illustrative. No independent field study or specialist review is claimed.