Dig Deeper

Questionnaires and Surveys

To collect standardized data from a large number of participants.

Construction of survey.
Explicit verbal and/or written instructions.
If not proctored, follow-up contact to increase return rate.

Paper, scannable forms, CBT embedded surveys, e-mail, intranet, or Internet can be used.
Variety of response alternatives can be used (Likert-type scales, multiple choice, open-ended).
Questions can be direct or indirect, general or specific
Easy and efficient to administer.

If not proctored, participants cannot ask for clarification or instructions.
Participants may choose more than one response or give invalid responses.
Open-ended responses may be grammatically incorrect or ambiguous.
Participants may skip items.
Response rate may be low, if not proctored.

Knowledge Assessments

To assess participants' knowledge acquired through training or in the workplace or other environment.

Construction of assessment test.
Assessment must have content validity and reliability.
Must use exact test or an equivalent test for pre- and post-assessment.

Paper, scannable forms, CBT embedded tests, e-mail, intranet, or Internet can be used.
Easy to administer pre- and post-assessment.

If a pretest is used, participants may score higher on the posttest due to familiarity.
Some participants may experience test anxiety and may not perform optimally.

Performance Assessments

To assess participants' application of skills acquired through training or in the workplace or other environment.

Construction of assessment (checklist, rating form).
Tasks or skills to be performed must be identified.
Criteria or standards of performance must be specified and realistic.
Assessment must have content validity and reliability.

Paper, scannable forms, and interactive multimedia embedded tests can be used.
Performance can be rated by instructor, participant, peer, team or supervisors.
Provides direct evidence or application of learning.
Can be obtrusive or unobtrusive.

Potential rater bias.
Equipment and tools may not be available in the classroom.
If obtrusive, the observation and rating may affect performance due to nervousness or anxiety.
May not have time to measure all critical skills.
Pretest may not be practical given time constraints.

Structured Observation

To watch an activity and record what is seen.

Construction of checklist or rating form.
Explicit directions for the observer
Observer must be objective.
Observer should not give directions, answers, or nonverbal cues when observing performance.
Consistent use of the checklist.
Percentage agreement between observers should be calculated.

Objective of interest (for example, learner, designer, instructor, work sample) can be observed by a senior trainer, subject matter expert, designer/developer, evaluation specialist, supervisor, or manager.
Observation of performance can be obtrusive or unobtrusive (if unobtrusive, observation can be naturalistic).

Potential observer bias.
If obtrusive, observation of behavior may affect performance due to nervousness or anxiety.

Focus Groups

To explore a topic in-depth with a small number of participants.

Development of session questions.
Trained moderator.
Limited number of participants (6-12)
Different sessions for different groups of participants.

Depth of inquiry possible.
Opportunity for clarification.
Synergistic and snowball effects .

Potential group bias.
Potential moderator bias.
A few participants may dominate.
Results are not representative given the limited number of participants.

Telephone Interviews

To collect standardized reporting data over the telephone.

Creation of interview transcript and recording form.
Trained interviewers.
Multiple contact attempts.

Probing of incomplete answers is possible.
Clarification of misunderstandings is possible.
Interviewer has greater control over data collection.

Potential interview bias.
Participants may give socially desirable answers.
Questions and response alternatives must be simple because participants are unable to see the questionnaire.
Participants may be difficult to contact.