BCBA® task C.8 · Area C

C.8 Validity, accuracy and reliability of measurement

3 min read · BCBA® Test Content Outline, 6th edition

What it means

Validity asks whether the right thing was measured: the socially significant behavior itself, on the dimension that matters, at times and places representative of the concern. Accuracy asks whether the recorded values match what really happened, which can be checked against a reference such as a carefully scored video. Reliability asks whether measuring the same events again gives the same values. A measure can be reliable but inaccurate, when two observers make the same mistake, and accurate but invalid, when words read per minute are counted perfectly while the concern is comprehension. Interobserver agreement (IOA) compares independent observers and is reported as a percentage. Event data: total count IOA divides the smaller total by the larger; mean count-per-interval IOA averages that ratio over short intervals; exact count-per-interval IOA counts intervals with identical counts; trial-by-trial IOA counts trials scored the same way. Timing data: total duration IOA divides the shorter total by the longer; mean duration-per-occurrence IOA averages that ratio over occurrences. Interval data: interval-by-interval IOA counts agreements over all intervals; scored-interval IOA uses only intervals where at least one observer scored the behavior, the stricter choice for low rates; unscored-interval IOA uses only intervals where at least one observer did not, the stricter choice for high rates. Observer drift, reactivity and observer expectations threaten the data and are managed with training, calibration and regular checks.

Everyday example

Two observers time a child’s crying in one session: 300 and 270 seconds. Total duration IOA is 270 ÷ 300 × 100 = 90%. That agreement makes the record believable and consistent; it does not prove that either stopwatch matched what really happened, and it says nothing about whether crying was the right target.

Common confusion

High IOA shows believability, not accuracy and not validity. Total count IOA can reach 100% when the observers recorded different instances that happened to add up to the same total. Observer drift is a change in how an observer applies the definition over time; reactivity is a change in the client’s behavior because someone is watching.

Key points

  • Valid measurement directly measures the socially significant behavior, on the dimension that matters, at times and places representative of the problem.
  • Accuracy is agreement with the true value; reliability is consistency of repeated measurement; measurement can be reliable and still inaccurate, and high interobserver agreement shows believability, not accuracy.
  • IOA methods differ in stringency: total count, mean count-per-interval, exact count-per-interval and trial-by-trial for event data; total duration and mean duration-per-occurrence for timing data; interval-by-interval, scored-interval (low rates) and unscored-interval (high rates) for interval data.
  • IOA is computed as smaller over larger (or agreements over agreements plus disagreements) times 100; total count agreement can be high even when observers recorded different instances.
  • Threats include observer drift, reactivity to being observed, observer expectations and poorly designed measurement systems; they are controlled by training, calibration, retraining and periodic checks.

Examples from practice

Example 1

A family wants to know whether bedtime arguments drop at home, so observation happens at home at bedtime: a valid choice of time and place.

Example 2

Two referees both miss the same fouls hidden behind other players; they agree closely, but their counts are low compared with the video.

Non-example

Concluding that data are accurate because two observers agreed at 97 percent confuses agreement with the true value.

Check yourself

One question from the course lesson on this task. Choose an answer to see why it is right or wrong.

A company wants to improve the quality of an employee's written reports. Two supervisors reliably count how many minutes he spends typing each day, with 96 percent IOA. What is the main weakness?

Show the answer and the explanations

Answer: B. Validity: time typing does not measure report quality

  • a. 96 percent agreement is high. The problem is not consistency but what is being measured.
  • b. Correct. Minutes typing are measured consistently, but they are the wrong dimension for the question about quality.
  • c. Nothing suggests the minutes are wrong; two observers can both be accurate. The issue is that minutes do not answer the question.

The full lesson is in the course

The course lesson on this task goes further, with more sections, more examples and more practice questions, and keeps a record of what you have mastered.

Open the lesson Practice this task

Try a sample free, without an account

Key terms

Each term opens its definition in the ABA glossary.

All tasks in area C All 104 tasks ABA glossary: every term in one page

Independent study resource. INTERLAZA is not affiliated with or endorsed by the BACB. The explanations and examples are original, are not official exam content and do not reproduce any exam item.