Bias, validity and precision in clinical studies

At a glance

• Every study estimate is the true value plus random error plus systematic error (bias).
• Precision is freedom from random error. It shows up as the width of the confidence interval and improves with sample size.
• Validity is freedom from systematic error. Bias doesn't shrink as the study grows.
• Internal validity: the answer is right for the people studied. External validity: it applies to your patient.
• Bias comes in two main families: selection (who gets in or stays in) and information (how exposure or outcome is measured). Confounding is often listed as a third and has its own note, Confounding and effect modification.

Precision and validity

The archery target is the standard picture. Tight cluster on the bullseye: precise and valid. Tight cluster off to one side: precise but biased. Arrows scattered evenly around the centre: valid on average but imprecise.

	Random error	Systematic error (bias)
Effect on estimate	Scatter around the truth	Shift away from the truth
Reduced by	Larger sample, more events, repeated measures	Better design: randomisation, concealment, blinding, complete follow-up
Quantified by	Confidence interval, p value	Not by statistics. Judged by appraising the methods

Worked example: sample size and precision

A trial of 400 patients reports an absolute risk reduction of 4% (95% CI 0% to 8%). The half-width of the interval is about 4 percentage points.

• The standard error scales with 1/√n. To cut the half-width to 2 points, you need about 1,600 patients (four times as many).
• If allocation in the original trial wasn't concealed, the bigger trial reproduces the same bias with a tighter interval. Precision can make a wrong answer look convincing.

Pitfall: a narrow confidence interval says nothing about bias. Questions often pair a huge observational study with a small RCT and ask which is more likely to be valid.

Selection bias

The groups compared differ in a way that relates to the outcome, because of how participants were chosen or lost.

Bias	Mechanism	Typical setting
Healthy worker effect	Employed people are healthier than the general population	Occupational cohorts compared with population rates
Berkson's bias	Exposure and disease each change the chance of hospital admission, so a spurious association appears among hospital patients (a form of collider bias). Separately, hospital controls whose illness is caused by the exposure are over-exposed and bias the OR towards 1	Hospital-based case-control studies
Loss to follow-up (attrition)	Those who drop out differ in prognosis	Cohorts and trials. Concern rises above about 20% loss or when loss differs between arms
Volunteer bias	Volunteers are healthier and more adherent	Screening programs, surveys
Survival (prevalence-incidence) bias	Prevalent cases are survivors, so fatal cases are missed	Cross-sectional and case-control studies using existing cases

Two screening biases are examined so often they deserve separate mention. Lead-time bias inflates survival from diagnosis because the clock starts earlier. Length-time bias over-represents indolent disease among screen-detected cases. Both explain why screening is judged by disease-specific mortality in an RCT, not by survival time. See Colorectal cancer and Breast cancer for the screening programs themselves.

Information bias

Measurement of exposure or outcome is wrong. The key question is whether the error is the same in both groups.

• Non-differential misclassification: errors occur equally in cases and controls (or exposed and unexposed). For a binary exposure it pulls the estimate towards the null on average.
• Differential misclassification: error rates differ between groups. It can push the estimate either way, often away from the null.
• Recall bias: cases search their memory harder than controls. Differential.
• Observer (detection) bias: outcome assessors who know exposure or treatment look harder. Blinding prevents it.
• Social desirability bias: under-reporting of alcohol, smoking or drug use.

Worked example: misclassification

A case-control study has 200 cases and 200 controls. Truly, 100 cases (50%) and 50 controls (25%) took a drug.

• True OR = (100 × 150) / (100 × 50) = 3.0.
• Non-differential: a questionnaire picks up only 80% of true users in both groups. Cases: 80 exposed, 120 unexposed. Controls: 40 exposed, 160 unexposed. OR = (80 × 160) / (120 × 40) = 2.67. The association shrinks.
• Differential: cases recall 100% of use, controls only 80%. Cases 100/100, controls 40/160. OR = (100 × 160) / (100 × 40) = 4.0. The association grows.

Validity of measurements

• Validity (accuracy) of a measurement: does it measure the true value? Assessed against a reference standard. See Diagnostic test accuracy.
• Reliability (reproducibility): does it give the same answer twice? Measured by kappa for categories and the intraclass correlation coefficient for continuous data. A reliable test can be consistently wrong.

How it's tested

• Classify the bias from a short scenario: recall bias in a birth defect study, healthy worker effect in a mining cohort, attrition in a weight-loss trial.
• Predict direction: non-differential towards the null, recall bias away from it.
• Precision versus validity: what a larger sample does and doesn't fix.
• Screening scenarios: improved 5-year survival with no change in mortality points to lead-time bias.
• Risk-of-bias tools: RoB 2 for randomised trials (domains: randomisation process, deviations from intended interventions, missing outcome data, outcome measurement, selection of the reported result). ROBINS-I for non-randomised studies of interventions.

Pitfalls

• Treating a small p value as evidence that a study is unbiased. p values only address chance.
• Assuming non-differential misclassification always biases to the null. That holds for a binary exposure with errors independent of other variables, and only on average (expected value); an individual study can still move away from the null by chance. With three or more exposure categories, a middle category can move either way.
• Trying to fix confounding by blinding. Confounding is handled by randomisation, restriction, matching or adjustment.
• Forgetting that external validity is a judgement about your patient. A trial that excluded people over 75 or with eGFR below 30 may be internally sound and still not apply.
• Mixing up lead-time and length-time bias. Lead time: same death date, earlier diagnosis. Length time: screening finds the slow tumours.