Document Type : CURRENT CONCEPTS REVIEW
Haaglanden Medisch Centrum, Leidschendam, Orthotrauma Research Center Amsterdam, Amsterdam, the Netherlands
Orthotrauma Research Center Amsterdam, Amsterdam, the Netherlands
Department of Plastic, Reconstructive, Hand en Burn Surgery, Martini Hospital, Burn Center Groningen, the Netherlands
Flinders Medical Center, Royal Adelaide Hospital, Adelaide, Australia
In orthopedic surgery, there is an increasing number of papers about online studies on the reliability of classification
systems. Useful classification systems need to be reliable and valid. Measurement of validity can be variable and is
prone to observer bias.
These online collaboratives derive adequate power to study reliability by having a large group of trained surgeons
review a small number of cases instead of the “classic” reliability studies in which a small number of observers evaluate
many cases. Large online interobserver studies have advantages (i.e., less than 15 minutes to complete the survey,
the ability to randomize, and the ability to study factors associated with reliability, accuracy, or decision-making). This
‘handbook’ paper gives an overview of current methods for online reliability studies. We discuss the study design,
sample size calculation, statistical analyses of results, pitfalls, and limitations of the study design.
Level of evidence: V