Principles of research: limitations of non-randomized studies

Controlled non-randomized trials (NRS), or cohort studies, are often used to evaluate the effectiveness of surgical treatments. NRS have limitations which must be considered carefully both when doing research and deciding whether to use research to inform practice. NRS have diverse design features. Three aspects of study design are the relationship in time between groups being compared; whether or not participants were treated before the study was conceived; and the basis for allocating a treatment to a participant. These are important because they are believed to influence the risk of bias. In randomized controlled trials (RCTs), treatment is allocated by chance: when randomization is concealed, this rules out systematic baseline differences between groups. In NRS, the decision to give a particular treatment to a patient is made either by researchers or collectively by the attending doctor and patient. Both methods are likely to give rise to imbalances in prognostic factors between the groups being compared. Imbalances can be reduced by matching, or controlled for by statistical methods, but their confounding effects can never be entirely removed. Imbalances can bias estimates of treatment effects and almost always increase their uncertainty. Recent research increases the pressure for researchers to do RCTs, if at all possible, when evaluating beneficial effects of treatments. Confounding is less likely when researching rare but serious harms, which are very difficult to investigate with RCTs. NRS may have a valuable role in this context.