Chapter 2: Methods in Real-World Evidence Generation — Study Design


  1. The European Network of Centres for Pharmacoepidemiology and Pharmacovigilance (ENCePP). Guide on Methodological Standards in Pharmacoepidemiology (Revision 9). EMA/95098/2010. Accessed January 5, 2022.
  2. Guidelines for Good Pharmacoepidemiology Practices (GPP) - International Society for Pharmacoepidemiology. Accessed January 5, 2022.
  3. Hernán MA, Robins JM. Using Big Data to Emulate a Target Trial When a Randomized Trial Is Not Available. Am J Epidemiol. 2016;183(8):758-764. doi:10.1093/aje/kwv254
  4. Didelez V. Commentary: Should the analysis of observational data always be preceded by specifying a target experimental trial? Int J Epidemiol. 2016;45(6):2049-2051. doi:10.1093/ije/dyw032
  5. Labrecque JA, Swanson SA. Target trial emulation: teaching epidemiology and beyond. Eur J Epidemiol. 2017;32(6):473-475. doi:10.1007/s10654-017-0293-4
  6. Ford I, Norrie J. Pragmatic Trials. N Engl J Med. 2016;375(5):454-463. doi:10.1056/NEJMra1510059
  7. Hernán MA, Sauer BC, Hernández-Díaz S, Platt R, Shrier I. Specifying a target trial prevents immortal time bias and other self-inflicted injuries in observational analyses. J Clin Epidemiol. 2016;79:70-75. doi:10.1016/j.jclinepi.2016.04.014
  8. Hernán MA. The C-Word: Scientific Euphemisms Do Not Improve Causal Inference From Observational Data. Am J Public Health. 2018;108(5):616-619. doi:10.2105/AJPH.2018.304337
  9. Sterne JA, Hernán MA, Reeves BC, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355:i4919. doi:10.1136/bmj.i4919
  10. Centers for Disease Control and Prevention. Real-World COVID-19 Vaccine Effectiveness in Healthcare Workers | CDC. Published August 25, 2021. Accessed January 5, 2022.
  11. Dagan N, Barda N, Kepten E, et al. BNT162b2 mRNA Covid-19 Vaccine in a Nationwide Mass Vaccination Setting. N Engl J Med. 2021;384(15):1412-1423. doi:10.1056/NEJMoa2101765
  12. Dagan N Barda N, Kepten E, et al. Protocol for: Dagan N, Barda N, Kepten E, et al. BNT162b2 mRNA Covid-19 vaccine in a nationwide mass vaccination setting. N Engl J Med 2021;384:1412-23. DOI: 10.1056/NEJMoa2101765. Published 2021. Accessed September 9, 2021.
  13. Thomas SJ, Moreira ED, Kitchin N, et al. Safety and Efficacy of the BNT162b2 mRNA Covid-19 Vaccine through 6 Months. N Engl J Med. 2021;385(19):1761-1773. doi:10.1056/NEJMoa2110345
  14. Gupta S, Wang W, Hayek SS, et al. Association Between Early Treatment With Tocilizumab and Mortality Among Critically Ill Patients With COVID-19. JAMA Intern Med. 2021;181(1):41-51. doi:10.1001/jamainternmed.2020.6252
  15. Leaf D. Study of the Treatment and Outcomes in Critically Ill Patients With COVID-19.; 2021. Accessed January 4, 2022.
  16. RECOVERY Collaborative Group. Tocilizumab in patients admitted to hospital with COVID-19 (RECOVERY): a randomised, controlled, open-label, platform trial. Lancet. 2021;397(10285):1637-1645. doi:10.1016/S0140-6736(21)00676-0
  17. Rosas IO, Bräu N, Waters M, et al. Tocilizumab in Hospitalized Patients with Severe Covid-19 Pneumonia. N Engl J Med. 2021;384(16):1503-1516. doi:10.1056/NEJMoa2028700
  18. Salama C, Mohan SV. Tocilizumab in Patients Hospitalized with Covid-19 Pneumonia. Reply. N Engl J Med. 2021;384(15):1473-1474. doi:10.1056/NEJMc2100217
  19. Roche Group Media Relations. Roche provides update on the phase III REMDACTA trial of Actemra/RoActemra plus Veklury in patients with severe COVID-19 pneumonia. Accessed January 5, 2022.
  20. Al-Samkari H, Gupta S, Leaf RK, et al. Thrombosis, Bleeding, and the Observational Effect of Early Therapeutic Anticoagulation on Survival in Critically Ill Patients With COVID-19. Ann Intern Med. 2021;174(5):622-632. doi:10.7326/M20-6739
  21. Thachil J, Juffermans NP, Ranucci M, et al. ISTH DIC subcommittee communication on anticoagulation in COVID-19. J Thromb Haemost. 2020;18(9):2138-2144. doi:10.1111/jth.15004
  22. Bikdeli B, Madhavan MV, Jimenez D, et al. COVID-19 and Thrombotic or Thromboembolic Disease: Implications for Prevention, Antithrombotic Therapy, and Follow-Up: JACC State-of-the-Art Review. J Am Coll Cardiol. 2020;75(23):2950-2973. doi:10.1016/j.jacc.2020.04.031
  23. Gatto N, Garry EM, Chakravarty A. Protocol: Effect of dexamethasone on inpatient mortality among hospitalized COVID-19 patients. Accessed January 5, 2022.
  24. Mahévas M, Tran VT, Roumier M, et al. Clinical efficacy of hydroxychloroquine in patients with covid-19 pneumonia who require oxygen: observational comparative study using routine care data. BMJ. 2020;369:m1844. doi:10.1136/bmj.m1844
  25. Mahévas M et al. No evidence of clinical efficacy of hydroxychloroquine in patients hospitalised for COVID-19 infection and requiring oxygen: results of a study using routinely collected data to emulate a target trial | medRxiv. Accessed January 5, 2022.
  26. Self WH, Semler MW, Leither LM, et al. Effect of Hydroxychloroquine on Clinical Status at 14 Days in Hospitalized Patients With COVID-19: A Randomized Clinical Trial. JAMA. 2020;324(21):2165-2176. doi:10.1001/jama.2020.22240
  27. Suissa S, Dell’Aniello S. Time-related biases in pharmacoepidemiology. Pharmacoepidemiol Drug Saf. 2020;29(9):1101-1110. doi:10.1002/pds.5083
  28. Grodstein F, Manson JE, Stampfer MJ. Hormone therapy and coronary heart disease: the role of time since menopause and age at hormone initiation. J Womens Health (Larchmt). 2006;15(1):35-44. doi:10.1089/jwh.2006.15.35
  29. Manson JE, Hsia J, Johnson KC, et al. Estrogen plus progestin and the risk of coronary heart disease. N Engl J Med. 2003;349(6):523-534. doi:10.1056/NEJMoa030808
  30. Renoux C, Dell’Aniello S, Brenner B, Suissa S. Bias from depletion of susceptibles: the example of hormone replacement therapy and the risk of venous thromboembolism. Pharmacoepidemiol Drug Saf. 2017;26(5):554-560. doi:10.1002/pds.4197
  31. Hernán MA, Alonso A, Logan R, et al. Observational Studies Analyzed Like Randomized Experiments: An Application to Postmenopausal Hormone Therapy and Coronary Heart Disease. Epidemiology. 2008;19(6):766-779. doi:10.1097/EDE.0b013e3181875e61
  32. Hernán MA, Alonso A, Logan R, et al. Observational studies analyzed like randomized experiments: an application to postmenopausal hormone therapy and coronary heart disease. Epidemiology. 2008;19(6):766-779. doi:10.1097/EDE.0b013e3181875e61
  33. Danaei G, Tavakkoli M, Hernán MA. Bias in Observational Studies of Prevalent Users: Lessons for Comparative Effectiveness Research From a Meta-Analysis of Statins. Am J Epidemiol. 2012;175(4):250-262. doi:10.1093/aje/kwr301
  34. Ray WA. Evaluating medication effects outside of clinical trials: new-user designs. Am J Epidemiol. 2003;158(9):915-920. doi:10.1093/aje/kwg231
  35. Catalogue of bias collaboration, Lee H, Aronson JK, Nunan D. Collider bias. Catalog of Bias. Published March 1, 2019. Accessed January 5, 2022.
  36. Rothman KJ, Greenland S, Lash TL. (2008). Modern Epidemiology, 3rd Edition. Lippincott Williams & Wilkins
  37. Strom BL, Kimmel  SE, Hennessy S. (Eds). Pharmacoepidemiology, 6th Edition. John Wiley & Sons. December 16, 2019.
  38. Dean NE, Hogan JW, Schnitzer ME. Covid-19 Vaccine Effectiveness and the Test-Negative Design. New England Journal of Medicine. 2021;385(15):1431-1433. doi:10.1056/NEJMe2113151
  39. Thompson MG, Stenehjem E, Grannis S, et al. Effectiveness of Covid-19 Vaccines in Ambulatory and Inpatient Care Settings. New England Journal of Medicine. 2021;385(15):1355-1371. doi:10.1056/NEJMoa2110362
  40. Vandenbroucke JP, Brickley EB, Vandenbroucke-Grauls CMJE, Pearce N. A Test-Negative Design with Additional Population Controls Can Be Used to Rapidly Study Causes of the SARS-CoV-2 Epidemic. Epidemiology. 2020;31(6):836-843. doi:10.1097/EDE.0000000000001251
  41. Endo A, Funk S, Kucharski AJ. Bias correction methods for test-negative designs in the presence of misclassification. Epidemiology & Infection. 2020;148. doi:10.1017/S0950268820002058
  42. Lewnard JA, Patel MM, Jewell NP, et al. Theoretical Framework for Retrospective Studies of the Effectiveness of SARS-CoV-2 Vaccines. Epidemiology. 2021;32(4):508-517. doi:10.1097/EDE.0000000000001366
  43. Schneeweiss S. A basic study design for expedited safety signal evaluation based on electronic healthcare data. Pharmacoepidemiology and Drug Safety. 2010;19(8):858-868. doi:10.1002/pds.1926
  44. Schneeweiss S, Rassen JA, Brown JS, et al. Graphical Depiction of Longitudinal Study Designs in Health Care Databases. Ann Intern Med. 2019;170(6):398-406. doi:10.7326/M18-3079
  45. Rassen JA, Bartels DB, Schneeweiss S, Patrick AR, Murk W. Measuring prevalence and incidence of chronic conditions in claims and electronic health record databases. Clin Epidemiol. 2019;11:1-15. doi:10.2147/CLEP.S181242
  46. Johnson ES, Bartman BA, Briesacher BA, et al. The incident user design in comparative effectiveness research. Pharmacoepidemiol Drug Saf. 2013;22(1):1-6. doi:10.1002/pds.3334
  47. Lund JL, Richardson DB, Stürmer T. The active comparator, new user study design in pharmacoepidemiology: historical foundations and contemporary application. Curr Epidemiol Rep. 2015;2(4):221-228. doi:10.1007/s40471-015-0053-5
  48. Franklin JM, Lin KJ, Gatto NM, Rassen JA, Glynn RJ, Schneeweiss S. Real-World Evidence for Assessing Pharmaceutical Treatments in the Context of COVID-19. Clinical Pharmacology & Therapeutics. 2021;109(4):816-828. doi:10.1002/cpt.2185
  49. Suissa S, Moodie EEM, Dell’Aniello S. Prevalent new-user cohort designs for comparative drug effect studies by time-conditional propensity scores. Pharmacoepidemiol Drug Saf. 2017;26(4):459-468. doi:10.1002/pds.4107
  50. Filion KB, Yu YH. Invited Commentary: The Prevalent New-User Design in Pharmacoepidemiology—Challenges and Opportunities. American Journal of Epidemiology. 2021;190(7):1349-1352. doi:10.1093/aje/kwaa284
  51. Webster-Clark M, Ross RK, Lund JL. Initiator Types and the Causal Question of the Prevalent New-User Design: A Simulation Study. American Journal of Epidemiology. 2021;190(7):1341-1348. doi:10.1093/aje/kwaa283
  52. Renoux C, Azoulay L, Suissa S. Biases in Evaluating the Safety and Effectiveness of Drugs for the Treatment of COVID-19: Designing Real-World Evidence Studies. Am J Epidemiol. 2021;190(8):1452-1456. doi:10.1093/aje/kwab028
  53. Suissa S. Immortal Time Bias in Pharmacoepidemiology. American Journal of Epidemiology. 2008;167(4):492-499. doi:10.1093/aje/kwm324
  54. Patorno E, Glynn RJ, Levin R, Lee MP, Huybrechts KF. Benzodiazepines and risk of all cause mortality in adults: cohort study. BMJ. 2017;358:j2941. doi:10.1136/bmj.j2941
  55. Wang SV, Schneeweiss S, Berger ML, et al. Reporting to Improve Reproducibility and Facilitate Validity Assessment for Healthcare Database Studies V1.0. Pharmacoepidemiol Drug Saf. 2017;26(9):1018-1032. doi:10.1002/pds.4295
  56. Wang SV, Pinheiro S, Hua W, et al. STaRT-RWE: structured template for planning and reporting on the implementation of real world evidence studies. BMJ. 2021;372:m4856. doi:10.1136/bmj.m4856
  57. Hernán MA, Hernández-Díaz S. Beyond the intention-to-treat in comparative effectiveness research. Clin Trials. 2012;9(1):48-55. doi:10.1177/1740774511420743
  58. Danaei G, Rodríguez LAG, Cantero OF, Logan R, Hernán MA. Observational data for comparative effectiveness research: An emulation of randomised trials of statins and primary prevention of coronary heart disease. Stat Methods Med Res. 2013;22(1):70-96. doi:10.1177/0962280211403603
  59. Stewart M, Rodriguez-Watson C, Albayrak A, et al. COVID-19 Evidence Accelerator: A parallel analysis to describe the use of Hydroxychloroquine with or without Azithromycin among hospitalized COVID-19 patients. Di Gennaro F, ed. PLoS ONE. 2021;16(3):e0248128. doi:10.1371/journal.pone.0248128
  60. Jarcho JA, Ingelfinger JR, Hamel MB, D’Agostino RB, Harrington DP. Inhibitors of the Renin-Angiotensin-Aldosterone System and Covid-19. N Engl J Med. 2020;382(25):2462-2464. doi:10.1056/NEJMe2012924
  61. Yoshida K, Solomon DH, Kim SC. Active-comparator design and new-user design in observational studies. Nat Rev Rheumatol. 2015;11(7):437-441. doi:10.1038/nrrheum.2015.30
  62. Cadarette SM, Maclure M, Delaney JAC, et al. Control yourself: ISPE‐endorsed guidance in the application of self‐controlled study designs in pharmacoepidemiology. Pharmacoepidemiol Drug Saf. 2021;30(6):671-684. doi:10.1002/pds.5227
  63. Farrington CP. Relative Incidence Estimation from Case Series for Vaccine Safety Evaluation. Biometrics. 1995;51(1):228. doi:10.2307/2533328
  64. Katsoularis I, Fonseca-Rodríguez O, Farrington P, Lindmark K, Fors Connolly AM. Risk of acute myocardial infarction and ischaemic stroke following COVID-19 in Sweden: a self-controlled case series and matched cohort study. The Lancet. 2021;398(10300):599-607. doi:10.1016/S0140-6736(21)00896-5
  65. Douglas IJ, Evans SJW, Hingorani AD, et al. Clopidogrel and interaction with proton pump inhibitors: comparison between cohort and within person study designs. BMJ. 2012;345(jul10 1):e4388-e4388. doi:10.1136/bmj.e4388
  66. Petersen I, Douglas I, Whitaker H. Self controlled case series methods: an alternative to standard epidemiological study designs. BMJ. Published online September 12, 2016:i4515. doi:10.1136/bmj.i4515
  67. Whitaker HJ, Ghebremichael-Weldeselassie Y, Douglas IJ, Smeeth L, Farrington CP. Investigating the assumptions of the self-controlled case series method: Investigating the assumptions of the self-controlled case series method. Statistics in Medicine. 2018;37(4):643-658. doi:10.1002/sim.7536
  68. Simpson CR, Shi T, Vasileiou E, et al. First-dose ChAdOx1 and BNT162b2 COVID-19 vaccines and thrombocytopenic, thromboembolic and hemorrhagic events in Scotland. Nat Med. 2021;27(7):1290-1297. doi:10.1038/s41591-021-01408-4
  69. Hippisley-Cox J, Patone M, Mei XW, et al. Risk of thrombocytopenia and thromboembolism after covid-19 vaccination and SARS-CoV-2 positive testing: self-controlled case series study. BMJ. Published online August 26, 2021:n1931. doi:10.1136/bmj.n1931
  70. Aschengrau A, Seage GR. Descriptive Epidemiology. In: Essentials of Epidemiology in Public Health. Fourth edition. Jones & Bartlett Learning; 2020.
  71. Chew MS, Kattainen S, Haase N, et al. A descriptive study of the surge response and outcomes of ICU patients with COVID‐19 during first wave in Nordic countries. Acta Anaesthesiol Scand. 2022;66(1):56-64. doi:10.1111/aas.13983
  72. Mehta HB, An H, Andersen KM, et al. Use of Hydroxychloroquine, Remdesivir, and Dexamethasone Among Adults Hospitalized With COVID-19 in the United States: A Retrospective Cohort Study. Ann Intern Med. 2021;174(10):1395-1403. doi:10.7326/M21-0857
  73. Prats-Uribe A, Sena AG, Lai LYH, et al. Use of repurposed and adjuvant drugs in hospital patients with covid-19: multinational network cohort study. BMJ. Published online May 11, 2021:n1038. doi:10.1136/bmj.n1038
  74. Savoia C, Volpe M, Kreutz R. Hypertension, a Moving Target in COVID-19: Current Views and Perspectives. Circ Res. 2021;128(7):1062-1079. doi:10.1161/CIRCRESAHA.121.318054
  75. Bradley MC, Graham DJ, Eworuke E, et al. Outpatient corticosteroid use for COVID-19 in the United States: a multi-database study. 2021.
  76. Anderson TS, O’Donoghue AL, Dechen T, Mechanic O, Stevens JP. Uptake of Outpatient Monoclonal Antibody Treatments for COVID-19 in the United States: a Cross-Sectional Analysis. J GEN INTERN MED. 2021;36(12):3922-3924. doi:10.1007/s11606-021-07109-5