Chapter 3: Methods in Real-World Evidence Generation - Sources of Error

References

  1. Salas M, Hofman A, Stricker BH. Confounding by indication: an example of variation in the use of epidemiologic terminology. Am J Epidemiol. 1999;149(11):981-983. doi:10.1093/oxfordjournals.aje.a009758
  2. Blais L, Ernst P, Suissa S. Confounding by indication and channeling over time: the risks of beta 2-agonists. Am J Epidemiol. 1996;144(12):1161-1169. doi:10.1093/oxfordjournals.aje.a008895
  3. Schneeweiss S, Rassen JA, Glynn RJ, Avorn J, Mogun H, Brookhart MA. High-dimensional Propensity Score Adjustment in Studies of Treatment Effects Using Health Care Claims Data. Epidemiology. 2009;20(4):512-522. doi:10.1097/EDE.0b013e3181a663cc
  4. CDC. Science Brief: Evidence Used to Update the List of Underlying Medical Conditions Associated with Higher Risk for Severe COVID-19. Centers for Disease Control and Prevention. Published February 11, 2020. Accessed January 7, 2022. https://www.cdc.gov/coronavirus/2019-ncov/science/science-briefs/underlying-evidence-table.html
  5. Kowall B, Stang A, Rathmann W, Kostev K. No reduced risk of overall, colorectal, lung, breast, and prostate cancer with metformin therapy in diabetic patients: database analyses from Germany and the UK: Metformin and Cancer Sites. Pharmacoepidemiol Drug Saf. 2015;24(8):865-874. doi:10.1002/pds.3823
  6. Paranjpe I, Fuster V, Lala A, et al. Association of Treatment Dose Anticoagulation With In-Hospital Survival Among Hospitalized Patients With COVID-19. J Am Coll Cardiol. 2020;76(1):122-124. doi:10.1016/j.jacc.2020.05.001
  7. Maley JH, Petri CR, Brenner LN, et al. Anticoagulation, immortality, and observations of COVID-19. Res Pract Thromb Haemost. 2020;4(5):674-676. doi:10.1002/rth2.12398
  8. Schultze A, Walker AJ, MacKenna B, et al. Risk of COVID-19-related death among patients with chronic obstructive pulmonary disease or asthma prescribed inhaled corticosteroids: an observational cohort study using the OpenSAFELY platform. Lancet Respir Med. 2020;8(11):1106-1120. doi:10.1016/S2213-2600(20)30415-X
  9. Lipsitch M, Tchetgen Tchetgen E, Cohen T. Negative controls: a tool for detecting confounding and bias in observational studies. Epidemiology. 2010;21(3):383-388. doi:10.1097/EDE.0b013e3181d61eeb
  10. Schuemie MJ, Hripcsak G, Ryan PB, Madigan D, Suchard MA. Empirical confidence interval calibration for population-level effect estimation studies in observational healthcare data. Proc Natl Acad Sci USA. 2018;115(11):2571-2577. doi:10.1073/pnas.1708282114
  11. Schneeweiss S. Developments in post-marketing comparative effectiveness research. Clin Pharmacol Ther. 2007;82(2):143-156. doi:10.1038/sj.clpt.6100249
  12. Schneeweiss S. Sensitivity analysis and external adjustment for unmeasured confounders in epidemiologic database studies of therapeutics. Pharmacoepidemiol Drug Saf. 2006;15(5):291-303. doi:10.1002/pds.1200
  13. Velentgas P DN Nourjah P, Smith SR, Torchia MM, eds. Developing a Protocol for Observational Comparative Effectiveness Research: A User’s Guide. AHRQ Publication No 12(13)-EHC099. Published online 2013. www.effectivehealthcare.ahrq.gov/
  14. Franklin JM, Lin KJ, Gatto NM, Rassen JA, Glynn RJ, Schneeweiss S. Real-World Evidence for Assessing Pharmaceutical Treatments in the Context of COVID-19. Clinical Pharmacology & Therapeutics. 2021;109(4):816-828. doi:10.1002/cpt.2185
  15. Webster-Clark M, Ross RK, Lund JL. Initiator Types and the Causal Question of the Prevalent New-User Design: A Simulation Study. American Journal of Epidemiology. 2021;190(7):1341-1348. doi:10.1093/aje/kwaa283
  16. Renoux C, Azoulay L, Suissa S. Biases in Evaluating the Safety and Effectiveness of Drugs for the Treatment of COVID-19: Designing Real-World Evidence Studies. Am J Epidemiol. 2021;190(8):1452-1456. doi:10.1093/aje/kwab028
  17. VanderWeele TJ, Ding P. Sensitivity Analysis in Observational Research: Introducing the E-Value. Ann Intern Med. 2017;167(4):268-274. doi:10.7326/M16-2607
  18. Rothman KJ, Greenland, Sander, Lash TL. Modern Epidemiology,  Third Edition.; 2008.
  19. Griffith GJ, Morris TT, Tudball MJ, et al. Collider bias undermines our understanding of COVID-19 disease risk and severity. Nat Commun. 2020;11(1):5749. doi:10.1038/s41467-020-19478-2
  20. Hernan MA, Hernandez-Diaz S, Robins JM. A structural approach to selection bias. Epidemiology. 2004;15(5):615-625. doi:10.1097/01.ede.0000135174.63482.43
  21. Thompson MG, Stenehjem E, Grannis S, et al. Effectiveness of Covid-19 Vaccines in Ambulatory and Inpatient Care Settings. New England Journal of Medicine. 2021;385(15):1355-1371. doi:10.1056/NEJMoa2110362
  22. Meng J, Xiao G, Zhang J, et al. Renin-angiotensin system inhibitors improve the clinical outcomes of COVID-19 patients with hypertension. Emerg Microbes Infect. 2020;9(1):757-760. doi:10.1080/22221751.2020.1746200
  23. Zhang P, Zhu L, Cai J, et al. Association of Inpatient Use of Angiotensin-Converting Enzyme Inhibitors and Angiotensin II Receptor Blockers With Mortality Among Patients With Hypertension Hospitalized With COVID-19. Circ Res. 2020;126(12):1671-1681. doi:10.1161/CIRCRESAHA.120.317134
  24. Zhou P, Yang XL, Wang XG, et al. A pneumonia outbreak associated with a new coronavirus of probable bat origin. Nature. 2020;579(7798):270-273. doi:10.1038/s41586-020-2012-7
  25. Daniels LB, Sitapati AM, Zhang J, et al. Relation of Statin Use Prior to Admission to Severity and Recovery Among COVID-19 Inpatients. Am J Cardiol. 2020;136:149-155. doi:10.1016/j.amjcard.2020.09.012
  26. Mahévas M, Tran VT, Roumier M, et al. Clinical efficacy of hydroxychloroquine in patients with covid-19 pneumonia who require oxygen: observational comparative study using routine care data. BMJ. 2020;369:m1844. doi:10.1136/bmj.m1844
  27. Arch B, Kovacs D, Scott J, et al. Evaluation of the effectiveness of remdesivir in treating severe COVID-19 using data from the ISARIC WHO Clinical Characterisation Protocol UK: a prospective, national cohort study. Published online 2021. doi:https://doi.org/10.1101/2021.06.18.21259072
  28. Patel AP, Paranjpe MD, Kathiresan NP, Rivas MA, Khera AV. Race, Socioeconomic Deprivation, and Hospitalization for COVID-19 in English participants of a National Biobank. medRxiv. Published online May 2, 2020. doi:10.1101/2020.04.27.20082107
  29. Howe CJ, Cole SR, Lau B, Napravnik S, Eron JJ Jr. Selection Bias Due to Loss to Follow Up in Cohort Studies. Epidemiology. 2016;27(1):91-97. doi:10.1097/EDE.0000000000000409
  30. Smith LH, VanderWeele TJ. Bounding Bias Due to Selection. Epidemiology. 2019;30(4):509-516. doi:10.1097/EDE.0000000000001032
  31. Seaman SR, White IR. Review of inverse probability weighting for dealing with missing data. Stat Methods Med Res. 2013;22(3):278-295. doi:10.1177/0962280210395740
  32. Suissa S, Dell’Aniello S. Time-related biases in pharmacoepidemiology. Pharmacoepidemiol Drug Saf. 2020;29(9):1101-1110. doi:10.1002/pds.5083
  33. Levesque LE, Hanley JA, Kezouh A, Suissa S. Problem of immortal time bias in cohort studies: example using statins for preventing progression of diabetes. BMJ. 2010;340(mar12 1):b5087-b5087. doi:10.1136/bmj.b5087
  34. Suissa S. Immortal time bias in observational studies of drug effects. Pharmacoepidem Drug Safe. 2007;16(3):241-249. doi:10.1002/pds.1357
  35. Suissa S. Immortal Time Bias in Pharmacoepidemiology. American Journal of Epidemiology. 2008;167(4):492-499. doi:10.1093/aje/kwm324
  36. Arshad S, Kilgore P, Chaudhry ZS, et al. Treatment with hydroxychloroquine, azithromycin, and combination in patients hospitalized with COVID-19. International Journal of Infectious Diseases. 2020;97:396-403. doi:10.1016/j.ijid.2020.06.099
  37. Geleris J, Sun Y, Platt J, et al. Observational Study of Hydroxychloroquine in Hospitalized Patients with Covid-19. N Engl J Med. 2020;382(25):2411-2418. doi:10.1056/NEJMoa2012410
  38. Guaraldi G, Meschiari M, Cozzi-Lepri A, et al. Tocilizumab in patients with severe COVID-19: a retrospective cohort study. The Lancet Rheumatology. 2020;2(8):e474-e484. doi:10.1016/S2665-9913(20)30173-9
  39. Rosenberg ES, Dufort EM, Udo T, et al. Association of Treatment With Hydroxychloroquine or Azithromycin With In-Hospital Mortality in Patients With COVID-19 in New York State. JAMA. 2020;323(24):2493. doi:10.1001/jama.2020.8630
  40. Gautret P, Lagier JC, Parola P, et al. Clinical and microbiological effect of a combination of hydroxychloroquine and azithromycin in 80 COVID-19 patients with at least a six-day follow up: A pilot observational study. Travel Medicine and Infectious Disease. 2020;34:101663. doi:10.1016/j.tmaid.2020.101663
  41. Paccoud O, Tubach F, Baptiste A, et al. Compassionate Use of Hydroxychloroquine in Clinical Practice for Patients With Mild to Severe COVID-19 in a French University Hospital. Clinical Infectious Diseases. 2021;73(11):e4064-e4072. doi:10.1093/cid/ciaa791
  42. Hernán MA, Alonso A, Logan R, et al. Observational studies analyzed like randomized experiments: an application to postmenopausal hormone therapy and coronary heart disease. Epidemiology. 2008;19(6):766-779. doi:10.1097/EDE.0b013e3181875e61
  43. Ross ME, Kreider AR, Huang YS, Matone M, Rubin DM, Localio AR. Propensity Score Methods for Analyzing Observational Data Like Randomized Experiments: Challenges and Solutions for Rare Outcomes and Exposures. American Journal of Epidemiology. 2015;181(12):989-995. doi:10.1093/aje/kwu469
  44. Suissa S, Moodie EEM, Dell’Aniello S. Prevalent new-user cohort designs for comparative drug effect studies by time-conditional propensity scores. Pharmacoepidemiol Drug Saf. 2017;26(4):459-468. doi:10.1002/pds.4107
  45. Barda N, Dagan N, Ben-Shlomo Y, et al. Safety of the BNT162b2 mRNA Covid-19 Vaccine in a Nationwide Setting. N Engl J Med. 2021;385(12):1078-1090. doi:10.1056/NEJMoa2110475
  46. Kimberlin CL, Winterstein AG. Validity and reliability of measurement instruments used in research. American Journal of Health-System Pharmacy. 2008;65(23):2276-2284. doi:10.2146/ajhp070364
  47. Hartzema A, Schneeweiss S. Addressing misclassification in pharmacoepidemiologic studies. In: Hartzema A, Tilson H, Chen K, eds. Pharmacoepidemiology and Therapeutic Risk Management. Harvey Whitney Books; 2008.
  48. Greenland S. The effect of misclassification in the presence of covariates. American Journal of Epidemiology. 1980;112(4):564-569. doi:10.1093/oxfordjournals.aje.a113025
  49. Chen Q, Galfalvy H, Duan N. Effects of Disease Misclassification on Exposure–Disease Association. Am J Public Health. 2013;103(5):e67-e73. doi:10.2105/AJPH.2012.300995
  50. Winterstein AG, Kubilis P, Bird S, Cooper-DeHoff RM, Nichols GA, Delaney JA. Misclassification in assessment of diabetogenic risk using electronic health records. Pharmacoepidemiol Drug Saf. 2014;23(8):875-881. doi:10.1002/pds.3656
  51. Rothman K, Greenland S. Cohort Studies. In: Rothman K, Greenland S, Lash T, eds. Modern Epidemiology. 3rd ed. Lippincott Williams & Wilkins; 2008.
  52. Chubak J, Pocobelli G, Weiss NS. Tradeoffs between accuracy measures for electronic health care data algorithms. Journal of Clinical Epidemiology. 2012;65(3):343-349.e2. doi:10.1016/j.jclinepi.2011.09.002
  53. Jonsson Funk M, Landi SN. Misclassification in Administrative Claims Data: Quantifying the Impact on Treatment Effect Estimates. Curr Epidemiol Rep. 2014;1(4):175-185. doi:10.1007/s40471-014-0027-z
  54. Schneeweiss S, Glynn RJ, Tsai EH, Avorn J, Solomon DH. Adjusting for Unmeasured Confounders in Pharmacoepidemiologic Claims Data Using External Information: The Example of COX2 Inhibitors and Myocardial Infarction. Epidemiology. 2005;16(1):17-24. doi:10.1097/01.ede.0000147164.11879.b5
  55. Knol MJ, Janssen KJM, Donders ART, et al. Unpredictable bias when using the missing indicator method or complete case analysis for missing confounder values: an empirical example. Journal of Clinical Epidemiology. 2010;63(7):728-736. doi:10.1016/j.jclinepi.2009.08.028
  56. Janssen KJM, Donders ART, Harrell FE, et al. Missing covariate data in medical research: To impute is better than to ignore. Journal of Clinical Epidemiology. 2010;63(7):721-727. doi:10.1016/j.jclinepi.2009.12.008
  57. Stürmer T, Wyss R, Glynn RJ, Brookhart MA. Propensity scores for confounder adjustment when assessing the effects of medical interventions using nonexperimental study designs. J Intern Med. 2014;275(6):570-580. doi:10.1111/joim.12197
  58. Perkins NJ, Cole SR, Harel O, et al. Principled Approaches to Missing Data in Epidemiologic Studies. American Journal of Epidemiology. 2018;187(3):568-575. doi:10.1093/aje/kwx348
  59. Public Policy Committee, International Society of Pharmacoepidemiology. Guidelines for good pharmacoepidemiology practice (GPP): Guidelines for good pharmacoepidemiology practice. Pharmacoepidemiol Drug Saf. 2016;25(1):2-10. doi:10.1002/pds.3891
  60. Wang SV, Pinheiro S, Hua W, et al. STaRT-RWE: structured template for planning and reporting on the implementation of real world evidence studies. BMJ. 2021;372:m4856. doi:10.1136/bmj.m4856
  61. Langan SM, Schmidt SA, Wing K, et al. The reporting of studies conducted using observational routinely collected health data statement for pharmacoepidemiology (RECORD-PE). BMJ. Published online November 14, 2018:k3532. doi:10.1136/bmj.k3532
  62. Bajwah S, Wilcock A, Towers R, et al. Managing the supportive care needs of those affected by COVID-19. Eur Respir J. 2020;55(4):2000815. doi:10.1183/13993003.00815-2020
  63. Sanders JM, Monogue ML, Jodlowski TZ, Cutrell JB. Pharmacologic Treatments for Coronavirus Disease 2019 (COVID-19): A Review. JAMA. Published online April 13, 2020. doi:10.1001/jama.2020.6019
  64. Beigel J, Tomashek KM, Dodd LE. Remdesivir for the Treatment of Covid-19 — Preliminary Report. N Engl J Med. 2020;383(10):992-994. doi:10.1056/NEJMc2022236
  65. Pottegård A, Kurz X, Moore N, Christiansen CF, Klungel O. Considerations for pharmacoepidemiological analyses in the SARS‐CoV ‐2 pandemic. Pharmacoepidemiol Drug Saf. 2020;29(8):825-831. doi:10.1002/pds.5029
  66. Wells BJ, Nowacki AS, Chagin K, Kattan MW. Strategies for Handling Missing Data in Electronic Health Record Derived Data. eGEMs. 2013;1(3):7. doi:10.13063/2327-9214.1035
  67. Lin KJ, Glynn RJ, Singer DE, Murphy SN, Lii J, Schneeweiss S. Out-of-system Care and Recording of Patient Characteristics Critical for Comparative Effectiveness Research: Epidemiology. 2018;29(3):356-363. doi:10.1097/EDE.0000000000000794
  68. Greenland S, Finkle WD. A Critical Look at Methods for Handling Missing Covariates in Epidemiologic Regression Analyses. American Journal of Epidemiology. 1995;142(12):1255-1264. doi:10.1093/oxfordjournals.aje.a117592
  69. Eekhout I, de Boer RM, Twisk JWR, de Vet HCW, Heymans MW. Missing Data: A Systematic Review of How They Are Reported and Handled. Epidemiology. 2012;23(5):729-732. doi:10.1097/EDE.0b013e3182576cdb
  70. Haneuse S, Bogart A, Jazic I, et al. Learning About Missing Data Mechanisms in Electronic Health Records-based Research: A Survey-based Approach. Epidemiology. 2016;27(1):82-90. doi:10.1097/EDE.0000000000000393
  71. Carpenter JR, Smuk M. Missing data: A statistical framework for practice. Biometrical Journal. 2021;63(5):915-947. doi:10.1002/bimj.202000196
  72. Jakobsen JC, Gluud C, Wetterslev J, Winkel P. When and how should multiple imputation be used for handling missing data in randomised clinical trials – a practical guide with flowcharts. BMC Med Res Methodol. 2017;17(1):162. doi:10.1186/s12874-017-0442-1
  73. Clark TG, Altman DG. Developing a prognostic model in the presence of missing data. Journal of Clinical Epidemiology. 2003;56(1):28-37. doi:10.1016/S0895-4356(02)00539-5
  74. Dempster AP, Laird NM, Rubin DB. Maximum Likelihood from Incomplete Data Via the EM Algorithm. Journal of the Royal Statistical Society: Series B (Methodological). 1977;39(1):1-22. doi:10.1111/j.2517-6161.1977.tb01600.x
  75. Leyrat C, Carpenter JR, Bailly S, Williamson EJ. Common Methods for Handling Missing Data in Marginal Structural Models: What Works and Why. American Journal of Epidemiology. 2021;190(4):663-672. doi:10.1093/aje/kwaa225
  76. Lin KJ, Singer DE, Glynn RJ, et al. Prediction Score for Anticoagulation Control Quality Among Older Adults. JAHA. 2017;6(10). doi:10.1161/JAHA.117.006814
  77. Beck BR, Shin B, Choi Y, Park S, Kang K. Predicting commercially available antiviral drugs that may act on the novel coronavirus (SARS-CoV-2) through a drug-target interaction deep learning model. Computational and Structural Biotechnology Journal. 2020;18:784-790. doi:10.1016/j.csbj.2020.03.025
  78. Lin K, Schneeweiss S. Considerations for the analysis of longitudinal electronic health records linked to claims data to study the effectiveness and safety of drugs. Clin Pharmacol Ther. 2016;100(2):147-159. doi:10.1002/cpt.359
  79. Lin KJ, Singer DE, Glynn RJ, Murphy SN, Lii J, Schneeweiss S. Identifying Patients With High Data Completeness to Improve Validity of Comparative Effectiveness Research in Electronic Health Records Data. Clin Pharmacol Ther. 2018;103(5):899-905. doi:10.1002/cpt.861
  80. Lin KJ, Rosenthal G, Murphy S, et al. External Validation of an Algorithm to Identify Patients with High Data-Completeness in Electronic Health Records for Comparative Effectiveness Research. CLEP. 2020;Volume 12:133-141. doi:10.2147/CLEP.S232540
  81. Gerlovin H, Posner DC, Ho YL, et al. Pharmacoepidemiology, Machine Learning, and COVID-19: An Intent-to-Treat Analysis of Hydroxychloroquine, With or Without Azithromycin, and COVID-19 Outcomes Among Hospitalized US Veterans. American Journal of Epidemiology. 2021;190(11):2405-2419. doi:10.1093/aje/kwab183
  82. Sohn MW, Arnold N, Maynard C, Hynes DM. Accuracy and completeness of mortality data in the Department of Veterans Affairs. Popul Health Metrics. 2006;4(1):2. doi:10.1186/1478-7954-4-2