We’re sorry, something doesn't seem to be working properly.
Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.
A framework to assess patient-reported adverse outcomes arising during hospitalization
BMC Health Services Researchvolume 16, Article number: 357 (2016)
The Erratum to this article has been published in BMC Health Services Research 2017 17:563
The assessment of adverse events from a patient-centered view includes patient-reported adverse outcomes. An adverse outcome refers to any suboptimal outcome experienced by the patient; when adverse outcomes are identified through a patient interview these are called patient-reported adverse outcomes. An adverse event is an adverse outcome that is more likely due to the processes of medical care rather than to the mere progression of disease. In the context of a large-scale study assessing post-hospitalization adverse events, we developed a conceptual framework to assess patient-reported adverse outcomes (PRAOs). This methodological manuscript describes this conceptual framework.
The PRAO framework builds on a validated adverse event ascertainment method including three phases: Phase 1 involves an inquiry to ascertain the occurrence of any patient-reported adverse outcome. It is completed by a structured telephone interview to obtain details – from a patient perspective – on symptoms that developed and/or worsened after hospitalization. Phase 2 involves the classification of PRAOs by physicians not involved in the patient care. Physician-reviewers then rate the PRAOs using well-adopted scales to determine whether the occurrence was the natural progression of the underlying illness or due to medical care. When the PRAO is rated as “due to medical care”, it is then classified as an “adverse event”. Phase 3 involves the classification of adverse events as preventable or ameliorable.
Out of the 1347 patients contacted at 1-month post-discharge, 469 reported AOs and after reviewing 369 cases, 29 were classified as AEs. Observed agreement levels between raters were 87.3, 85.5, and 85.2 % respectively displaying a good agreement (k > 0.60).
The framework incorporates PRAOs as a way to identify cases that need to be evaluated for adverse events. Further validation of this framework is warrant with the final aim of implementation at larger scale. The implementation of this framework will enable clinicians, researchers and healthcare institutions to compare outcome rates across providers and over time.
Since the late 1990s when the Institute of Medicine released its influential report “To err is human”, there have been increased efforts to enhance patient safety and to quantify the rate of adverse events (AEs) . The reported rate of AEs for in-hospital patients ranges from 2.5 to 7.5 % [2, 3], although recent literature suggests that these rates may be a gross underestimation, with the actual rate ten times greater than previously reported . Two recent North American studies found an incidence of post-hospital discharge AEs ranging from 19 to 23 % [5–7]. The discrepancies in these numbers highlight the lack of a gold standard technique for determining and assessing AEs.
Indeed, various methods exist within the literature for the collection and ascertainment of AEs, including self [8–10] and incidence reporting, conduction of full retrospective patient record reviews, malpractice claims analyses and mortality and morbidity reviews [11, 12]. The problem with these methods is that they are time intensive, resource heavy, often identify different types of AEs and miss injuries occurring in the post discharge period.
There has been a movement within the literature to turn towards patients to help identify adverse events [13–18]. Zhu et al., demonstrated that patients recognize and accurately identify in-hospital adverse events . In addition, Weingart et al. showed that many patient identified events were not captured by traditional incident reporting systems . Furthermore, King et al.  discussed how the integration of patients’ perspectives facilitates the understanding of adverse events and suggest the inclusion of patients’ perspective as a complementary measuring tool.
Forster et al.  identified that one in five patients experienced an adverse event in the transition from hospital to home, otherwise known as the post discharge period, a period that is missed by the traditional methods of adverse event ascertainment. These studies show the necessity of involving patients to improve patient safety.
Forster et al.  developed a method where the patient identified potential adverse events. Patients were contacted within 2–5 weeks after discharged and were asked to identify any new or worsening conditions, or otherwise known as a patient reported adverse outcomes (PRAO), as well as health services utilized. Patients who identified a PRAO had a case summary prepared that was reviewed by two board certified internists, to determine if an adverse event occurred . In a follow up study, the authors assessed the reliability of the peer review process for the ascertainment of adverse events and concluded that for the best agreement, at least three similarly-trained reviewers are required .
This methodological paper describes the framework developed for the assessment of patient-reported adverse outcomes (PRAOs) occurring at the transition between the hospital and the community. The framework is an extension of the method developed by Forster et al. This modified method substitutes the medical chart review, for the review of patient cases that include clinical information from discharge summaries and the patient reports [6, 7, 16, 19–21].
The present study was nested in a large-scale randomized clinical trial (RCT). This RCT assessed the efficacy of a structured web-accessible electronic discharge summary being compared to usual care discharge summary dictations [22, 23]. The full details of this RCT has been described elsewhere . This study uses data collected during the RCT.
In this section we present the multi-phased method included in the framework overview.
The conceptual framework for assessing PRAOs was developed to aid in the understanding of concepts and to act as a guide for future researchers conducting similar research . Using an iterative process, we made modifications to the approach designed by Forster et al. [6, 7, 16, 20, 21] to enable the assessment of PRAO that could be performed using hospital discharge summaries. Figure 2 displays the framework, and its three phases of information including, Phase 1 describes the identification of PRAOs; Phase 2 includes two steps: 1) evaluating causation and severity of the PRAO. When the PRAO is likely to be related to medical care (omission or commission) then the PRAO is considered an adverse event; 2) using a modified 6-point scale to assess whether the AE was due to treatment/suboptimal medical care. ; Phase 3 classifies the identified AE as ameliorable, preventable or neither.
We have clarified the terminology used to describe the conceptual framework (Table 1).
The most important terms underlying the framework are: adverse outcomes, patient-reported adverse outcomes (PRAO), adverse events (AE), ameliorability, and preventability. Figure 1 is a schematic representation of how the concepts relate to one another. An adverse outcome is defined as any suboptimal outcome experienced by the patient, including a new or worsening symptom, an unanticipated visit to a health facility, or death [6, 7]. Adverse outcomes can be identified by medical record review or through a patient interview. If captured only through the latter, we define the outcomes as “patient reported adverse outcomes” or PRAOs. An adverse event is an adverse outcome that is more likely due to the processes of medical care rather than by the mere progression of disease, where medical care refers to all aspects of care including the commission or omission of required acts [6, 7]. An ameliorable adverse event is an outcome whose severity could have been substantially reduced if different actions or procedures had been performed or followed [6, 7]. A preventable adverse event, meanwhile, is an outcome that could have been potentially avoided.
Phase 1: PRAO identification
In our study [22, 23], PRAO identification occurs at one month post-hospitalization . A trained interviewer administers a brief structured interview over the telephone that takes approximately ten minutes to complete. The questionnaires (see Additional files 1, 2 and 3) gives the patient an opportunity to describe new or worsening health conditions as symptoms. We ask patients to describe: 1) timing of the onset of symptoms; 2) duration of symptoms; 3) severity of the symptoms (on a 10-point scale; 1 = not bothersome; 10 = bother a great deal). Patients also explain the actions taken to alleviate the symptoms, including their use of health services and report if and how the symptom limited their usual physical activities. The PRAO data generates a narrative interview summary. This summary is used in subsequent phases.
Phase 2: adverse event determination
The narrative interview summary is combined with information obtained from the electronic medical record (specifically, the transfer-of-care summaries, otherwise known as discharge summaries, related to the index hospital encounter and subsequent visits) [6, 7, 20] generating a case report. This case report is used by the physician raters to determine the cause of the adverse event. Two physicians review the report independently to determine whether the cause of the outcome was due to the natural progression of the patient’s health condition or to the medical care received. Our team of reviewers consists of three academic physicians; two Royal College certified internists and one College-certified family physician.
During this phase, the physicians perform their classification of causation based on a simple dichotomous choice (i.e. the outcome was likely caused by medical care or not). If the PRAO is rated by the reviewing physician as being likely to have been caused by medical care, it is then classified as an adverse event. Once a PRAO is classified as an AE, the physician reviewers rate their type. Type includes the following groups: adverse drug events, procedure-related injuries, nosocomial infections, care related falls, therapeutic errors, diagnostic errors, and other. Severity is measured in terms of the impact on the patient (how it affected the patient and how the patients felt and functioned) and in terms of the impact on health services used (e.g. visit to the emergency department).
Recognizing that a dichotomous assessment of relation of a PRAO to medical care may be difficult, we also presented reviewers with a more nuanced 6-point scale for judging the link of a PRAO to quality of medical care. The 6-point scale’s response options are:
No evidence that outcome was due to treatment and/or suboptimal medical care;
Little evidence that outcome was due to treatment and/or suboptimal medical care;
Outcome was possibly due to treatment and/or suboptimal medical care (50/50 chance) but was more likely due to disease;
Outcome was possibly due to treatment and/or suboptimal medical care (50/50 chance) and was more likely due to treatment and/or medical care than disease;
Outcome was probably due to treatment and/or suboptimal medical care;
Outcome was definitely due to treatment and/or suboptimal medical care.
Phase 3: preventability and ameliorability determinations
We then brought forward any cases with a rating of ≥5 on the above-mentioned 6-point scale to phase 3 for a final evaluation step using Additional file 4. In this final step, the physician reviewers were asked to classify whether the adverse event is preventable, ameliorable or neither, with definitions of these constructs as outlined earlier.
Ascertainment of agreement
Disagreements among reviewers can occur during phases 2 and 3 of this multi-phased process. In phase 2, when a disagreement between raters occurs, the raters can meet face-to-face to come to a consensus. If a disagreement persists, a third rater can act as a tiebreaker.
When using the 6-point scale as the primary tool for determining link of a PRAO to medical care, reviewers similarly need to review and discuss cases when their ratings are disparate. This is needed when one physician rates ≥5, while the other physician rates ≤3. If one physician rates ≥5 and the other rates 4, the higher rating is endorsed and this is considered an AE. Similarly, when one physician rates ≤3 while the other rates 4, the lower rating is endorsed and this is not considered an AE.
All patients admitted to a medical teaching unit in a Canadian tertiary hospital were invited to participate in the study . During a period of 18 months, 1347 patients were interviewed to report any adverse outcome that rose at one month after discharge from hospital. Of these patients 469 reported adverse outcomes, 369 cases were reviewed of which 29 were rated as adverse events by the raters with an overall adverse event rate of 7.9 %. Out of these, 11.5 % were preventable, 69.3 % were ameliorable and 19.2 % were classified as neither preventable nor ameliorable.
All 369 cases were reviewed by at least 2 reviewers. Reviewers 1, 2, and 3 each reviewed 266, 199, and 273 cases, respectively. Reviewers 1 and 2 rated a total of 96 cases between themselves; reviewers 2 and 3 rated 103 cases between themselves; and reviewers 1 and 3 rated a total of 170 cases between themselves. Table 2 shows the phase 2 agreements between raters prior to any re-rating efforts, with observed agreement levels of 65.3, 75.2, and 52.4 % respectively, and corresponding kappa (k) values of 0.30, 0.33 and 0.14, respectively. These kappa levels indicate only slight to fair agreement . After re-rating, the observed agreement levels were 87.3, 85.5, and 85.2 % respectively, and corresponding kappa (k) values of 0.62, 0.76 and 0.67 (p < 0,001) .
Table 3 shows the agreement between reviewers after re-rating occurred using the 6-point scale of the PRAO framework (Fig. 2). Reviewers 1 and 2 had a 94.8 % agreement with k = 0.82 (p = 0.001). Reviewers 2 and 3 had a 91.3 % agreement, k = 0.81 (p = 0.001) and reviewers 1 and 3 had a 91.8 % agreement, k = 0.74 (p = 0.001). There were 21 cases that reviewers had full agreement on after re-rating as being classified an AE due to medical care (i.e. both reviewers rated the PRAO between a 5 and 6), which indicates an initial adverse event rate of 5.7 %.
There were 30 instances in which reviewers still disagreed after meeting and reassessing cases (i.e. one reviewer rated that PRAO between 1 and 3, while the other rated it between 5 and 6). In such instances, a third reviewer assessed the case and this rating was taken as final (Table 4).
To further illustrate the PRAO rating process, we have provided real examples that can be found in Additional file 5 (a, b, c). These examples include full agreement on AE, agreement on not an AE, and an example of disagreement.
In this paper, we have refined and presented a conceptual and methodological framework for the assessment of PRAOs. The framework builds on a validated adverse event ascertainment method, with modifications to streamline the process by including only the discharge summary as well as the patient report [6, 7, 19, 20]. This method identifies PRAOs using a structured interview via the telephone, classifies the cause of the PRAOs and determines if it was due to medical care and whether the PRAO was ameliorable or preventable. The framework presents a potential tool to assess patient safety that can be used by other groups such as healthcare quality improvement management, and with wide spread adoption, it can help standardize the ascertainment of AEs.
The importance of using PRAO in the ascertainment of adverse events is highlighted in several studies [13–18, 26–28]. Traditionally it has been argued that patients lack the expertise to evaluate care received . However, patients’ experiences and perceptions are related to indicators such as access to medical care, patient-provider communication, and transitions of care [9, 27]. Positive patient experiences are linked to patient health status , while negative patient experiences have been linked to adverse health outcomes . In addition, patients are aware of adverse events and are often accurate in reporting when adverse events have occurred. It has also been found that patients report events that would have been missed using standard chart review techniques [14, 26, 27, 29, 30]. Utilizing PRAO can be a less costly method for the identification of adverse events as compared to the traditional methods. The novelty of this framework is the incorporation of PRAOs as a way to identify cases that needed to be evaluated for adverse events . To our knowledge, no framework has been developed describing the assessment of adverse events utilizing electronic transfer-of-care discharge summaries and PRAO.
Many different methods exist within the literature for the collection and ascertainment of AEs, but each method is faced with limitations [3, 10, 31–33]. Self-reporting of adverse events has been one common method utilized in institutions, although voluntary reporting often underreports the rate of AEs [10–12, 34]. Conducting full retrospective patient record reviews has been regarded by some groups as the gold standard for the assessment of AEs [33, 35]. This method typically consists of a two-phase chart review process, involving a research nurse and a physician rating adverse events [3, 10, 31–33]. This is an onerous process that requires a significant amount of time and resources, and relies on events having already occurred or a healthcare professional identifying an event . Furthermore, this method does not capture post-hospital AEs and excludes patients’ reports, which we know are invaluable and capture missed events [13, 14].
Adverse event ascertainment is often time and resource intensive, due to amount of documentation required from the chart. In order to streamline the adverse event ascertainment process, our framework proposes the use of hospital discharge summaries as a key source of information for PRAO assessments, supplemented by direct interviews with patients. Previous studies have shown that discharge summaries contain the necessary information required to assess adverse events . When Melton and Hripcsak  compared assessments of adverse events using full chart reviews versus discharge summaries, their results showed that the assessments were similar. Furthermore, some studies have shown the potential for the use of automated detection systems that are able to scan discharge summaries for adverse event triggers [36–40]. Electronic transfer-of-care summaries can provide a platform for this technology.
There are four limitations to this method of AE detection. Firstly, patients can report a large variation in the amount of outcomes, not all of which are related to adverse events. Previous studies have shown that approximately 50 % of patients will report an adverse occurrence to an interviewer after being discharged from the hospital, although the actual AE rate was 9 per 100 admissions . This high rate of PRAOs leads to a large volume of cases that need to be evaluated by reviewers, increasing the burden to raters. Faced with this challenge, the use of well-trained research assistants conducting patient follow up calls is a potential solution.
Secondly, as with any retrospective study utilizing an interview technique, there is a potential for recall bias. To minimize recall bias, interviewers provide patients with probes to assist recall along with possible responses that offer patients set time frames around which they can structure their answers [41–44]. For example, for symptom duration, patients can report less than one day, less than one week, less than two weeks, two weeks or more, or still occurring. By proposing a standard process for data collection and review, we hope that the effect of this bias will be minimized. If all institutions and researchers perform this work in the same way, then the impact of this recall bias will be consistent.
Novick suggests that the traditional bias against telephone interviews, low quality of data is not supported by robust evidence . This bias had been attributed to the lack of visual cues during telephone interviews that could potentially result in probing and interpretation of responses. However on the other hand, telephone-interviews may allow respondents to feel relaxed and able to disclose sensitive information [45–47].
Thirdly, a known limitation within the traditional methodology to ascertain AEs is the inter-rater reliability within the point scale [6, 16, 48]. It has been our observation that different raters have varying exposures to AEs resulting in each individual viewing a case differently. It is for this reason that we have suggested incorporating an “all-or-none” phenomenon, which leaves little room for disagreement. However, if a point scale is used, we suggest that reviewers are carefully chosen to have similar training, exposure to AEs, are from the same institution and that regular and frequent meetings are conducted to ensure all reviewers have a firm understanding of the definitions associated with AEs and what exactly is being rated. Furthermore, we suggest that reviewers undergo a training program similar to that used by Brennan and colleagues to help improve inter-rater reliability . While there is debate within the literature concerning the reliability of using one rater versus two or more, most studies suggest that more than one reviewer is required to agree on a case before the event can be classified as an AE [15, 48]. These are all factors that can be modified as our knowledge of optimal rating procedures improves.
Finally, we found in our study that in some instances reviewers had difficulty deciding between a 3 (the outcome was possibly due to treatment and/or suboptimal medical care (50/50 chance) but was likely due to disease) or a 4 (outcome was possibly due to treatment and/or suboptimal medical care (50/50 chance) and was more likely due to treatment and/or medical management than disease) response options on the 6-point scale. However, raters recognized that the inclusion of both options was important, as in ‘real world’ there are situations in which is difficult to differentiate between disease progression and treatment/medical management and the two options make raters to consider this difference. Perhaps, testing a modify version of this scale, compressing the scale to a 5-point scale (merging options 3 and 4) could be consider. Any such adjustment of the 6-point scale to a 5-point scale would require some retesting of the performance of the revised scale in inter-observer agreement.
Adverse events are an important patient safety issue and have become a focal point for research over the past 30 years. Despite the increased awareness surrounding AEs, recent studies have pointed to the disappointing numbers in AEs trends: rates have not been decreasing significantly [50, 51]. A caveat to these findings, however, is that standardized methodologies for the detection and measurement of AEs need further development and study. Our proposed framework can offer a standardized method that incorporates patients’ views in the assessment of adverse events. Future studies can focus on the validity and reliability of this framework. Widespread adoption of this tool can create a platform for future researchers to compare AE statistic following patient safety interventions.
The framework presented here is an approach to determining AEs that can be used as a template by clinicians, researchers and health care institutions. The data collected using this framework could help health care systems in their efforts to improve patient safety.
AE(s), adverse event(s); AO, adverse outcome; PRAO, patient –reported adverse outcome; RCT, randomized clinical trial.
Institute of Medicine. To Err is human: building a safer health system. Washington, DC: The National Academies Press; 1999.
Thomas EJ, Studdert DM, Newhouse JP, et al. Costs of medical injuries in Utah and Colorado. Inquiry. 1999;36:255–64.
Leape LL, Brennan TA, Laird N, et al. The nature of adverse events in hospitalized patients: results of the harvard medical practice study II. N Engl J Med. 1991;324:377–84.
Classen DC, Resar R, Griffin F, et al. ‘Global trigger tool’ shows that adverse events in hospitals may be ten times greater than previously measured. Health Aff (Millwood). 2011;30:581–9.
Baker D, Stevens C, Brook R. Patients who leave a public hospital emergency department without being seen by a physician. J Am Med Assoc. 1991;266:1085–90.
Forster AJ, Clark HD, Menard A, et al. Adverse events among medical patients after discharge from hospital. Can Med Assoc J. 2004;170:345–9.
Forster AJ, Murff HJ, Peterson JF, et al. The incidence and severity of adverse events affecting patients after discharge from the hospital. Ann Intern Med. 2003;138:161–74.
Bjertnaes O, Skudal KE, Iversen HH, Lindahl AK. Patient-reported incident in hospital instrument – PRIH. BMJ Qual Saf. 2013;22:743–51.
Kemp K, Warren S, Chan N, McCormack B, Santana M, Quan H. Qualitative complaints and their relation to overall hospital rating using an H-CAHPS-derived instrument. BMJ Qual Saf 2015 Oct 28. doi:10.1136/bmjqs-2015-004371.
Murff HJ, Patel VL, Hripcsak G, et al. Detecting adverse events for patient safety research: a review of current methodologies. J Biomed Inform. 2003;36:131–43.
World Health Organization. WHO Draft Guidelines for Adverse Event Reporting and Learning System. WHO Press. 2005.http://www.who.int/patientsafety/events/05/Reporting_Guidelines.pdf (accessed 27 Sep2014).
Olsen S, Neale G, Schwab K, et al. Hospital staff should use more than one method to detect adverse events and potential adverse events: incident reporting, pharmacist surveillance and local real-time record review may all have a place. Qual Saf Health Care. 2007;16:40–4.
Zhu J, Stuver SO, Epstein AM, et al. Can we rely on patients’ reports of adverse events? Med Care. 2011;49:948–55.
Weingart SN, Pagovich O, Sands DZ, et al. What can hospitalized patients tell us about adverse events? Learning from patient-reported incidents. J Gen Intern Med. 2005;20:830–6.
Weissman JS, Schneider EC, Weingart SN, et al. Comparing patient-reported hospital adverse events with medical record review: do patients know something that hospitals do not? Ann Intern Med. 2008;149:100–8.
Ward JK, Armitage G. Can patients report patient safety incidents in a hospital setting? A systematic review. BMJ Qual Saf. 2012;21(10):685–99.
Massó Guijarro P, Aranaz Andrés JM, Mira JJ, et al. Adverse events in hospitals: the patient’s point of view. Qual Saf Health Care. 2010;19(11):144–7.
King A, Daniels J, Lim J, et al. Time to listen: a review of methods to solicit patient reports of adverse events. Qual Saf Health Care. 2010;19:148–57.
Forster AJ, Taljaard M, Bennett C, et al. Reliability of the peer-review process for adverse event rating. PLoS One. 2012;7:e41239.
Forster AJ, Asmis TR, Clark HD, et al. Ottawa hospital patient saftey study: incidence and timing of adverse events in patients admitted to a Canadian teaching hospital. Can Med Assos J. 2004;170:1235–40.
Jabareen Y. Building a conceptual framework: philosophy, definitions, and procedure. 2009;49–62.
Okoniewska BM, Santana MJ, Holroyd-Leduc J, et al. The Seamless Transfer-of-Care Protocol: a randomized controlled trial assessing the efficacy of an electronic transfer-of-care communication tool. BMC Health Serv Res. 2012;12:414.
Santana MJ, Holroyd-Leduc J, Flemons WW, et al. The Seamless Transfer of Care: A Pilot Study Assessing the Usability of an Electronic Transfer of Care Communication Tool. Am J Med Qual Published Online First: 2013.
Michel P, Quenon JL, de Sarasqueta AM, et al. Comparison of three methods for estimating rates of adverse events and rates of preventable adverse events in acute care hospitals. BMJ. 2004;328:199.
Goldman RL. The reliability of peer assessments of quality of care. J Am Med Assoc. 1992;267:958–60.
Blendon RJ, Desroches CM, Brodie M, et al. Views of practicing physicians and the public on medical records. N Engl J Med. 2002;347:1933–40.
Davis K, Schoenbaum SC, Collins KS, et al. Room for improvement: patients report on the quality of their health care. 2002.
Hofer TP, Asch SM, Hayward RA, Rubenstein LV, Hogan MM, Adams J, Kerr EA. Profiling quality of care: Is there a role for peer review?. BMC Health Serv Res. 2004;4(1):1.
Weingart SN, Gandhi TK, Seger AC, et al. Patient-reported medication symptoms in primary care. Arch Intern Med. 2005;165:234–40.
Tsilimingras D, Bates DW. Addresing postdischarge adverse events: a neglected area. Jt Comm J Qual Patient Saf. 2008;34:85–97.
Thomas EJ, Studdert DM, Burstin HR, et al. Incidence and types of adverse events and negligent care in Utah and Colorado. Med Care. 2000;38:261–71.
Mills DH. Medical Insurance Feasibility Study. West J Med. 1978;128:360–5.
Hiatt H, Barnes B, Brennan T, et al. A study of medical injury and medical malpractice. N Engl J Med. 1989;321:480–4.
Cullen DJ, Sweitzer BJ, Bates DW, et al. Preventable adverse drug events in hospitalized patients: a comparative study of intensive care and general care units. Crit Care Med. 1997;25:1289–97.
Mendes W, Martins M, Rozenfeld S, et al. The assessment of adverse events in hospitals in Brazil. Int J Qual Heal care. 2009;21(31):279–84.
Murff HJ, Forster AJ, Peterson JF, et al. Electronically screening discharge summaries for adverse medical events. J Am Med Informatics Assoc. 2003;10:12–4.
Melton GB, Hripcsak G. Automated detection of adverse events using natural language processing of discharge summaries. J Am Med Informatics Assoc. 2005;12(33):448–58.
Forster AJ, Andrade J, Van Walraven C. Validation of a discharge summary term search method to detect adverse events. J Am Med Informatics Assoc. 2005;12:200–7.
Bates DW, Evans RS, Murff H, et al. Detecing adverse events using information technology. J Am Med Informatics Assoc. 2003;10:115–29.
Linder JA, Haas JS, Iyer A, et al. Secondary use of electronic health record data: spontaneous triggered adverse drug event reporting. Pharmacoepidemiol Drug Saf. 2010;19:1211–5.
Corwin RG, Krober M, Roth HP. Patients’ accuracy in reporting their past medical history, a study of 90 patients with peptic ulcer. J Chronic Dis. 1971;23:875–9.
Preston-Martin S, Bernstein L, Maldonado A, et al. A dental x-ray validation study. Am J Epidemiol. 1985;121:430–9.
Sudman S, Bradburn N. Response effects in surveys: a review and synthesis. Chicago: Aldine; 1974.
Feinstein A. Clinical Epidemiology: the architecture of clinical research. Philadelphia: W.B. Saunders Co; 1985.
Novick G. Is there a bias againt telephone interviews in qualitative research? Res Nurs Health. 2008;31(4):391–8.
Burke L, Miller M. Phone interviewing as a means of data collection: Lessons learned and practical recommendations [Online journal] Forum: Qualitative Social Research. 2001;2. Retrieved August 28, 2007 from http://www.qualitative-research.net/fqs-texte/2-01/2-01burkemiller-e.htm.
Burnard P. The telephone interview as a data collection method. Nurse Educ Today. 1994;14:67–72.
Zegers M, de Bruijne MC, Wagner C, et al. The inter-rater agreement of retrospective assessments of adverse events does not improve with two reviewers per patient record. J Clin Epidemiol. 2010;63:94–102.
Brennan TA, Localio RJ, Laird NL. Reliability and validity of judgments concerning adverse events suffered by hospitalized patients. Med Care. 1989;27:1148–58.
Shojania KG, Thomas EJ. Trends in adverse events over time: why are we not improving? BMJ Qual Saf. 2013;22:273–7.
Vincent C, Aylin P, Franklin BD, et al. Is health care getting safer? BMJ. 2008;337:a2426.
We would like to acknowledge the contributions of Chloe de Grood for her support with data collection.
Funding was received from the Canadian Institute for Health Research.
Availability of data and materials
The dataset supporting the conclusions of this article is presented in the manuscript.
BO participated in the data collection and analysis. She interpreted the data and drafted the manuscript. MJS participated in the data collection and analysis. She interpreted the data and drafted the manuscript. JHL was one of the three raters. She also contributed to the analysis and interpretation of the results and helped to draft the manuscript. WF was one of the three raters. He contributed to the analysis and interpretation of the results and helped to draft the manuscript. MO was one of the three raters. She also contributed to the analysis and interpretation of the results and helped to draft the manuscript. DW contributed to the study planning and design and helped to draft the manuscript. WO contributed to data collection and interpretation of results. GW contributed to the design of the study and interpretation of results. AJF contributed to the design of the study and interpretation of results, and helped to draft the manuscript. All authors read and approved the final manuscript.
The authors declare that they have no competing interests.
Consent for publication
Ethics approval and consent to participate
Ethics approval was obtained from the Ethics board at the University of Calgary. Written consent was obtained from all participants.
An erratum to this article is available at https://doi.org/10.1186/s12913-017-2392-z.
Patient Reported Adverse Outcome Questionnaire. (DOC 25 kb)
Health Services Utilization Questionnaire. (DOCX 56 kb)
Patient Reported Outcome (PRAO) Assessment. (DOCX 53 kb)
Type of Adverse Event. (DOCX 61 kb)
Cases reported. (DOCX 989 kb)