Impact of referral templates on the quality of referrals from primary to secondary care: a cluster randomised trial
© Wåhlberg et al. 2015
Received: 21 October 2014
Accepted: 21 August 2015
Published: 29 August 2015
The referral letter is an important document facilitating the transfer of care from a general practitioner (GP) to secondary care. Hospital doctors have often criticised the quality and content of referral letters, and the effectiveness of improvement efforts remains uncertain.
A cluster randomised trial was conducted using referral templates for patients in four diagnostic groups: dyspepsia, suspected colorectal cancer, chest pain and chronic obstructive pulmonary disease. The GP surgery was the unit of randomisation. Of the 14 surgeries served by the University Hospital of North Norway Harstad, seven were randomised to the intervention group. Intervention GPs used referral templates soliciting core clinical information when initiating a new referral in one of the four clinical areas. Intermittent surgery visits by study personnel were also carried out. A total of 500 patients were included, with 281 in the intervention and 219 in the control arm. Referral quality scoring was performed by three blinded raters. Data were analysed using multi-level regression modelling. All analyses were conducted on intention-to-treat basis.
In the final multilevel model, referrals in the intervention group scored 18 % higher (95 % CI (11 %, 25 %), p < 0.001) on the referral quality score than the control group. The model also showed that board certified GPs and GPs in larger surgeries produced referrals of significantly higher quality.
In this study, the dissemination of referral templates coupled with intermittent surgery visits produced higher quality referrals.
This trial has been registered at ClinicalTrials.gov. The trial registration number is NCT01470963.
A referral facilitates the transition of care from a general practitioner (GP) to secondary care to establish a diagnosis, to provide treatment including surgery, and to offer advice or reassurance. Hospital specialists frequently have complained about the perceived quality of referral letters. Several studies have highlighted the varying quality and content of referrals across a range of clinical specialities [1–12]. A recent Canadian survey of more than 3000 GPs and specialists found that, among the main problems specialists identified, 51 % of referral letters had an unclear reason for referral . This variation in quality makes the evaluation and prioritisation of incoming referrals difficult, with one author stating that prioritisation cannot be performed based on referral letters alone .
A high quality referral process will generally involve referral letters containing all necessary information in a context of shared understanding between GPs, patients and hospital staff . There have been previous definitions of what referral letters should contain . In Norway, the Norwegian Centre for Informatics in Health and Social Care (KITH) has developed guidelines governing the content of electronic referral and discharge letters [17, 18]. However, these guidelines present headings and content categories, but do not specify the precise clinical information required for different clinical areas. A recent Norwegian study has highlighted that it is the lack of information in the referrals, rather than the structure and categories of the referral, that hospital doctors perceive as a barrier to high quality cooperative care . To measure referral quality, it therefore seems necessary to focus more on measuring the informational quality of referrals than on measuring their structure. This conclusion is echoed by other publications in the field [8, 12, 20]. Several of these studies developed scoring systems collaboratively between hospital doctors and GPs [12, 20].
The introduction of electronic health records and communication has, to some extent, eliminated some of the structural problems with referral letters, but further work is needed to elucidate the relationship between the quality of clinical information in referrals and high quality health care processes. This is important as healthcare costs are rising globally  and services are being delivered within a framework of increasingly limited resources. In this context, it is imperative to improve patient prioritisation based on referrals in order to aid swift diagnosis in those with more serious disease and to provide evidence based high quality care to each individual patient. Tools to improve referral quality are paramount.
This paper reports the effects of a referral intervention on the quality of referrals in a cluster randomised trial. We hypothesized that the referral intervention would improve informational quality in the referrals. We assess whether other GP-related factors, including patient list size and years of experience, affect the quality of referrals written. This paper is part of a larger study assessing the effect of a referral intervention on the quality of health care delivered to individual patients. Information about further assessments within the referral project is available in the published methods paper .
The Norwegian health care system is relatively uniform throughout the country. Each person has a regular GP who acts as a gatekeeper to secondary care . GPs work either privately, with capitation payment and fee-for-service reimbursement, or as municipality employees. Specialist health care is delivered through governmentally owned regional health authorities, mainly in public hospitals. Some specialist outpatient care is purchased by the regional health authorities from private specialists, but access to this is very limited in the geographical area of the current study. Electronic health records are almost ubiquitous and referrals are sent according to a national standard that automatically includes demographic information including address, contact details and GP details .
This study was designed as a cluster randomised trial with the general practitioner surgery as the clustering unit. All 14 community GP surgeries in the area served by the University Hospital of North Norway (UNN) Harstad were randomised to the intervention or control group. The cluster design was chosen to avoid contamination between GPs, which could have occurred if individual GPs at the same surgery were randomised to different groups.
The referring GP could not be blinded because the intervention was actively used by the GP. Patients, hospital doctors and outcome evaluators were blinded to the patient’s intervention status. However, in some cases the referral letter revealed the intervention status. Further information about study methods are available in detail in the methods paper .
Referral template for patients with suspected colorectal cancer
Change in bowel habit
Blood in stool
Family history of colorectal cancer
Previous medical history of bowel disease or results of previous bowel investigations
Results of digital rectal examination (DRE)
Iron deficiency anaemia
Clinical findings at abdominal examination
Result of faecal occult blood test (FOBT)
The general practitioners clinical suspicion
The templates were distributed by the corresponding author (HW) during educational and/or lunch meetings at the intervention surgeries. Prior to the distribution of templates, the project had been presented to the GPs at similar meetings. The intervention was in use for approximately 2 years, from September 2011 to November 2013.
Additional follow-up was provided in the form of lunchtime visits to the intervention surgeries approximately twice yearly and intermittent mail leaflets and reminders. The lunchtime visits were performed by HW and provided information about the progress of the study, reminders to use the intervention templates and answers to questions about the project. In addition, personal letters to participating doctors were sent when it was evident that the intervention had not been used in a received referral.
In the control group, normal referral practice continued. No information about the study was provided to the control surgeries.
patients referred with dyspepsia
patients referred with suspected colorectal cancer (CRC)
patients referred with chest pain
patients referred with confirmed or suspected chronic obstructive pulmonary disease (COPD)
At UNN Harstad in 2008 these diagnostic areas accounted for approximately 26 % of all patients in the medical outpatient clinics (own data), although separating new referrals from control patients in this material was not possible. In addition, patients in these clinical areas often represent a diagnostic challenge in primary care  and are well suited for simple referral templates.
The 14 GP surgeries in the area primarily served by UNN Harstad were included in the randomisation process. In 2013, these surgeries had a total list size of 39,253 patients. Five surgeries were town-based and nine were rural. To ensure equal sociodemographic backgrounds between groups, the surgeries were randomised stratified by town or countryside location, although the location of the surgery itself was not expected to influence the main outcome variables. Two centres initially randomised to the intervention group declined to participate, one because of lack of interest and one because the GP was about to retire. Two further centres were therefore randomly selected. The final intervention group consisted of three urban and four rural surgeries, with two urban and five rural surgeries in the control group.
New patients referred to the UNN Harstad medical outpatient clinics in any of the four diagnostic groups received written information about the study and a participant consent form along with their appointment letter. Patients were orally reminded of the study by the hospital doctor at their hospital outpatient appointment and were given a new consent form if appropriate. Children (<18 years of age) and patients with reduced capacity to consent were excluded from the project. Further details about the GP surgeries and the recruitment process are published in the methods paper .
The study followed the directions in the Helsinki Declaration. Before recruitment started, the study was presented to the Regional Ethical Committee for Medical Research in North-Norway, who determined it not to be within the scope of the Health Research Act (REK NORD 2010/2259). The study was approved by the Data Protection Official for Research. The study is registered at ClinicalTrials.gov. The trial registration number is NCT01470963. All patients provided written informed consent.
For the overall study, sample size calculation was performed for the main outcome variable (a health care quality score), as shown in the methods paper . No specific sample size calculation was performed for the referral quality outcome reported in this paper.
The referrals were rated according to a scoring system derived directly from the referral templates used. These templates specified the clinical information that specialists deemed most important in the referral, based on literature documented above. One point was awarded for the presence in the referral of each of the clinical details requested in the referral template. The maximum score possible for referrals for dyspepsia, suspected CRC, COPD and chest pain were 17, 10, 15 and 13 respectively. Other studies have also awarded points for the presence of core information including full contact details and legibility of referrals . Because all but six referrals in this project were electronic, and such information is automatically included in the electronic referral, this information was not included in the scoring. The final referral score therefore represents how many of the information points in the template were actually articulated in the referral. Each score was then transformed to a percentage value.
Three raters scored the referrals. A sample of 100 out of the 500 referrals was scored independently by two raters. No referral was scored by all three raters, and all three rater pairs shared at least 25 referrals. The raters were blinded to the intervention status of the referring GP.
For further analysis, a small amount of data was imputed. For referrals initiated by interns, the list size was missing by default. Interns in Norway spend six months attached to a GP surgery where they do not have their own list of patients. Instead, they take part in the general workload of the surgery. For each referral initiated by an intern, the list size value in the dataset was set to the average list size value for the surgery the intern was attached to. This was believed to best represent the anticipated workload of each intern.
Speciality status was available for all referring GPs. Years of experience as a GP and in hospitals was available for 499 of 500 referrals, while data for the last case were imputed as mean values.
Analyses were stratified by intervention group and control group. To ensure rater consistency an interrater reliability analysis using the weighted Kappa statistic was performed, as developed by Cohen . Referral scores were divided into centiles, and a weighted Kappa analysis was performed using quadratic weights. Additional analysis was carried out for each rater pair separately and with the data divided into quintiles to ensure consistency of analysis.
We chose to use standard weights (quadratic) to improve interpretability, in concordance with discussion on the appropriate use of Kappa analysis . In this way, the weighted Kappa coefficient approximates the intraclass correlation coefficient . The weighted Kappa coefficient increases with increasing numbers of categories, especially when using quadratic weights . However, this increase seems more pronounced in the range of two to five categories.
The cluster randomised design necessitates an analysis that is suitable for clustering. In this project, multi-level regression modelling was used. A stepwise approach was used to build the multi-level model. Likelihood ratio tests were used to evaluate whether random regression coefficients should be considered. Continuous variables were centred to facilitate interpretation. Because the intervention was randomised at the level of the GP surgery, no slope could be added for the intervention effect. To assess the addition of confounders to level one of the model, a change in the magnitude of the regression coefficient for the intervention effect of more than 10 % was considered indicative of a confound. In addition, the following variables were included based on prior subject knowledge: whether the referring GP was a specialist, the length of GP experience and GP list size. Effect modification was checked for relevant variables using p < 0.10 as the significance level. This level was chosen because the power to detect relevant interactions is often low, especially concerning cross-level interactions in multilevel studies . Although increasing the type 1 error rate has been shown to be ineffective , it was judged to be better than missing important interactions. Analysis was done on an intention-to-treat basis, as recommended . All referrals from intervention surgeries were therefore analysed as if they had used the referral intervention, even though it may have been evident that the intervention was not used. Stata version 13 (StataCorp. 2013. Stata Statistical Software: Release 13. College Station, TX: StataCorp LP) was used for all analysis.
Selected patient baseline characteristics by intervention status
Female/male, n (%)
166 (59.07)/115 (40.93)
127 (57.99)/92 (42.01)
p = 0.807
59.21 ± 13.64
57.08 ± 15.26
p = 0.101
Urban/rural, n (%)
169 (60.14)/112 (39.86)
121 (55.25)/98 (44.75)
p = 0.272
Clinical group, n (%)
- suspected colonic malignancy
- chest pain
- chronic obstructive pulmonary disease
Hospital appointment with senior house officer/specialist, n (%)
130 (46.26)/151 (53.74)
96 (43.84)/123 (56.16)
p = 0.588
Selected general practitioner (GP) baseline characteristics by intervention status1
GP surgery variables2
830.79 ± 208.78
865.48 ± 100.69
p = 0.475
Female/male GP, n (%)
14 (58.33)/10 (41.67)
10 (43.48)/13 (56.52)
p = 0.308
Specialist yes/no, n (%)
18 (75)/6 (25)
11 (47.83)/12 (52.17)
p = 0.055
16.02 ± 10.40
15.15 ± 11.15
p = 0.784
Years experience in hospital
2.81 ± 5.94
1.89 ± 3.06
p = 0.510
Number of GPs in surgery
4.33 ± 1.61
4.04 ± 1.58
GP referral variables per referral in dataset2
Female/male referring GP, n (%)
182 (64.77)/99 (35.23)
93 (42.47)/126 (57.53)
p < 0.000014
Number of GPs in surgery
4.43 ± 1.46
4.01 ± 1.62
Specialist yes/no n (%)
189 (67.26)/92 (32.74)
p = 0.0005564
16.21 ± 11.96
15.41 ± 11.70
p = 0.456
Years experience in hospital
1.54 ± 1.88
1.54 ± 2.46
p = 0.994
Other variables per referral in dataset2
Electronic/paper referral n (%)
281 (100)/0 (0)
213 (97.26)/ 6 (2.74)
p = 0.0053
The interrater reliability was found to be Kappa = 0.93 (p < 0.0001), 95 % CI (0.73, 1). Additional analysis with the data divided into quintiles showed Kappa = 0.90 (p < 0.0001), 95 % CI (0.71, 1). Analysis for each rater pair separately yielded Kappa values ranging from 0.85 to 0.93 (further details available upon request).
Average referral quality by diagnostic group, uncorrected for clustering1
57.3 (53.0, 61.7)
31.4 (28.7, 34.1)
Suspected colonic malignancy
70.1 (65.6, 74.6)
53.4 (48.5, 58.3)
43.4 (7.1, 79.6)
40.0 (0, 83.3)
61.9 (56.2, 67.5)
45.3 (38.3, 52.3)
Baseline evaluation of the main outcome variable (referral quality) demonstrated that it was nearly normally distributed. Naïve analysis of data—that is, using a mixed models approach without adding the level of GP surgery into the analysis—was compared with a model including the GP surgery as clustering unit. Adding the random intercept to the model decreased the −2 log likelihood by 4529.25–4493.50 = 35.75. This is highly significant according to the Chi squared distribution with one degree of freedom. From the above analysis, the intraclass correlation coefficient (ICC) was calculated to be 0.14 (95 % CI (0.02, 0.25)). The final model corrected for whether the GP was a board certified specialist, centred mean GP patient list size, GP hospital experience (in years), and categorised GP surgery size. GP experience (in years) was removed from the model because it had no impact on the outcome of interest and had no clear association with referral quality. In addition, GP experience was clearly correlated with being a GP specialist, and its inclusion would thus reduce the power of the analysis without adding further insight.
Allowing the result to vary randomly at the level of the referring GP further decreased the −2 log likelihood of the baseline model to 4419.70 and reduced residual variance. The addition of a third level added complexity to the model and only changed the estimation of the intervention effect by 2 percentage points. It was therefore decided to keep the two level model proposed in the methods paper .
Intervention effect estimates
95 % CI
p < 0.001
p < 0.001
- GP specialist yes/no
p < 0.001
- GP list size (centred)
p = 0.196
- GP experience in hospital (in years)
p < 0.001
- GP surgery size3
p = 0.211
Because only roughly 50 % of the intervention group referrals were sent to the intervention hospital electronic address, we performed a non-protocol multilevel model analysis comparing the quality of referrals between the intervention GPs who used the referral address and those intervention GPs who did not. We found a referral quality difference that was approximately as large as between the intervention and control group in the main analysis (21.9 %, 95 % CI (16.5, 26.2), p < 0.001).
The current paper presents the impact of the dissemination of a referral template on referral quality. The intervention improved referral quality by 18 %, which is presumably clinically relevant. Our finding is consistent with reported increases in referral quality in similar intervention studies [41, 42], whereas another study reported a smaller effect size . However, as discussed in a Cochrane review on the subject, there have been few studies aimed at improving the quality of referrals, and several have had methodological weaknesses .
The current study suggests that dissemination of referral templates combined with local follow-up measures can indeed improve referral quality in the communication between primary and specialist health care, which is consistent with the conclusions in the Cochrane report . The data also suggest that being a board certified GP improves the quality of written communication to specialist health services as measured by the referral scoring system. Surprisingly, a GP’s experience as a hospital doctor does not appear to predict referrals that include more of the content requested by hospital consultants. Years of experience as a GP showed no association with the outcome of interest and was left out of the model entirely. This may suggest that it is the communication and collaboration between hospital-based specialists and dedicated GPs that can produce better referral quality, and not the experience of the GP or the presence of a referral template per se.
However, it is important to recognise that while a hospital physician will try to prioritise received referrals based on the risk of serious illness, many referrals are sent for reasons other than ruling out or diagnosing disease. These can include patient reassurance, reduction of medico-legal risk, handing over of care, or to obtain a second opinion . Others have shown that factors including GP gender and GP speciality can affect referral rates and have discussed whether higher professional insecurity and/or higher responsiveness to patient demand may explain some of this variation . It is conceivable that some of these factors also affect referral content, and that referrals are not purely based on the GP’s perception of the patient’s individual risk of serious disease.
As shown above, in this study the quality of the referrals varied between the four clinical areas, with referrals for suspected colonic malignancy scoring highest (average 70.1 % in the intervention group and 53.4 % in the control group) and referrals for patients with COPD scoring lowest (43.3 vs. 40.0 %). We have found no comparable studies in which referrals for different clinical areas have been scored using the same scoring technique, and it is therefore hard to assess whether this quality difference is a general phenomenon. However, the referral template for colonic malignancy contained the fewest requested clinical data points, and the scientific basis for these points was better documented than those for the COPD or dyspepsia referrals. This may suggest that for referral templates to be effective, simpler templates based on solid scientific research may be more acceptable and user-friendly in clinical practice than complicated templates.
The referral scores varied widely for each diagnostic area (Fig. 2). This confirms the pre-trial clinical suspicion of variation, which was one of the motivations for this study. Especially within the area of dyspepsia, wide variation was seen in both the intervention and control groups, with several outliers. This suggests that some GPs produce referrals of high quality, regardless of the referral intervention, and that some general practitioners in the intervention group took no interest in the intervention at all. This wide range in performance has also been noted when the referral rate has been assessed [46, 47]. Although uniformity does not necessarily equate to quality, it is intuitive that some degree of increased uniformity in referral quality would improve equity in the health care delivered to patients.
Adding interactions to the model showed a significant interaction between intervention status and being a board certified GP. This suggested a stronger intervention effect amongst those who were board certified. This was felt to be adequately represented in the model by the combination of the terms ‘board certified GP’ and ‘hospital experience’.
The weighted Kappa analysis equates to ‘almost perfect’ agreement among raters (Kappa 0.81–1.00) according to Landis and Koch . Even considering the increasing Kappa values with increasing categories discussed above, this shows not only excellent overall reliability, but also excellent agreement between all three rater pairs.
This study has several limitations. The referral templates used in the project were developed according to international literature and local practices. Referrals were scored based on how closely they followed this referral template. Conscientious GPs were therefore likely to score very high on the referral score, and this could bias the results in favour of the intervention. Nonetheless, the scoring system does equate with referral quality measurement scores used in other referral evaluation studies [2, 5]. In addition, further work in this project aimes to assess whether the presence of a greater quantity of relevant clinical information improves the quality of the health care process, and consequently this scoring system seemed appropriate. It is possible that some of the effect size noted above was caused by GPs who took a special interest in the study.
A further weakness is that the current project does not provide a clear indication of the proportion of the referring GPs who actively used the referral intervention. The referral templates were distributed and follow-up visits were arranged to ensure adherence to the study protocol. As presented above, only about 50 % of the referrals from intervention GPs were sent to the newly formed intervention electronic address. This suggests a fairly modest uptake of the intervention, and the non-protocol analysis showed a higher intervention effect amongst those GPs who actively utilised the electronic address. Because intention-to-treat analysis was used, this has probably attenuated the intervention effect. Similar difficulties have been seen in other projects, with an uptake as low as 18 % in a referral intervention study for patients with lower bowel symptoms . However, a recent Norwegian project using referral guidance as an electronic pop-up reported that the 88 % of the intervention GPs used the intervention ‘all the time’ .
Many known barriers to changes of behaviour and application of clinical knowledge exist, including lack of knowledge/awareness, lack of applicability to the individual patient and organisational factors . Feedback from the GPs in this project suggests that the intervention was used and appreciated, but also easily forgotten in hectic everyday clinical work. If wider application of referral guidelines is to be considered, careful assessment should be undertaken to identify barriers to their use and to indicate tailored interventions to overcome these barriers, as this has been shown to be more likely to improve professional practice .
Another limiting factor is that the rate of inclusion of relevant patients is unknown. During the inclusion phase, regular reviews took place to assess the rate of inclusion at the hospital, which was estimated to be approximately 60 % of possible outpatient candidates. A completely accurate figure is not available, as this would require a manual search of the charts of every patient with an outpatient appointment, which is beyond the ethical approval of this project.
It is also clear that more patients were recruited from intervention GP surgeries than control GP surgeries. The total number of listed patients in the intervention and control group GP surgeries was very similar (19,347 vs. 19,906). The study did not have access to referral rates, and it is not clear whether these varied between the practices. There is no clear indication of major baseline differences between the GP surgeries and the study patients that can explain the difference in inclusion. One possible explanation is that the focus on the four diagnostic areas in the intervention offices caused more patients to be referred, but this cannot be demonstrated from the current data.
It is important to note that making referral information more in line with the hospital physicians’ wishes does not automatically predict improved outcomes. It is conceivable that referrals that are more pleasing to the hospital consultant may give a false sense of precision in the evaluation and prioritisation of referrals. For colorectal cancer, a review of symptoms and diagnostic tests in primary care suggests that few symptoms and signs are sensitive and specific enough to be used to identify patients at higher risk. However, it does indicate that referral guidelines and symptom combinations may aid in this process . Further research is necessary to identify the clinical symptoms, signs or tests that will allow a clear prediction of risk in order to guide the information included in referrals. As we wait for this guidance, we must use the tools currently available, including good communication and clear and informative referrals.
Healthcare costs are rising . For healthcare managers and policymakers, it would be helpful if the implementation of referral guidelines can improve patient prioritisation, as suggested in a recent Norwegian report supporting the use of referral guidelines and their implementation in the electronic health record . As discussed, caution must be exercised, because a more precise referral may not predict better quality of care. Further analysis within this project is currently underway to determine whether improved referral quality results in a meaningful change in patient prioritisation and quality of care.
This cluster randomised study assessing the impact of the dissemination of referral templates coupled with intermittent surgery visits by study personnel demonstrates a significant and substantial improvement in the measured quality of referrals in the intervention group. Further analysis is underway to determine whether this improvement in observed referral quality will predict an increase in the quality of care delivered to individual patients. For future studies, it appears prudent to utilise simple referral guidance, developed in collaboration between primary and secondary care. The referral guidance will need to be embedded in the patient record system to ensure its implementation.
We thank Øyvind Isaksen, Astrid Gjervold Lunde and Silje Lehne Michalsen for scoring the referrals. The study is funded by the Northern Norway Regional Health Authority. The funding source did not have any role in designing the study or writing the manuscript.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Garasen H, Johnsen R. The quality of communication about older patients between hospital physicians and general practitioners: a panel study assessment. BMC Health Serv Res. 2007;7:133.View ArticlePubMedPubMed CentralGoogle Scholar
- Tuomisto LE, Erhola M, Kaila M, Brander PE, Kauppinen R, Puolijoki H, et al. The Finnish national asthma programme: communication in asthma care--quality assessment of asthma referral letters. J Eval Clin Pract. 2007;13:50–4.View ArticlePubMedGoogle Scholar
- Ong SP, Lim LT, Barnsley L, Read R. General practitioners’ referral letters--Do they meet the expectations of gastroenterologists and rheumatologists? Aust Fam Physician. 2006;35:920–2.PubMedGoogle Scholar
- Dentith GE, Wilson KE, Dorman M, Girdler NM. An audit of patient referrals to the sedation department of Newcastle Dental Hospital. Prim Dent Care. 2010;17:85–91.View ArticlePubMedGoogle Scholar
- Syed AA, Large DM. Quality of GPs’ referral letters to diabetes secondary care. Practical Diabetes International. 2003;20:165–9.View ArticleGoogle Scholar
- Newton J, Hutchinson A, Hayes V, Mccoll E, Mackee I, Holland C. Do clinicians tell each other enough? An analysis of referral communications in two specialties. Fam Pract. 1994;11:15–20.View ArticlePubMedGoogle Scholar
- Shaw I, Smith KM, Middleton H, Woodward L. A letter of consequence: referral letters from general practitioners to secondary mental health services. Qual Health Res. 2005;15:116–28.View ArticlePubMedGoogle Scholar
- Bodek S, Ghori K, Edelstein M, Reed A, MacFadyen RJ. Contemporary referral of patients from community care to cardiology lack diagnostic and clinical detail. Int J Clin Pract. 2006;60:595–601.View ArticlePubMedGoogle Scholar
- Grol R, Rooijackers-Lemmers N, van Kaathoven L, Wollersheim H, Mokkink H. Communication at the interface: do better referral letters produce better consultant replies? Br J Gen Pract. 2003;53:217–9.PubMedPubMed CentralGoogle Scholar
- Westerman RF, Hull FM, Bezemer PD, Gort G. A study of communication between general practitioners and specialists. Br J Gen Pract. 1990;40:445–9.PubMedPubMed CentralGoogle Scholar
- Cheng J, Beltran-Agullo L, Trope GE, Buys YM. Assessment of the quality of glaucoma referral letters based on a survey of glaucoma specialists and a glaucoma guideline. Ophthalmology. 2014;121:126–33.View ArticlePubMedGoogle Scholar
- Jiwa M, Coleman M, McKinley RK. Measuring the quality of referral letters about patients with upper gastrointestinal symptoms. Postgrad Med J. 2005;81:467–9.View ArticlePubMedPubMed CentralGoogle Scholar
- Canadian Medical Association. National Survey: experiences with referrals from primary to specialty care. Ottawa, Ontario: Canadian Medical Association; 2013.Google Scholar
- Patel NN, D’Souza J, Rocker M, Townsend E, Morris-Stiff G, Manimaran M, et al. Prioritisation of vascular outpatient appointments cannot be based on referral letters alone. Surgeon. 2008;6:140–3.View ArticlePubMedGoogle Scholar
- Goodwin N, Dixon A, Poole T, Raleigh V. Improving the quality of care in general practice. London: The King's Fund; 2011Google Scholar
- Newton J, Eccles M, Hutchinson A. Communication between general practitioners and consultants: what should their letters contain? BMJ. 1992;304:821–4.View ArticlePubMedPubMed CentralGoogle Scholar
- Ree AO: Medisinsk faglig innhold i henvisninger. Trondheim: The Norwegian Direcorate for Health; 2003.Google Scholar
- Ree AO: Medisinsk faglig innhold i epikriser. Trondheim: The Norwegian Direcorate for Health; 2002.Google Scholar
- Martinussen PE. Referral quality and the cooperation between hospital physicians and general practice: The role of physician and primary care factors. Scand J Public Health. 2013;41:874.View ArticlePubMedGoogle Scholar
- Rokstad IS, Rokstad KS, Holmen S, Lehmann S, Assmus J. Electronic optional guidelines as a tool to improve the process of referring patients to specialized care: an intervention study. Scand J Prim Health Care. 2013;31:166–71.View ArticlePubMedPubMed CentralGoogle Scholar
- OECD. What Future for Health Spending? OECD Economics Department Policy Notes, No. 19 June 2013; 2013.Google Scholar
- Wahlberg H, Valle PC, Malm S, Broderstad AR. Practical health co-operation - the impact of a referral template on quality of care and health care co-operation: study protocol for a cluster randomized controlled trial. Trials. 2013;14:7.View ArticlePubMedPubMed CentralGoogle Scholar
- Carlsen B, Norheim OF. Changes in Norwegian physicians’ perception of their gatekeeper role. Scand J Prim Health Care. 2003;21:209–13.View ArticlePubMedGoogle Scholar
- Jiwa M, Mathers N, Walters S. Quality of information on referrals to colorectal surgeons: towards consensus. Curr Med Res Opin. 2002;18:72–7.View ArticlePubMedGoogle Scholar
- Nygaard E, Kårikstad V. Prioriteringsveileder - Fordøyelsessykdommer. Oslo: Helsedirektoratet; 2009.Google Scholar
- Nygaard E, Kårikstad V. Prioriteringsveileder - Lungesykdommer. Oslo: Helsedirektoratet; 2008.Google Scholar
- Nygaard E, Kårikstad V. Prioriteringsveileder - Hjertemedisinske sykdommer. Oslo: Helsedirektoratet; 2009.Google Scholar
- British Thoracic Society Standards of Care Committee. BTS statement on criteria for specialist referral, admission, discharge and follow-up for adults with respiratory disease. Thorax. 2008;63:i1–16.View ArticleGoogle Scholar
- Snow V, Barry P, Fihn SD, Gibbons RJ, Owens DK, Williams SV, et al. Evaluation of primary care patients with chronic stable Angina: guidelines from the American College of physicians. Ann Intern Med. 2004;141:57–64.View ArticlePubMedGoogle Scholar
- Malm S, Schirmer H. Hjertesak: Riktigere prioritering av pasienter! Fastlegenytt (2). 2010. UNN. 6-3-0012.Google Scholar
- Erhardt L, Herlitz J, Bossaert L, Halinen M, Keltai M, Koster R, et al. Task force on the management of chest pain. Eur Heart J. 2002;23:1153–76.View ArticlePubMedGoogle Scholar
- Kostopoulou O, Delaney BC, Munro CW. Diagnostic difficulty and error in primary care--a systematic review. Fam Pract. 2008;25:400–13.View ArticlePubMedGoogle Scholar
- Murphy CL, Sheane B, Durcan I, O’Shea FD, Doran M, Cunnane G. Benefits of pre-referral preparation for rheumatology clinics. Ir Med J. 2013;106:275–7.PubMedGoogle Scholar
- Cohen J. Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychol Bull. 1968;70:213–20.View ArticlePubMedGoogle Scholar
- Maclure M, Willett WC. Misinterpretation and misuse of the kappa statistic. Am J Epidemiol. 1987;126:161–9.View ArticlePubMedGoogle Scholar
- Fleiss JL, Cohen J. The equivalence of weighted Kappa and the intraclass correlation coefficient as measures of reliability. Educ Psychol Meas. 1973;33:613–9.View ArticleGoogle Scholar
- Brenner H, Kliebsch U. Dependence of weighted kappa coefficients on the number of categories. Epidemiology. 1996;7:199–202.View ArticlePubMedGoogle Scholar
- Mathieu JE, Aguinis H, Culpepper SA, Chen G. Understanding and estimating the power to detect cross-level interaction effects in multilevel modeling. J Appl Psychol. 2012;97:951–66.View ArticlePubMedGoogle Scholar
- Marshall SW. Power for tests of interaction: effect of raising the type I error rate. Epidemiol Perspect Innov. 2007;4:4.View ArticlePubMedPubMed CentralGoogle Scholar
- Eldridge S, Kerry S. A practical guide to cluster randomised trials in health services research. Chichester: John Wiley & Sons, Ltd; 2012.View ArticleGoogle Scholar
- Emslie C, Grimshaw J, Templeton A. Do clinical guidelines improve general practice management and referral of infertile couples? BMJ. 1993;306:1728–31.View ArticlePubMedPubMed CentralGoogle Scholar
- Jiwa M, Walters S, Mathers N. Referral letters to colorectal surgeons: the impact of peer-mediated feedback. Br J Gen Pract. 2004;54:123–6.PubMedPubMed CentralGoogle Scholar
- Jillian M, Louise C, Sara T, Iain C, Jeremy G, Alastair L, et al. Pragmatic randomised controlled trial to evaluate guidelines for the management of infertility across the primary care-secondary care interface. BMJ. 2001;322:1282.View ArticleGoogle Scholar
- Akbari A, Mayhew A, Al-Alawi MA, Grimshaw J, Winkens R, Glidewell E, et al. Interventions to improve outpatient referrals from primary care to secondary care. Cochrane Database Syst Rev. 2008;4:CD005471.PubMedGoogle Scholar
- Coulter A, Noone A, Goldacre M. General practitioners’ referrals to specialist outpatient clinics. I. Why general practitioners refer patients to specialist outpatient clinics. BMJ. 1989;299:304–6.View ArticlePubMedPubMed CentralGoogle Scholar
- Ringberg U, Fleten N, Deraas T, Hasvold T, Forde O. High referral rates to secondary care by general practitioners in Norway are associated with GPs’ gender and specialist qualifications in family medicine, a study of 4350 consultations. BMC Health Serv Res. 2013;13:147.View ArticlePubMedPubMed CentralGoogle Scholar
- Forde OH, Breidablik HJ, Ogar P. Do differences in referral rates threaten the goal of equity in health care? Tidsskr Nor Laegeforen. 2011;131:1878–81.View ArticlePubMedGoogle Scholar
- Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–74.View ArticlePubMedGoogle Scholar
- Jiwa M, Skinner P, Coker AO, Shaw L, Campbell MJ, Thompson J. Implementing referral guidelines: lessons from a negative outcome cluster randomised factorial trial in general practice. BMC Fam Pract. 2006;7:65.View ArticlePubMedPubMed CentralGoogle Scholar
- Légaré F. Assessing barriers and facilitators to knowledge use. In: Straus S, Tetroe J, Graham ID, editors. Knowledge Translation in Health Care. Oxford: Blackwell Publishing Ltd; 2009.Google Scholar
- Baker R, Camosso-Stefinovic J, Gillies C, Shaw EJ, Cheater F, Flottorp S, et al. Tailored interventions to overcome identified barriers to change: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2010;3:CD005470.PubMedGoogle Scholar
- Jellema P, van der Windt DA, Bruinvels DJ, Mallen CD, van Weyenberg SJ, Mulder CJ, et al. Value of symptoms and additional diagnostic tests for colorectal cancer in primary care: systematic review and meta-analysis. BMJ. 2010;340:c1269.View ArticlePubMedPubMed CentralGoogle Scholar
- Norheim OF, Allgott B, Gjul GK, Kjellevold A, Moen A, Sjøli S, et al. Åpent og rettferdig - prioriteringer i helsetjenesten (open and fair - priorities in the health care services). Oslo: Norwegian Ministry of Health and Care Services; 2014.Google Scholar