- Research article
- Open Access
- Open Peer Review
Identifying barriers and tailoring interventions to improve the management of urinary tract infections and sore throat: a pragmatic study using qualitative methods
© Flottorp and Oxman; licensee BioMed Central Ltd. 2003
- Received: 17 October 2002
- Accepted: 4 February 2003
- Published: 4 February 2003
Theories of behaviour change indicate that an analysis of factors that facilitate or impede change is helpful when trying to influence professional practice. The aim of this study was to identify barriers to implementing evidence-based guidelines for urinary tract infection and sore throat in general practice in Norway, and to tailor interventions to address these barriers.
We used a checklist to identify barriers and possible interventions to address these in an iterative process that included a review of the literature, brainstorming, focus groups, a pilot study, small group discussions and interviews.
We identified at least one barrier for each category. Both guidelines recommended increased use of telephone consultations and reduced use of laboratory tests, and the barriers and the interventions were similar for the two guidelines. The complexity of changing routines involving patients, general practitioners and general practitioner assistants, loss of income with telephone consultations, fear of overlooking serious disease, perceived patient expectations and lack of knowledge about the evidence for the guidelines were the most prominent barriers. The interventions that were tailored to address these barriers included support for change processes in the practices, increasing the fee for telephone consultations, patient information leaflets and computer-based decision support and reminders.
A systematic approach using qualitative methods helped identify barriers and generate ideas for tailoring interventions to support the implementation of guidelines for the management of urinary tract infections and sore throat. Lack of resources limited our ability to address all of the barriers adequately.
- Focus Group
- Urinary Tract Infection
- Telephone Consultation
- Sore Throat
- Advisory Group
Systematic reviews of interventions to change professional practice show that passive dissemination of information has little or no effect, while other interventions have small to moderate effects at best . There is "no magic bullet", and little evidence to tell which interventions to use for specific problems and settings. Many theories of behaviour change suggest that an analysis of factors that prevent or motivate change in behaviour might be helpful [3, 4].
We conducted a cluster randomised trial in general practice of tailored interventions to support implementation of evidence-based guidelines for urinary tract infections in women and sore throat. The interventions had only a small effect .
The main recommendation for sore throat was that most patients do not need antibiotics. Clinical examination and laboratory tests are therefore generally not necessary, and patients can be given advice by telephone . The main recommendation for urinary tract infections was that non-pregnant women aged 16–55 years with typical symptoms can be treated with antibiotics without any testing. Women who have had a previous urinary tract infection can be offered treatment by telephone .
The aim of the present study is to describe the methods that we used to identify barriers and tailor interventions to increase the likelihood of adherence to these guidelines. The resulting barriers and interventions are presented.
Checklist for identifying barriers to change and possible solutions*
Barriers to change 
Perception of liability
Standards of practice
Knowledge and attitudes
Sense of competence
Compulsion to act
We did not rely on any specific theory of behaviour change, but we used elements from adult learning theories , theory of innovation , the transtheoretical model of behaviour change , and social influence theory . We used a pragmatic approach with a series of largely qualitative methods to identify barriers and tailor interventions to address these.
We considered our own clinical experience and knowledge of primary care in Norway relevant sources of information along with multiple other sources. The project leader (SF) has 20 years experience as a general practitioner (GP) and is still working part-time as a GP. The project coordinator has worked as a GP assistant for 15 years, and AO worked as a GP in Norway before becoming a researcher.
Based on our clinical experience from primary care we had several ideas about factors that might impede implementation of the guidelines. We were genuinely interested in critically checking our assumptions and getting additional information, since our aim was to develop as effective interventions as possible to support adherence to the guidelines.
Methods used to identify barriers to change
The guidelines were systematically developed based on a review of the literature [6, 7]. The included articles were screened for information about impediments to changing practice. We also searched for studies focusing upon barriers to implementation, studies of tailored interventions, and reviews of the effectiveness of different implementation strategies.
Guideline development process
Those who were involved in the process of developing the guidelines were specifically asked to comment upon factors that could influence implementation of the guidelines. This included the advisory group for the project, the guideline development panel, and 48 colleagues and professional organisations that received the draft guidelines for comments. The commentators were asked the following questions:
What problems do you think there might be implementing these guidelines?
What factors can facilitate the implementation of these guidelines?
The advisory group consisted of three general practitioners with experience both from practice and research, two GP assistants and one patient representative with experience as a patient ombudsman. The guideline panel included four members from the advisory group of the Norwegian centre for quality assurance of laboratory services in primary care (NOKLUS): two general practitioners, one microbiologist and one clinical chemist. SF and AO took notes from the meetings with the advisory group and the guideline panel.
Identification of barriers and facilitators to the implementation of the guidelines for sore throat was used as a case in a workshop in the BIOMED-project "Changing Professional Practice" . The workshop participants were implementation researchers from different European countries and with different professional backgrounds. Only a few were GPs. They contributed by means of brainstorming in small groups and in the plenary. Colleagues at our institute, with different professional backgrounds, also participated in a brainstorming session. SF took notes from the brainstorming sessions.
We conducted two focus group interviews with patients and one focus group interview with GP assistants. The patients were 15 women between 16 and 55 years old who had been treated for uncomplicated urinary tract infections, selected from SF's group practice. The focus groups discussed the guidelines for both urinary tract infections and sore throat, and obstacles to implementation of the guidelines. SF was moderator while AO and an anthropologist observed the focus group interviews and took notes.
The focus group interview with GP assistants was conducted with an established small group for continuing education. The nine GP assistants worked in different practices in or near Oslo. SF was the moderator and AO observed the discussion and took notes.
The moderator and the observer(s) discussed the focus groups immediately after the meetings, and took notes from these meetings as well. The focus groups were audiotaped, but not transcribed.
We conducted a pilot study with five practices to get feedback on all aspects of the project, including factors influencing the implementation of guidelines. We visited four practices and discussed implementation of the guidelines with the practice staff.
Small group discussions at interactive courses
In the earliest phase of the intervention period (May and June 2000) we invited practices (both GPs and their assistants) to separate interactive courses for the two intervention groups. Twenty-two of 59 practices (19 general practitioners and 23 assistants) in the urinary tract group and 11 of 61 practices (10 general practitioners and 14 assistants) in the sore throat group participated in these courses. The participants were divided in small groups and asked to describe their current practice, to identify barriers to guideline implementation and to propose solutions to overcome them. The groups used tables based on the checklist with 12 categories of barriers to facilitate this process (table 1).
Informal interviews throughout the project
The project coordinator and other members of the project team made brief notes from telephone contacts with GPs and GP assistants in all the practices during the project, among other things to get useful ideas to improve the interventions.
SF carefully studied the notes that were taken in the guidelines development process, brainstorming sessions, focus group interviews, pilot study, informal interviews and small groups at courses, carried out content analysis of the data and screened the notes for information about barriers and possible solutions [14, 15]. SF and AO discussed the identified barriers and solutions and categorised them using the pre-specified table (table 1).
Tailoring of interventions
Example of scheme used to identify barriers to change and to suggest interventions to address them
Barriers to change
Loss of income: Fee for telephone consultations 22 NOK compared to 110 NOK for ordinary visits (145 NOK for specialists in general practice).
Advisory group, guidelines panel, focus groups, brainstorming in workshops
Increase the fee for telephone consultations.
Telephone consultations should be preferred to ordinary consultations when it is safe and gives better service for patients. An increased fee addresses the financial disincentive.
We obtained funding from The Norwegian Ministry of Health and Social Affairs to evaluate the effect of increased fees for telephone consultations.
We did not use formal methods to select the interventions that were used. SF and AO developed the interventions in an iterative process by focusing upon the suggested solutions to identified barriers, also considering evidence of effectiveness of interventions to change professional behaviour, and available resources.
Examples of barriers to change and interventions tailored to overcome them*
Barrier to change
Barrier adequately addressed?
Loss of income with telephone consultations
Increased fee for telephone consultations
Yes, although we are not sure that the increase in fee was large enough
Changing routines from ordinary visits to increased use of telephone consultations is a complex process
Plan and support for the practices to discuss existing routines and make changes, patient information
No. We did not have the means to actively support the practices discussing and changing their routines
Fear of overlooking serious disease with telephone consultations, delegating responsibility to GP assistants, and not using lab tests
Computer-based decision support with structured questions including check list with warning symptoms
Partially. Most practices needed more active support in using the computer-based system with confidence
Patients want and expect testing for urinary tract infection and sore throat and treatment for sore throat
Patient information, both in written format and computer-based, linked to the decision support system
Partially, this intervention depends on GPs giving patient information. Supplying leaflets is not enough
Not enough time to read information about the project and study the guidelines
Brief versions of guidelines, computer-based reminders, incentives for participation
Partially. We were not able to help practices prioritise time for studying and discussion
Loss of income with telephone consultations was identified as a barrier towards increased use of telephone consultations in practices working on fee for service. The fee for telephone consultations was 22 Norwegian kroner (NOK) (€1.92; €2,95), whereas the fee for an office visit was 110 NOK (€9.58; €14.81) for non-specialists and 155 NOK (€13.50; €20.86) for GP specialists. Although some GPs stated that the low fee for telephone consultation did not influence their practice, the GP assistants underlined this problem in the focus group, in interviews during the project and at the workshops. Poor accessibility by telephone in many practices was another impediment for delivering information and care by telephone.
The complexity of changing routines involving both patients, GPs and their assistants, was considered a major obstacle to implementing the guidelines. Many GPs were reluctant to let GP assistants take care of patients with sore throat or women with uncomplicated urinary tract infections.
GPs, GP assistants and patients were concerned about the risk of overlooking more serious disease if patients were managed by telephone consultations and if GP assistants took more responsibility for managing patients with uncomplicated urinary tract infections and sore throats. Patients appreciated the option of getting information and care by telephone, but they underlined that the offer of a telephone consultation should not be presented as a denial of an ordinary consultation. Patients wanted to have easy access to visit the general practitioners if they preferred so.
Many GPs and GP assistants believed that patients prefer visits to the physician with tests and treatment, rather than telephone consultations. The focus groups with patients and patient view in the advisory group indicated that telephone consultations would be appreciated and preferred to office visits by many women with urinary tract infection. One woman in a focus group said: "The urine sample is for the doctor, not for me; I know when I have got a urinary tract infection." The patients in the focus groups also appreciated evidence-based information about sore throat and the recommendation that testing and consultations were generally not necessary.
Selection and tailoring of interventions
The link between identified barriers and selected interventions is illustrated in table 3. This table also indicates the degree to which we anticipated that the interventions that were chosen adequately addressed the barriers.
We did not identify one major barrier for either of the guidelines. Although there was variation in the barriers across the practices, it was not feasible in this project to assess which intervention might be most effective for each particular practice or each particular practitioner. We developed and offered a package of interventions. Information about the interventions was disseminated with newsletters.
Increased fee for telephone consultations
We applied for and received separate funding from the Norwegian Ministry of Health and Social Affairs to increase the fees for telephone consultations for urinary tract infection in women and sore throat in the respective intervention groups. The fees were increased from 22 to 50 Norwegian kroner (NOK) (€1.92 to €4.35; € 2.96 to € 6.73) with no change in the fee for an office visit, which was 110 NOK (€9.58; € 14.81) for non-specialists and 155 NOK (€13.50; € 20.86) for general practitioner specialists. The new code for the increased fee to be used for the relevant consultations during the trial was installed in the electronic medical record system for the participating practices.
Computer-based decision support
We developed a computer-based decision support program. Algorithmic decision rules for uncomplicated urinary tract infections in women and for sore throat were developed based on the guidelines. The software was linked to the electronic medical record system. The program was sent to the practices on disks and installed by practice members. The program was triggered by relevant International classification of Primary Care (ICPC) codes entered in the electronic medical record for sore throat with patients over three years old and for urinary tract infections in women between 16 and 55 years old. ICPC codes are required for all consultations that are to be reimbursed, and these codes are applied in the diagnosis field in the electronic medical records for most consultations even when there is no fee for service reimbursement. A screen with a few key questions popped up when triggered. The GP or the GP assistant could tick off answers to these questions. This resulted in specific advice based on the guidelines. The program would for instance suggest that a woman with symptoms indicating upper urinary tract infection should visit her GP, whereas a patient with sore throat without any symptoms of serious disease should be offered information by phone about the limited benefits of antibiotics. The program also gave reminders on test use and treatment based on specific information about signs and symptoms, and made it possible to print out patient information. The information gathered by ticking off answers to the questions on the pop-up-screens could be pasted into the electronic medical record. We assumed that the computer-based decisions support would be less useful if the practitioner entered the ICPC code at the end of the consultation. We therefore urged the participants to fill in a relevant diagnosis as soon as possible to have better information during these consultations.
Printed material to facilitate discussions in the practice
To stimulate change processes in the practices we distributed a plan for four brief meetings. This guide included the elements in the "quality circle": review of actual performance, specification of desired performance, implementation of changes and further review of performance.
Incentives for participating
GPs were given credit for continuing medical education (CME) from the Norwegian Medical Association for participating in the project if they organised local meetings in their practices.
Supporting organisational changes
The advisory group, the project staff and participants in the workshops came up with several ideas of technical and administrative solutions to poor accessibility by telephone. It was suggested that automated response systems might be helpful. A system of forwarding calls to a GP assistant dedicated to telephone consultations seemed to work well in some practices. We were not able to translate these ideas and experiences into feasible interventions to support the use of telephone consultations.
In newsletters and telephone contacts with the practices it was repeatedly underlined that GP assistants played a key role in this project. New responsibilities require training and organisational reorientation, but many GPs were reluctant to delegate responsibilities to their assistants. Both GPs and their assistants were invited to courses focused upon implementation of guidelines. The computer-based decision support system was developed to support GP assistants with telephone consultations.
Both the GPs and GP assistants were invited to interactive courses with small group activities. The courses focused on the evidence and the recommendations for either urinary tract infection in women or sore throat, desired changes, obstacles to change and solutions (table 1). We offered three similar one-day courses for each topic.
Patient information in printed and electronic format
We tried to make practitioners aware of the potential gap between the practitioners' perceptions of patients' expectations and the actual expectations of patients, based on evidence from patient view in focus groups and advisory board. Patient information was delivered both in printed and electronic format. An advertising agency assisted in the production of easy to read and attractive leaflets and laminated posters.
Laminated posters with short versions of the guidelines
In addition to reprints of the published reports of the guidelines, short versions of the guidelines were disseminated as laminated posters and in electronic format. The recommendations were presented as briefly and clearly as possible.
Because of the different methods used, we had rich sources of information about barriers to implementation of the guidelines and many suggestions for how to overcome them. Although we used a package of varied interventions, we did not succeed in overcoming all the identified barriers. Table 3 indicates how well we perceived that we addressed the different barriers.
Relatively little is known about the usefulness of alternative methods to identify barriers to change and design interventions to address these. In table 4 (see Additional file 1) we have indicated what the different methods that we used contributed to identifying each barrier. None of the methods we used identified all of the barriers that we considered important. The different methods were not used independently (figure 1), so it is not possible to draw conclusions about the extent to which any of the methods used identified additional barriers that we would not otherwise have considered. Although it may be possible and desirable to use fewer methods than we did in this study, the use of multiple methods strengthened our inferences about what the main barriers were and increased our confidence that we had not overlooked important barriers to change.
The focus group interviews with GP assistants and with patients provided information that was in conflict with the information obtained directly from GPs. The GP assistants through the focus group, advisory group, and informal interviews contributed important insights into daily practice. The GP assistants perceived much more clearly than the GPs that economic incentives influenced practice. Both the patients and the GP assistants expressed concerns about variation in practice among different practitioners within and across practices.
Patients in the focus groups and the advisory group clearly indicated that patients appreciate evidence-based information and the possibility of receiving help by telephone. In contrast, GPs perceived that the patients expected tests and antibiotics. Thus, the GPs' perceptions appeared to be more of a barrier than patient expectations.
Strengths and weaknesses of the study
The categorisation scheme that we used might have biased our search for barriers (table 1). The pre-specified categories are not specific barriers, however, and the aim of this study was not to develop concepts or categories of barriers, but to develop tailored interventions. The way the barriers were categorised was not decisive for this pragmatic purpose. We did not identify barriers that did not fit into the table. The categorisation was useful as a checklist and helped to structure the analyses.
Our samples of informants were not necessarily representative in the brainstorming sessions, guidelines development process, focus groups, pilot study or small groups. The aim was not to do a quantitative study and weight the different barriers in relation to a representative sample of practitioners, however. We therefore used strategic sampling with a large number of informants in different positions to cover varying opinions. There was a great overlap in the barriers that were identified with the different methods that we used (table 4, see Additional file 1). We felt that we reached saturation of information during the project, as it was increasingly difficult to identify new barriers or solutions.
The choice of interventions was limited both by the number of alternatives we were aware of and by restricted resources in the project. We tried to develop interventions that were potentially effective, based on evidence from systematic reviews. Didactic, passive continuing medical education has limited effect on practice, whereas interactive methods can be effective [18, 19]. Computer-based decision support and reminders can change practice [1, 20]. Multi-faceted interventions may be more effective than single interventions [1, 21]. We did not have the resources to develop solutions to overcome all identified barriers, or to use more active interventions like outreach visits.
The identified barriers and suggested solutions not only informed the selection of interventions, but also influenced and coloured the content of the interventions; it helped us design the messages in newsletters, patient information, courses and computer program.
In table 5 (see Additional file 2) we have summarised our perceptions of the strengths and weaknesses of the methods that we used. Several of the methods that we used had other aims as well and required few additional resources, except for the analysis. This included activities connected to developing the guidelines, the pilot study, small-group discussions at workshops, and interviews with participants in the trial. It also included the focus groups with patients, although the focus groups in general required more of an investment of time to organise and conduct.
The number of focus groups was restricted by time constraints in the project. The selection of participants in the groups was pragmatic rather than strategic. The patients were recruited from the project leader's group practice, and this might to some degree have influenced these focus groups. The input from the patients and the GP assistants was, however, congruent with the perspective of the patients and the GP assistants in the brainstorming sessions, the advisory group and interviews with participants in the trial.
Lessons learnt from other studies of tailored interventions
In five of 18 trials in a Cochrane Review on educational outreach visits , one or more barriers to change were identified and the interventions were designed to address these barriers [23–27]. The barriers to change were patient expectations, information management (including the inability to recall pertinent information) and administrative constraints (such as lack of time). A review of continuing education meetings and workshop identified only one study that formally assessed barriers to change. To address perceived patient expectations this study used small focus groups comprised of both patients and health care providers. In another study there was an explicit attempt to involve learners by conducting a formal assessment of participant learning needs. A review of audit and feedback versus alternative strategies extracted data regarding barriers to changing practice in the 12 studies identified. The following descriptions of barriers were used: information management, clinical uncertainty, sense of competence, perception of liability, patient expectations, standards of practice, financial disincentive and administrative issues. All eight barriers to change were mentioned at least once and three were addressed by the intervention in at least one trial. Information management was the most common barrier discussed. A review of computerised advice on drug dosage to improve prescribing practice including 15 trials found that most studies did not identify potential barriers to change . Of those that did, the most common barrier was clinical uncertainty about the best course of action. A review of 41 studies on improving the management of diabetes mellitus found that most included studies identified one or more barriers to change in diabetes care and interventions were designed to address these barriers . Reported barriers to change were lack of acceptance of guidelines, lack of knowledge of diabetology, poor co-operation of staff members, poor quality of documentation of provided care that lead to discontinuous care, the complexity of the guidelines and the lack of information needed to incorporate them into practice, non-attendance and poor compliance of patients.
Different methods for identifying barriers and developing interventions have been used in studies of tailored interventions: informal meetings between study investigators and physician leaders , or attending faculty, discussions about barriers in studies of continuing medical education [37, 38], interviews with practitioners [23, 39–41], helping staff resolve organisational problems blocking acceptance to guidelines, focus groups [25, 39, 43], questionnaires, and a meeting of opinion leaders .
Barriers to improving management of hypertension in the elderly were identified by semi-structured visits in small groups within each practice, led by a trained facilitator . Barriers to appropriate antibiotic use were discussed in focus groups with primary care physicians . In-depth interviews and psychological theory were used to explain obstacles and develop interventions to implement guidelines for management of depression in primary care. Focus groups, direct observation and interviews were used to explore why public health physicians seldom use research-based information, and to develop an intervention to remedy this. Semi-structured interviews, group discussions and a mailed survey helped design an intervention to improve the use of active sick leave by patients with low back pain.
One way of determining whether problems with tailoring interventions are due to shortcomings in identifying barriers or in the interventions that are used to address identified barriers, is through attempting to identify additional barriers in the context of evaluations of interventions and process evaluations. It is unlikely that we missed important barriers to change before we developed our interventions, as neither the participants nor we identified additional barriers during the course of the study or in discussing the results[5, 48]. We were aware that our interventions were not able to address all the barriers thoroughly. More resources and creativity might have increased our ability of tailoring more effective interventions. Comparative research is needed to determine the marginal value of using more resource intensive methods to design interventions to improve healthcare practice.
A pragmatic, iterative approach using largely qualitative methods identified many barriers to change and generated ideas for how to address these. Restricted resources limited the choice of the interventions and hence our potential to address all of the identified barriers. Differences in the information that we collected from GPs, GP assistants and patients highlight the need to seek multiple perspectives and not to rely solely on self-perception. Although we found focus groups useful, practical and rewarding, they required a greater investment of resources than other methods that could more easily be used in connection with guideline development, pilot testing and evaluating interventions.
It is a paradox that so few resources are available to design and evaluate interventions to improve practice while vast sums are being wasted on ineffective forms of care and ineffective forms of continuing medical education and quality improvement.
We thank Kari Håvelsrud who as the project coordinator was responsible for telephone contacts with practices. The study was funded by the Quality Assurance Fund of the Norwegian Medical Association and the National Institute of Public Health. We thank all informants for generous support.
- Grimshaw JM, Shirran L, Thomas R, Mowatt G, Fraser C, Bero L, et al: Changing provider behavior: an overview of systematic reviews of interventions. Med Care. 2001, 39: II2-45. 10.1097/00005650-200108002-00002.View ArticlePubMedGoogle Scholar
- Oxman AD, Thomson MA, Davis DA, Haynes RB: No magic bullets: a systematic review of 102 trials of interventions to improve professional practice. CMAJ. 1995, 153: 1423-1431.PubMedPubMed CentralGoogle Scholar
- Grol R: Personal paper. Beliefs and evidence in changing clinical practice. BMJ. 1997, 315: 418-421.View ArticlePubMedPubMed CentralGoogle Scholar
- Wensing M, Laurant M, Hulscher M, Grol R: Methods for identifying barriers and facilitators for implementation. In Changing Professional Practice. Theory and Practice of Clinical Guidelines Implementation. Edited by: Thorsen T, Mäkelä M. 1999, Copenhagen: DSI, 119-132.Google Scholar
- Flottorp S, Oxman AD, Havelsrud K, Treweek S, Herrin J: Cluster randomised controlled trial of tailored interventions to improve the management of urinary tract infections in women and sore throat. BMJ. 2002, 325: 367-10.1136/bmj.325.7360.367.View ArticlePubMedPubMed CentralGoogle Scholar
- Flottorp S, Oxman AD, Cooper JG, Hjortdahl P, Sandberg S, Vorland LH: Retningslinjer for diagnostikk og behandling av sår hals. Tidsskr Nor Laegeforen. 2000, 120: 1754-1760.PubMedGoogle Scholar
- Flottorp S, Oxman AD, Cooper JG, Hjortdahl P, Sandberg S, Vorland LH: Retningslinjer for diagnostikk og behandling av akutte vannlatingsplager hos kvinner. Tidsskr Nor Laegeforen. 2000, 120: 1748-1753.PubMedGoogle Scholar
- Oxman AD, Flottorp S: An overview of strategies to promote implementation of evidence based health care. In Evidenced Based Practice in Primary Care. Edited by: Silagy C, Haines A. 1998, London: BMJ Books, 91-109.Google Scholar
- Davis DA, Fox RD, editor: The physician as learner: Linking research to practice. Chicago: American Medical Association;. 1994Google Scholar
- Rogers EM: Diffusions of innovations. New York, The Free Press. 1995, 4thGoogle Scholar
- Prochaska JO, DiClemente CC, Norcross JC: In search of how people change. Applications to addictive behaviors. Am Psychol. 1992, 47: 1102-1114. 10.1037//0003-066X.47.9.1102.View ArticlePubMedGoogle Scholar
- Mittman BS, Tonesk X, Jacobson PD: Implementing clinical practice guidelines: social influence strategies and practitioner behavior change. QRB Qual Rev Bull. 1992, 18: 413-422.PubMedGoogle Scholar
- Thorsen T, Mäkelä M, editors: Changing Professional Practice. Theory and Practice of Clinical Guidelines Implementation. Copenhagen, DSI. 1999Google Scholar
- Krueger RA: Focus groups. A practical guide for applied research. Thousand Oaks: Sage Publications. 1994Google Scholar
- Kitzinger J: Qualitative research. Introducing focus groups. BMJ. 1995, 311: 299-302.View ArticlePubMedPubMed CentralGoogle Scholar
- Treweek S, Flottorp S: Using electronic medical records to evaluate healthcare interventions. Health Informatics Journal. 2001, 7: 96-102.View ArticleGoogle Scholar
- Parkerson GRJ, Bridges-Webb C, Gervas J, Hofmans-Okkes I, Lamberts H, Froom J, et al: Classification of severity of health problems in family/general practice: an international field trial. Fam Pract. 1996, 13: 303-309.View ArticlePubMedGoogle Scholar
- Davis D, O'Brien MA, Freemantle N, Wolf FM, Mazmanian P, Taylor-Vaisey A: Impact of formal continuing medical education: do conferences, workshops, rounds, and other traditional continuing education activities change physician behavior or health care outcomes?. JAMA. 1999, 282: 867-874. 10.1001/jama.282.9.867.View ArticlePubMedGoogle Scholar
- Thomson O'Brien MA, Freemantle N, Oxman AD, Wolf F, Davis DA, Herrin J: Continuing education meetings and workshops: effects on professional practice and health care outcomes (Cochrane Review). In: The Cochrane Library, Issue 4. 2001, . Oxford: Update SoftwareGoogle Scholar
- Hunt DL, Haynes RB, Hanna SE, Smith K: Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA. 1998, 280: 1339-1346. 10.1001/jama.280.15.1339.View ArticlePubMedGoogle Scholar
- Wensing M, Grol R: Single and combined strategies for implementing changes in primary care: a literature review. Int J Qual Health Care. 1994, 6: 115-132.View ArticlePubMedGoogle Scholar
- Thomson O'Brien MA, Oxman AD, Davis DA, Haynes RB, Freemantle N, Harvey EL: Educational outreach visits: effects on professional practice and health care outcomes (Cochrane Review). In: The Cochrane Library, Issue 4. 2001, . Oxford: Update SoftwareGoogle Scholar
- Avorn J, Soumerai SB, Everitt DE, Ross-Degnan D, Beers MH, Sherman D, et al: A randomized trial of a program to reduce the use of psychoactive drugs in nursing homes. N Engl J Med. 1992, 327: 168-173.View ArticlePubMedGoogle Scholar
- Ross-Degnan D, Soumerai SB, Goel PK, Bates J, Makhulo J, Dondi N, et al: The impact of face-to-face educational outreach on diarrhoea treatment in pharmacies. Health Policy Plan. 1996, 11: 308-318.View ArticlePubMedGoogle Scholar
- Santoso B, Suryawati S, Prawaitasari JE: Small group intervention vs formal seminar for improving appropriate drug use. Soc Sci Med. 1996, 42: 1163-1168. 10.1016/0277-9536(95)00390-8.View ArticlePubMedGoogle Scholar
- Soumerai SB, Salem-Schatz S, Avorn J, Casteris CS, Ross-Degnan D, Popovsky MA: A controlled trial of educational outreach to improve blood transfusion practice. JAMA. 1993, 270: 961-966. 10.1001/jama.270.8.961.View ArticlePubMedGoogle Scholar
- Avorn J, Soumerai SB: Improving drug-therapy decisions through educational outreach. A randomized controlled trial of academically based "detailing". N Engl J Med. 1983, 308: 1457-1463.View ArticlePubMedGoogle Scholar
- Hadiyono JE, Suryawati S, Danu SS, Sunartono , Santoso B: Interactional group discussion: results of a controlled trial using a behavioral intervention to reduce the use of injections in public health facilities. Soc Sci Med. 1996, 42: 1177-1183. 10.1016/0277-9536(95)00391-6.View ArticlePubMedGoogle Scholar
- Jennett PA, Laxdal OE, Hayton RC, Klaassen DJ, Swanson RW, Wilson TW, et al: The effects of continuing medical education on family doctor performance in office practice: a randomized control study. Med Educ. 1988, 22: 139-145.View ArticlePubMedGoogle Scholar
- Thomson O'Brien MA, Oxman AD, Davis DA, Haynes RB, Freemantle N, Harvey EL: Audit and feedback versus alternative strategies: effects on professional practice and health care outcomes (Cochrane Review). In: The Cochrane Library, Issue 4. 2001, . Oxford: Update SoftwareGoogle Scholar
- McPhee SJ, Bird JA, Jenkins CN, Fordham D: Promoting cancer screening. A randomized, controlled trial of three interventions. Arch Intern Med. 1989, 149: 1866-1872. 10.1001/archinte.149.8.1866.View ArticlePubMedGoogle Scholar
- Walton RT, Harvey E, Dovey S, Freemantle N: Computerised advice on drug dosage to improve prescribing practice (Cochrane Review). In: The Cochrane Library, Issue 4. 2001, . Oxford: Update SoftwareGoogle Scholar
- Renders CM, Valk GD, Griffin S, Wagner EH, Eijk JThMv, Assendelft WJJ: Interventions to improve the management of diabetes mellitus in primary care, outpatient and community settings (Cochrane Review). In: The Cochrane Library, Issue 4. 2001, . Oxford: Update SoftwareGoogle Scholar
- Cheater F, Baker R, Hearnshaw H, Robertson N, Hicks N, Oxman A, et al: Tailored interventions to overcome identified barriers to change: effects on professional practice and health care outcomes (Protocol for a Cochrane Review). In: The Cochrane Library, Issue 4. 2001, . Oxford: Update SoftwareGoogle Scholar
- A controlled trial to improve care for seriously ill hospitalized patients. The study to understand prognoses and preferences for outcomes and risks of treatments (SUPPORT). The SUPPORT Principal Investigators. JAMA. 1995, 274: 1591-1598.Google Scholar
- Aucott JN, Pelecanos E, Dombrowski R, Fuehrer SM, Laich J, Aron DC: Implementation of local guidelines for cost-effective management of hypertension. A trial of the firm system. J Gen Intern Med. 1996, 11: 139-146.View ArticlePubMedGoogle Scholar
- Cranney M, Barton S, Walley T: Addressing barriers to change: an RCT of practice-based education to improve the management of hypertension in the elderly. Br J Gen Pract. 1999, 49: 522-526.PubMedPubMed CentralGoogle Scholar
- Dietrich AJ, Sox CH, Tosteson TD, Woodruff CB: Durability of improved physician early detection of cancer after conclusion of intervention support. Cancer Epidemiol Biomarkers Prev. 1994, 3: 335-340.PubMedGoogle Scholar
- Leviton LC, Goldenberg RL, Baker CS, Schwartz RM, Freda MC, Fish LJ, et al: Methods to encourage the use of antenatal corticosteroid therapy for fetal maturation: a randomized controlled trial. JAMA. 1999, 281: 46-52. 10.1001/jama.281.1.46.View ArticlePubMedGoogle Scholar
- Baker R, Reddish S, Robertson N, Hearnshaw H, Jones B: Randomised controlled trial of tailored strategies to implement guidelines for the management of patients with depression in general practice. Br J Gen Pract. 2001, 51: 737-741.PubMedPubMed CentralGoogle Scholar
- Schers H, Wensing M, Huijsmans Z, van Tulder M, Grol R: Implementation barriers for general practice guidelines on low back pain a qualitative study. Spine. 2001, 26: E348-E353. 10.1097/00007632-200108010-00013.View ArticlePubMedGoogle Scholar
- Evans D, Mellins R, Lobach K, Ramos-Bonoan C, Pinkett-Heller M, Wiesemann S, et al: Improving care for minority children with asthma: professional education in public health clinics. Pediatrics. 1997, 99: 157-164.View ArticlePubMedGoogle Scholar
- Hux JE, Melady MP, DeBoer D: Confidential prescriber feedback and education to improve antibiotic use in primary care: a controlled trial. CMAJ. 1999, 161: 388-392.PubMedPubMed CentralGoogle Scholar
- van der Weijden T, Grol RP, Knottnerus JA: Feasibility of a national cholesterol guideline in daily practice. A randomized controlled trial in 20 general practices. Int J Qual Health Care. 1999, 11: 131-137. 10.1093/intqhc/11.2.131.View ArticlePubMedGoogle Scholar
- Soumerai SB, McLaughlin TJ, Gurwitz JH, Guadagnoli E, Hauptman PJ, Borbas C, et al: Effect of local medical opinion leaders on quality of care for acute myocardial infarction: a randomized controlled trial [see comments]. JAMA. 1998, 279: 1358-1363. 10.1001/jama.279.17.1358.View ArticlePubMedGoogle Scholar
- Forsetlund L, Bjørndal A: Identifying barriers to the use of research faced by public health physicians in Norway and developing an intervention to reduce them. J Health Serv Res Policy. 2002, 7: 10-18. 10.1258/1355819021927629.View ArticlePubMedGoogle Scholar
- Scheel IB, Hagen KB, Oxman AD: Active sick leave for patients with back pain: all the players onside, but still no action. Spine. 2002, 27: 654-659. 10.1097/00007632-200203150-00016.View ArticlePubMedGoogle Scholar
- Flottorp S, Håvelsrud K, Oxman AD: Process evaluation of a randomised trial of interventions to implement guidelines in primary care – why is it so hard to change practice?. Fam Pract.Google Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6963/3/3/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article: verbatim copying and redistribution of this article are permitted in all media for any purpose, provided this notice is preserved along with the article's original URL.