Impact of interprofessional education about psychological and medical comorbidities on practitioners’ knowledge and collaborative practice: mixed method evaluation of a national program
© The Author(s). 2016
Received: 28 January 2016
Accepted: 26 August 2016
Published: 2 September 2016
Many patients with chronic physical illnesses have co-morbid psychological illnesses, which may respond to interprofessional collaborative care. Continuing education programs frequently focus on skills and knowledge relevant for individual illnesses, and unidisciplinary care. This study evaluates the impact of “Mind the Gap”, an Australian interprofessional continuing education program about management of dual illnesses, on practitioners’ knowledge, use of psychological strategies and collaborative practice.
A 6-h module addressing knowledge and skills needed for patients with physical and psychological co-morbid illnesses was delivered to 837 practitioners from mixed health professional backgrounds, through locally-facilitated workshops at 45 Australian sites. We conducted a mixed-methods evaluation, incorporating observation, surveys and network analysis using data collected, before, immediately after, and three months after training.
Six hundred forty-five participants enrolled in the evaluation (58 % GPs, 17 % nurses, 15 % mental health professionals, response rate 76 %). Participants’ knowledge and confidence to manage patients with psychological and physical illnesses improved immediately. Among the subset surveyed at three months (response rate 24 %), referral networks had increased across seven disciplines, improvements in confidence and knowledge were sustained, and doctors, but no other disciplines, reported an increase in use of motivational interviewing (85.9 % to 96.8 %) and mindfulness (58.6 % to 74 %).
Interprofessional workshops had an immediate impact on the stated knowledge and confidence of participants to manage patients with physical and psychological comorbidities, which appears to have been sustained. For some attendees, there was a sustained improvement in the size of their referral networks and their use of some psychological strategies.
KeywordsContinuing education Chronic disease Team Mental health Interprofessional education
Chronic physical illnesses frequently co-exist with psychological illnesses . Chronic illnesses and psychological illnesses can each escalate progression of the other, and each impacts on patients’ self-management capacities. The standard medications for many psychiatric illnesses have metabolic side-effects, just as some medications for chronic illnesses can have psychological side-effects. The general practice consultation may be the best opportunity for many patients with co-morbid psychological and physical illness to be identified and commence treatment . However, incident depression in patients with chronic disease [3, 4], and chronic disease in those with psychological illness are often under-detected in primary care [5, 6].
One of the challenges for clinicians working with co-morbid psychological and physical illnesses is that the evidence base for best practice [7, 8], and continuing education programs, tend to focus on individual illnesses in isolation. In practice, clinicians cannot abstract decisions about treatment of one illness from consideration of the patient’s other illnesses.
Patients with co-morbidities often experience uncoordinated care from a porous network of clinicians and services [9, 10]. Collaborative care models can improve clinical outcomes for patients with a range of chronic diseases and depression [11–13]. A number of Medicare-subsidised items exist in Australia that could support more interdisciplinary care for patients with co-morbid psychological and physical illnesses. Patients with chronic disease can access chronic disease management and team care items  under Medicare, as well as subsidised referrals to some allied health practitioners. Patients with psychological illnesses can be referred for a set number of Medicare-subsidised consultations from mental health professionals in the private sector, under the Better Access initiative . All these policy initiatives rely on health professionals having a suite of management approaches for both physical and psychological illnesses, and the ability to collaborate with one another on a joint plan.
More than two-thirds of treating psychologists in Australia work in private practice . Their connections with GPs have historically been patchy, since patients can self-refer to both disciplines, and there has been no obligation for the professionals to communicate with each other. Even though there has been good take-up of the Better Access Medicare items, these appear to be mainly accessed by people under the age of 45 years, rather than the older age groups more likely to suffer from co-morbid physical illnesses . The most recent review of the Better Access initiative called for more pathways between GPs and psychologists . Interprofessional education (IPE) offers the potential for disciples to learn perspectives shared by one another, and to become better collaborators. Although IPE has proven benefits in developing knowledge and skills in individual practitioners, there are few studies exploring organisational changes or changes in collaborative practice with other health professionals . How to effectively develop and deliver educational programs addressing co-morbidity also receives little attention in the evaluation literature .
This paper aims to contribute to knowledge in both of these areas. We report immediate knowledge, and longer term skills and collaborative practice outcomes of “Mind the Gap”, a national interprofessional educational program for allied health professionals, GPs and mental health professionals, addressing the care of patients with co-morbid psychological and chronic physical illnesses.
Learning objectives, domain of learning and educational strategies used in the Mind the Gap advanced learning module
Domain of learning
Increase understanding of the aetiology, epidemiology, and interrelationships of psychological and chronic medical illnesses
Summary of existing research presented in accessible format by facilitator with back-up reading material
Increase understanding of patient experiences of co-morbid psychological and chronic medical illnesses
Personal presentation by carer or consumer from the local region about their experiences with the health sector, and managing co-morbid illnesses. Reflective exercise between participants.
Develop knowledge and skills in assessment of co-morbid psychological and chronic medical illnesses
Clinical skills, organisational changes
Practise using assessment tools, management and relapse prevention strategies. Case study exploring case management. Distribution of materials on use of practice systems to enhance continuity of care
Recognise the roles and skills of mental health and allied health practitioners in treatment of co-morbid psychological and chronic medical illnesses
Inter-professional case study discussion. Exercise identifying local inter-professional referral pathways
Become conversant with strategies and treatments suitable for co-morbid psychological and chronic medical illnesses in general practice populations
Introduce basic principles of mindfulness, behavioural activation, motivational interviewing, solution-focused therapy, relaxation strategies, and dealing with grief. Group exercise in mindfulness; facilitated reflection on video example of motivational interviewing
The theory of instructional design underpinning the Mind the Gap learning module reflects Reigeluth’s Elaboration Theory : that is, that effective teaching about complex topics requires the information to be reconfigured into smaller units of information which are scaffolded in order to accommodate learning and memory limitations. Recognition and management of co-morbid psychological and medical conditions is a complex topic. Most clinicians are likely to have some prior knowledge or pre-conceptions about this topic, which may be called upon to act as scaffolding for deeper learning. The instructional design therefore began with familiar information, drawing on the statistics of co-morbidity, encouraging clinicians to contextualize this in their own clinical experiences. The less familiar material – counselling techniques, referral processes – were then sequenced and arranged around more familiar concepts, such as one’s own referral networks.
Participants completed two questionnaires at baseline and three months after the workshop: a 27 item questionnaire addressing confidence, knowledge and skills used in managing patients with co-morbid illness, and attitudes to other professionals (Additional file 1: Figure S1); and a 15-item social network questionnaire exploring the size and structure of the individual’s networks with other health professionals (Additional file 2: Figure S2).
The 27-item questionnaire comprised the Interprofessional Socializing and Valuing Scale (ISVS) , measured on a seven-point Likert scale, and a number of items about participants’ knowledge, experience and confidence managing patients with comorbid illnesses, measured on a five-point Likert scale.
The 15-item social network questionnaire explored ego (personal) networks, the networks centred on individual practitioners . Participants were asked to rate relationships with a total of 15 health professionals identified as being of particular relevance to management of co-morbid chronic disease and psychological illness; they could also nominate up to 6 additional professionals. Additional file 3: Table S1 shows the range of disciplines identified by participants in this study. The survey collected data on three different professional relationships within each personal network: the sharing of information, referral patterns, and working together in other ways. Data were paired using an anonymised unique identifier.
Participants were also asked to complete a 10-item questionnaire immediately after the workshop, addressing confidence or knowledge, and their plans (if any) to change using a commitment to change (CTC)  approach (Additional file 4: Figure S3). CTC statements from participants focused on intended changes as a result of training (maximum of three). The 3 month follow-up survey (Additional file 5: Figure S4) also included some open questions on whether the changes had been achieved and were likely to be continued, and challenges in instituting the change. The degree of difficulty in instituting changes was rated on a five-point Likert Scale.
Four workshops were observed by three evaluators, using a structured observation sheet (total observation, 23 h).
Distribution of surveys
Pre and post workshop data were analysed using SPSS (v20.0, IBM). As a quality assurance measure, data entries from surveys collected from 9 % of the workshops were double-checked by a second evaluator (concordance 96 %). Changes in knowledge, awareness, confidence and attitudes were assessed by treating Likert scales as continuous data, once it had been confirmed that the means were normally distributed. Matched pairs analysis using t-tests was conducted for before-after, and before-follow up groups. Participants did not all use their unique identifying codes on all three surveys, so comparisons were not made across all three groups. Use of different psychological strategies at follow-up was stratified by discipline group, to accommodate a slight over-representation of doctors in the three-month follow-up data. Commitment to change data were compared to pre-workshop and 3 month follow up data using matched samples. Network analysis was performed using UCINET  software, comparing network characteristics before and after the intervention. Social network derived data were also entered into an SPSS-X compatible dataset and analysed using a paired samples analysis where indicated.
Forty-five workshops were held across seven states and territories, with between 10 and 54 attendees (mean, 19) per workshop. Of the 837 workshop participants, 645 enrolled in the evaluation (54 % GPs, 16 % nurses, 14 % mental health professionals, response rate 77 %). Other allied health practitioners who attended were occupational therapists, pharmacists, physiotherapists, podiatrists and social workers. At three-month follow-up, doctors were over-represented (64 %), and mental health workers and nurses relatively under-represented (11 % and 6 % respectively).
Workshops observed in several states were consistent in their delivery of the program as designed, with the same materials being used in all of them. The workshops were generally well received. Of 590 who provided feedback, 75 % felt that the five learning objectives had been fully met.
Improvement in knowledge and confidence
Matched-pairs analysis of practitioners’ knowledge and confidence to treat patients with physical and psychological illnesses
(n = 522)
(n = 522)
At three monthsb
(n = 160)
Knowledge of patient self-management approaches and strategies
Knowledge of how to assess co-morbid chronic illness and psychological illness
Knowledge of management planning for co-morbid chronic illness and psychological illness
Knowledge of relapse prevention strategies planning
Knowledge of consumer perspectives and carer experiences
Confidence to recognise patients with comorbid physical and psychological illnesses
Confidence in using psycho-educational strategies
Confidence in integrating therapies
Confidence about meeting the needs of carers
Increase in use of psychological strategies
Psycho-educational techniques used by doctors, before and three months after Mind the Gap training workshop
Techniques used at least once in the preceding three months
n = 348
n = 127
Solution focused therapy
Grief and loss counselling
Changes in clinical practice
Actively seeking and assessing patients for physical and psychological co-morbidities.
Improved care planning and shared management, including better collaboration with local services
Involving patients and carers more in disease management planning
One hundred fifty-three participants provided usable responses to the follow-up questionnaire on changes to practice. Respondents indicated that changes instituted were the same as planned changes in 76 % (322) of 421 changes. Matched data between post and follow-up workshop questionnaires were available for 130 respondents from 30 workshop sites, reflecting on 369 changes, or which 77 % were achieved. For this group, changes proposed post-workshop and those instituted and reported at follow up were broadly consistent although often varied in detail. Actual changes in practice tended to be more specific and incremental, for example “try to undertake more chronic disease management” became “I now schedule more time with complex patients” or “collaborating better with local GPs” became “including more detail in letters to doctors”. Respondents who had instituted changes suggested that practice changes had been easy or very easy to make in 61 % of cases. In 22 % (62 changes) they were considered difficult to implement. There were no significant differences between participant discipline or type of change across those changes rated as hard and those rated as easy, suggesting that local or individual factors determined the ease of instituting change.
Change in attitude to interprofessional working
Matched-pairs analysis of interprofessional socialization and valuing scale (ISVS), before and three months after workshops
(n = 154)
At three months
(n = 154)
I feel comfortable initiating discussions about sharing responsibility for client care
I am comfortable engaging in shared decision making with clients
I feel comfortable clarifying misconceptions about the role of someone in my profession
I see myself as preferring to work on an inter-professional team
I am comfortable being the leader in a team situation
I feel confident taking on different roles in a team
I feel comfortable speaking out within the team when others are not keeping the client’s best interests in mind
I believe that inter-professional practice is difficult to implement
Change in collaborative practice
Size of different network types, at baseline and three months, matched pairs analysis (n = 116)
Mean size at baseline
Mean size at 3 months
Statistically significant changes in respondents network ties by discipline, matched pairs analysis (n = 116)
Mean number of ties per network type
Proportion of respondents with any relationship (%)
Outgoing referrals were the most common type of network tie overall, with incoming referrals the least common. This remained constant at baseline and at follow-up (Additional file 6: Table S2). By individual discipline, respondents consistently indicated the strongest type of tie with practice nurses was collaboration, and with pharmacists, seeking information.
The interprofessional workshops had an immediate impact on the stated knowledge and confidence of participants to manage patients with physical and psychological comorbidities. For participants for whom follow-up data was available, there was a sustained improvement in stated knowledge and confidence and in their use of some psychological interventions.
The collaborative practice outcomes were less marked. There was a sustained improvement in attitudes to inter-professional recorded at three months. The increase in actual networking was most marked between doctors and psychologists, perhaps reflecting the fact that these were the two largest participant groups. In this regard, “Mind the Gap” met its objectives of improving attitudes and pathways between some primary care givers of patients with co-morbid medical and psychological illnesses.
There were no overall increases in the size of professional networks for information sharing, referral or other forms of collaboration. Apparent decreases in the use of non-specified counsellors in favour of increased links with specific psychological services such as psychiatrists, psychologists and the Veterans and Veterans Families Counselling Service (VVCS) suggests a shift in practice towards the overt and deliberate prioritisation of mental health supports for this patient group, and a greater awareness of the role and utility of the VVCS, in keeping with the awareness-raising of this service incorporated into the program.
The workshops were ambitious in scope, and occasionally fell short in execution. We attended some workshops where almost all the attendees were doctors, with a few nurses, and others where psychologists were the largest single group. The content of the workshop sometimes struggled to be meaningful for specific health disciplines, rather than for a generic ‘health worker’ audience. Some psychologists expressed concern that the rapid overview of complex psychological interventions might engender in other participants over-confidence about their ability to deliver complex psychological interventions.
Our evaluation found an increase in the use of mindfulness and motivational interviewing, but not other strategies, among participants who were not mental health professionals. Neither of these strategies is new in the general practice therapeutic landscape. Motivational interviewing, in particular, forms the basis of standard general practice guidelines such as those for smoking cessation . In the absence of a control group (unwieldy in the evaluation design for such a large, multi-centred program), it is possible that the increase in stated use may be due to relabelling. Some clinicians may have simply relabelled strategies they already used after recognising their own practices described at the workshop as a named psychological strategy. If this were the case, the pre-workshop low rates of usage may have been an underestimate.
An important limitation to this study is the low numbers at follow-up. Typically, longitudinal surveys used in health education and evaluation have significant attrition in response rates to repeated surveys [31, 32] and our evaluation is no different. The response rate at three months had decreased from 73 % (post-workshop survey) to 47 % of the relevant population. An additional methodological issue may have been respondent burden, which seemed to particularly impact on the social network survey, with 16 % of those who completed the pre-workshop survey and 15 % of those who completed the three month survey failing to complete the social network survey. Social network surveys are recognised as imposing significant respondent burden . In our study the minimum number of items in the social network survey was 75 (15 names, 5 types of relationships), to a maximum of 90. We organised the survey by alters (asking all five questions about each alter, or health discipline) rather than by questions (asking one question at a time for all alters), due to the high number of alters we identified. For written surveys, response rates are higher if the surveys are organised by question rather than alter, probably because the appearance of respondent burden is reduced . We also found that the unique identifier – a combination of middle name and year of birth was irregularly used, limiting our capacity to undertake paired analyses.
Although social network analysis is likely to be a valuable evaluation tool for interprofessional education, our study results suggest that the total number of names generated should be limited and should be arranged by question rather than alter. This holds true for surveys in both paper and web-based form. Additionally, resources should be specifically devoted to enhancing response – for example, through using computer assisted telephone calls, or by placing an incentive on return of survey (such as continuing education recognition).
The second reason for low response rates is the relatively lengthy roll-out of the program, as it was delivered through organisations across Australia. This meant that some of the participants received the training too late in the study period to participate in the formal three-month follow-up survey. We did not find any geographic bias in our study, and although the follow-up data overall includes a slight over representation of medical practitioners, the matched-pairs analysis enables this potential bias to be contained. Modelling for attrition bias in follow-up data suggest that attrition rates up to 60 % are acceptable if the missing cases are missing at random  Our attrition rate is around 69 % (74 % for social network data), and our survey is underpowered to detect subtle changes in social network characteristics. The responses from the three month survey may have been susceptible to respondent bias, in that those who felt most positively about the program may have been more likely to respond to repeated surveys. If this were the case, the level of improvement at three months in this study may be overstated.
Despite these limitations, our evaluation suggests that there were positive outcomes in relation to content knowledge, skills and collaborative networks through the use of inter-professional educational workshops addressing health care for primary care patients with co-morbid illnesses. The evidence for the effectiveness of interprofessional education on collaboration or teamwork has not to date been particularly strong for primary care settings . It is worth noting therefore that particular structural drivers existed in the Australian primary care setting, which increased the likelihood of success of Mind the Gap. The changes that were sought, at least in general practice, were not complex, and could be readily incorporated into existing organisational and clinical work. The person-power to support enhanced case-finding now exists with the widespread uptake of practice nurses into general practice over the last decade . Private health professionals other than GPs had structural incentives to collaborate with GPs, since GPs may refer patients with psychological and medical co-morbidities to private allied health professionals and psychologists for services fundable under the national insurer, Medicare. Mental health professionals in the public sector had an interest in reducing the size of their complex workload through better collaboration. Thus, there were few attendees at the workshops who could not identify immediate benefits to themselves arising from interprofessional collaboration for patients with psychological and medical co-morbidities.
Simply placing health professionals in proximity to one another in an educational session is not sufficient to generate changes in attitude or collaborative practice. If run poorly, it may even cement prejudicial attitudes towards each other , Collaboration is a complex, polyfactorial organisational outcome. Education designers should be cognizant of local referral and collaborative networks, and structural drivers for collaboration, in order to identify and capitalise on effective drivers for change. Mind the Gap was “localised” – delivered by local facilitators, with local interprofessional participants, by the local support organisation - while also being framed by financial and organisational drivers to productive collaboration. A follow-up learning exercise that enabled the continuation of reflective engagement would have helped consolidate gains. Nevertheless, our evaluation suggests that there is likely to be value in educating groups of primary care professionals together on matters that are of direct clinical relevance to them, and for which there are existing structural drivers of collaboration.
This project was funded by the Australian Medicare Local Alliance, through a grant from the Department of Veterans Affairs.
Availability of data and materials
The datasets supporting the conclusions of this article are available in the Open Science Framework repository, DOI 10.17605/OSF.IO/JKTM7 | ARK c7605/osf.io/jktm7; https://osf.io/jktm7/.
CP and SH designed the study; MI oversighted quality assurance and data acquisition; CP, MI and SH analysed the data; SH performed the social network analysis; SH and CP interpreted the results, CP drafted the paper. All authors reviewed and approve final version. The Mind the Gap Advanced Learning Module (ALM) was developed by the Australian General Practice Network.
The authors declare that they have no competing interest.
Consent for publication
Ethics approval and consent to participate
This study was approved by the Australian National University Human Research Ethics Committee, #2011/546. All participants provided informed consent to participate in this evaluation.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Clarke DM, Currie KC. Depression, anxiety and their relationship with chronic disease: a review of the epidemiology, risk and treatment evidence. Med J Aust. 2009;190:54–60.Google Scholar
- Ford DE. Optimizing outcomes for patients with depression and chronic medical illnesses. Am J Med. 2008;121 Suppl 2:S38–44.View ArticlePubMedGoogle Scholar
- Ani C, Bazargan M, Hindman D, Bell D, Rodriguez M, Baker RS. Comorbid chronic illness and the diagnosis and treatment of depression in safety net primary care settings. J Am Board Fam Med. 2009;22:123–35.View ArticlePubMedPubMed CentralGoogle Scholar
- Ormel J, Van Den Brink W, Koeter MWJ, Giel R, Van Der Meer K, Van De Willige G, et al. Recognition, management and outcome of psychological disorders in primary care: a naturalistic follow-up study. Psychol Med. 1990;20:909–23.View ArticlePubMedGoogle Scholar
- Lunay B, Bywood P. Co-morbidity of mental and physical illness: meeting unmet care needs. Research Roundup Issue 18. Adelaide: Primary Health Care Research & Information Service; 2011.Google Scholar
- Whiteford HA. Depression in primary care: expanding the evidence base for diagnosis and treatment. Med J Aust. 2008;188(Suppl):S101–102.PubMedGoogle Scholar
- Boyd CM, Darer J, Boult C, Fried LP, Boult L, Wu AW. Clinical practice guidelines and quality of care for older patients with multiple comorbid diseases: implications for pay for performance. JAMA. 2005;294:716–24.View ArticlePubMedGoogle Scholar
- Kruzikas DT, Jiang HJ, Remus D, Barrett ML, Coffey RM, Andrews R. Preventable hospitalizations: a window into primary and preventive care, 2000. Washington: Agency for Healthcare Research and Quality; 2004.Google Scholar
- Kathol RG, Butler M, McAlpine DD, Kane RL. Barriers to physical and mental condition integrated service delivery. Psychosom Med. 2010;72:511–8.View ArticlePubMedGoogle Scholar
- Henderson J, Battams S. Mental health and barriers to the achievement of the ‘right to health’. Aust J Primary Health. 2011;17:220–6.View ArticleGoogle Scholar
- Atlantis E, Fahey P, Foster J. Collaborative care for comorbid depression and diabetes: a systematic review and meta-analysis. BMJ Open. 2014;4:e004706. doi:https://doi.org/10.1136/bmjopen-2013-004706.View ArticlePubMedPubMed CentralGoogle Scholar
- McGregor M, Lin EH, Katon WJ. TEAMcare: An integrated multicondition collaborative care program for chronic illnesses and depression. J Ambul Care Manage. 2011;34:152–62.View ArticlePubMedPubMed CentralGoogle Scholar
- Morgan MAJ, Coates MJ, Dunbar JA, Reddy P, Schlicht K, Fuller J. The TrueBlue model of collaborative care using practice nurses as case managers for depression alongside diabetes or heart disease: a randomised trial. BMJ Open. 2013;3:e002171. doi:https://doi.org/10.1136/bmjopen-2012-002171.View ArticlePubMedPubMed CentralGoogle Scholar
- Department of Health. Chronic disease management (formerly Enhanced Primary Care or EPC) - GP services. http://www.health.gov.au/internet/main/publishing.nsf/content/mbsprimarycare-chronicdiseasemanagement Updated 2 Apr 2014. Accessed 21 Aug 2016.
- Department of Health. Better access to psychiatrists, psychologists and general practitioners through the MBS (Better Access) initiative. http://www.health.gov.au/mentalhealth-betteraccess. Updated 16 Dec 2015. Accessed 21 Aug 2016.
- Health Workforce Australia. Australia’s Health Workforce Series – psychologists in focus HWA: Adelaide; 2014. http://pandora.nla.gov.au/pan/133228/20150419-0017/www.hwa.gov.au/sites/default/files/HWA_Australia-Health-Workforce-Series_Psychologists%20in%20focus_vF_LR.pdf Accessed 21 Aug 2016.
- Australian Bureau of Statistics. Characteristics of people using mental health services and prescription medication, 2011. Cat No. 4329.0. ABS: Canberra. Released 28 Oct 2014. http://www.abs.gov.au/ausstats/abs@.nsf/productsbytopic/39A5EDBC47806B28CA257D7E000F488B Accessed 21 Aug 2016.
- King D, Tan Y, Wainer J, Smith L, Fitzpatrick D, Sun L, et al. Evaluation of the Better Access initiative component C: analysis of the allied mental health workforce supply and distribution, 2010. http://www.health.gov.au/internet/publications/publishing.nsf/Content/mental-ba-eval-c-toc Updated Sep 2010. Accessed 21 Aug 2016.
- Reeves S, Perrier L, Goldman J, Freeth D, Zwarenstein M. Interprofessional education: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2013;3:CD002213. doi:https://doi.org/10.1002/14651858.CD002213.pub3.Google Scholar
- Schoenborn NL, Boyd C, Cayea D, Nakamura K, Xue QL, Ray A, et al. Incorporating prognosis in the care of older adults with multimorbidity: description and evaluation of a novel curriculum. BMC Med Educ. 2015;15:215.View ArticlePubMedPubMed CentralGoogle Scholar
- Veterans and Veterans Families Counselling Service. About VVCS. http://www.vvcs.gov.au/AboutVVCS/index.htm Accessed 21 Aug 2016.
- Reigeluth C. Elaborating the elaboration theory. Educ Technol Res Dev. 1992;40:80–6.View ArticleGoogle Scholar
- Bushnell DS. Input, process, output: A model for evaluating training. Train Dev J. 1990;44:41–3.Google Scholar
- King G, Shaw L, Orchard C, Miller S. The interprofessional socialization and valuing scale: a tool for evaluating the shift toward collaborative care approaches in health care settings. Work. 2010;35:77–85.PubMedGoogle Scholar
- Hanneman RA, Riddle M. Introduction to social network analysis. Riverside CA: University of California; 2005. Available online: http://www.faculty.ucr.edu/~hanneman/nettext/. Accessed 21 Aug 2016.
- Shershneva MC, Wang M, Lindeman GC, Savoy JN, Olson CA. Commitment to practice change: an evaluator’s perspective. Eval Health Prof. 2010;33:256–75.View ArticlePubMedPubMed CentralGoogle Scholar
- Borgatti SP, Everett MG, Freeman LC. Ucinet for Windows: software for social network analysis. Harvard: Analytic Technologies; 2002.Google Scholar
- Gloster AT, Rhoades HM, Novy D, Klotsche J, Senior A, Kunik M, et al. Psychometric properties of the Depression Anxiety and Stress Scale-21 in older primary care patients. J Affect Disord. 2008;110:248–59.View ArticlePubMedPubMed CentralGoogle Scholar
- Sheik JI, Yesavage JA. Geriatric Depression Scale (GDS): recent evidence and development of a shorter version. In: Brink TL, editor. Clinical gerontology: a guide to assessment and intervention. New York: Haworth Press; 1986.Google Scholar
- Royal Australian College of General Practitioners. Supporting smoking cessation: a guide for health professionals. Melbourne: The Royal Australian College of General Practitioners; 2011.Google Scholar
- Green L. Evaluation and measurement: some dilemmas for health education. Am J Publ Health. 1977;67:155–61.View ArticleGoogle Scholar
- Kristman V, Manno M, Cote P. Loss to follow-up in cohort studies: how much is too much? Eur J Epidemiol. 2004;19:751–60.View ArticlePubMedGoogle Scholar
- McCarty C, Killworth PD, Rennell J. Impact of methods for reducing respondent burden on personal network structural measures. Social Networks. 2007;29:300–15.View ArticleGoogle Scholar
- Coromina L, Coenders G. Reliability and validity of egocentered network data.Collected via web: a meta-analysis of multilevel multitrait multimethod studies. Social Networks. 2006;28(3):209–31.View ArticleGoogle Scholar
- Zwarenstein M, Reeves S, Perrier L. Effectiveness of pre-licensure interprofessional education and post-licensure collaborative interventions. J Interprof Care. 2005;19 Suppl 1:148–65.View ArticlePubMedGoogle Scholar
- Phillips C, Pearce CM, Hall S, Kljakovic M, Sibbald B, Dwan K, Porrit J. Enhancing care, improving quality: the six roles of the general practice nurse. Med J Aust. 2009;191:92–7.PubMedGoogle Scholar
- Carpenter J, Dickinson H. Understanding interprofessional education as an intergroup encounter: the use of contact theory in programme planning. J Interprof Care. 2016. doi:https://doi.org/10.3109/13561820.2015.1070134.PubMedGoogle Scholar