Skip to content

Advertisement

You're viewing the new version of our site. Please leave us feedback.

Learn more

BMC Health Services Research

Open Access
Open Peer Review

This article has Open Peer Review reports available.

How does Open Peer Review work?

A mechanism for revising accreditation standards: a study of the process, resources required and evaluation outcomes

  • David Greenfield1Email author,
  • Mike Civil2,
  • Andrew Donnison2,
  • Anne Hogden1,
  • Reece Hinchcliff1,
  • Johanna Westbrook3 and
  • Jeffrey Braithwaite1
BMC Health Services Research201414:571

https://doi.org/10.1186/s12913-014-0571-8

Received: 7 February 2014

Accepted: 27 October 2014

Published: 21 November 2014

Abstract

Background

The study objective was to identify and describe the process, resources and expertise required for the revision of accreditation standards, and report outcomes arising from such activities.

Methods

Secondary document analysis of materials from an accreditation standards development agency. The Royal Australian College of General Practitioners’ (RACGP) documents, minutes and reports related to the revision of the accreditation standards were examined.

Results

The RACGP revision of the accreditation standards was conducted over a 12 month period and comprised six phases with multiple tasks, including: review methodology planning; review of the evidence base and each standard; new material development; constructing field trial methodology; drafting, trialling and refining new standards; and production of new standards. Over 100 individuals participated, with an additional 30 providing periodic input and feedback. Participants were drawn from healthcare professional associations, primary healthcare services, accreditation agencies, government agencies and public health organisations. Their expertise spanned: project management; standards development and writing; primary healthcare practice; quality and safety improvement methodologies; accreditation implementation and surveying; and research. The review and development process was shaped by five issues: project expectations; resource and time requirements; a collaborative approach; stakeholder engagement; and the product produced. The RACGP evaluation was that participants were positive about their experience, the standards produced and considered them relevant for the sector.

Conclusions

The revision of accreditation standards requires considerable resources and expertise, drawn from a broad range of stakeholders. Collaborative, inclusive processes that engage key stakeholders helps promote greater industry acceptance of the standards.

Keywords

AccreditationHealthcareStandardsGeneral practiceResearch

Background

Government, quality improvement and accreditation agencies frequently engage in the development or revision of clinical and organisational standards. These are significant tasks that utilise considerable human and financial resources [1]. Different organisations produce standards according to their own processes and requirements, and it is believed that inclusive processes result in greater acceptance of the standards produced [1]. However, we do not know what might be evidence-informed practice in the development or revision of accreditation standards [2]. To date, no empirical study has been published that sought to identify the process, resources and expertise required for either of these endeavours [2]. This is a significant gap in the evidence base for the healthcare accreditation field [2]-[4], as accreditation programs have increasingly become an important strategy by which governments seek to regulate healthcare quality and safety [5],[6]. There are now more than 44 health service accreditation programs which have been implemented in over 70 countries [3],[7].

The aim of this study was to identify and describe the process, resources and expertise required for, and to report evaluation outcomes from, a revision of a set of healthcare accreditation standards. The standards and associated revision activities under examination concern the Standards for General Practice (4th Edition) developed and revised by The Royal Australian College of General Practitioners (RACGP), which are used for accrediting general practice nationally. Through implementing an instrumental case study [8],[9] of the RACGP accreditation standards we sought to highlight the process, resource and expertise issues relevant for other accreditation standard setting bodies. The RACGP is representative of other bodies that similarly have responsibility for the development and revision of standards, but do not themselves apply or assess services using the standards [10]. Previous studies have revealed the common issues and challenges facing standard setting and accreditation bodies [7],[11].

Methods

Study context

The Accreditation Collaborative for the Conduct of Research, Evaluation and Designated Investigations through Teamwork (ACCREDIT) was funded across 2011-15 to investigate health service accreditation in Australia [12]. The collaboration comprises university researchers, accreditation agency personnel and staff from leading quality improvement bodies in Australia (Table 1). The collaboration was awarded an Australian Research Council Linkage Project grant (LP100200586) in 2010. Ethics approval for the study was given by the University of New South Wales (UNSW) Human Research Ethics Committee (approval number HREC 10274). The ACCREDIT study protocols are publically available [12]-[15] and are informed by previous accreditation research conducted by UNSW and The Australian Council on Healthcare Standards (ACHS) [10]-[12],[16]-[24], including reviews of the healthcare accreditation literature [2]-[4].
Table 1

ACCREDIT collaborative partners

Partner category

Organisations

University researchers

Centre for Clinical Governance Research, and Centre for Health Systems and Safety Research, in the Australian Institute of Health Innovation at The University of New South Wales

Accreditation agencies

Australian General Practice Accreditation Limited

Aged Care Standards and Accreditation Agency

The Australian Council on Healthcare Standards

Quality improvement bodies

Australian Commission on Safety and Quality in Health Care

New South Wales Clinical Excellence Commission

Setting

Australia has over 7,100 general practices in which more than 23,500 doctors work. There were 125 million consultation services provided during 2010-11, costed at A$5.3 billion through the Medicare Benefits Scheme [25] (this excludes out of pocket expenses of patients). In 2011-12 there were over 4000 general practices accredited against the RACGP Standards for General Practice [26]. The standards cover five areas: practice services (7 standards and 19 criteria); rights and needs of patients (1 standard and 3 criteria); safety, quality improvement and education (2 standards and 7 criteria); practice management (2 standards and 4 criteria); and physical factors (3 standards and 8 criteria). (See: http://www.racgp.org.au/your-practice/standards/standards4thedition/).

Study methodology

An expert group was formed by UNSW researchers and RACGP staff. During 2012, they collaboratively conducted a study with three stages. First, informed by the accreditation and evaluation literatures, the expert group purpose-designed an analysis framework with seven categories including: phase, task, objective, time frame, components, people involved and National Expert Committee - Standards for General Practice (NEC-SGP) involvement. The role of the NEC-SGP includes developing and maintaining standards for general practices, and ensuring that the standards reflect quality practice and are independent of government policies and initiatives. The NEC-SGP comprises experts in standards development with professional backgrounds including general practitioners, practice nurses and managers, and a consumer representative. Since 2011 the NEC-SGP is known as the National Standing Committee - Standards for General Practice. Second, using the framework, thematic analysis [8] of RACGP documents, minutes and evaluation reports related to the revision of the accreditation standards was conducted. More than 50 documents were accessed from the RACGP information system. Third, the group reviewed the findings to clarify the process, resources and expertise utilised, and reported evaluation outcomes. Over several months the expert group discussed the findings in meetings and electronic forums to work through the material, with differences resolved by negotiation [8].

Results

The analysis framework facilitated the identification of a standards review process comprising six sequential and overlapping phases with multiple components (Table 2). The six phases occurred over a 12 month period across 2009-10. Phase 1 comprised the `review methodology planning’ phase, which occurred over two months. This phase involved two tasks: developing the review feedback methodology and tools; and reviewing the evidence base for information on methods of standards development. The review of the evidence base and current standards were the tasks that formed Phase 2, which was completed over a five month period. Following Phase 1, and overlapping with Phase 2, Phase 3 was a five month activity requiring the development of new material for the new standards. The construction of field trial methodology, the sole task in Phase 4, occurred in parallel and was completed in two months. The completion of the initial four phases led to Phase 5 and the combined task of drafting, trailing and refining the new standards, which occupied five months. To complete the project, the final task was the formatting and production of the new standards. This was Phase 6 and occurred over a two month period.
Table 2

Analysis of the review process for the RACGP Standards for general practices (4th edition)

Phase

Task

Objective(s)

Time frame

Components

People involved

National Expert Committee - Standards for General Practice (NEC-SGP) involvement

1 Development of review methodology and tools

Develop the review feedback methodology and tools

Develop methodology to collect feedback from members and stakeholders on current standards

1/09/09 - 30/10/09

Review methodology of previous standards review

Senior Project Officer

Review documentation and meet

Develop method of collection and analysis of feedback

Review and update draft tools

Secretariat

Decide on accepted review methodology (October)

Recommend to NEC-SGP review methodology

Project Manager

Recommend review methodology to RACGP Council (November)

NEC-SGP (5)

RACGP Council ratify review methodology

Review the evidence base for current method of standards development

Ensure development of the new standards are supported by latest evidence

1/09/09 - 30/10/09

Literature search of methods of standards development and assessment

Senior Project Officer

Secretariat provide recommendations to NEC-SGP on methodology of standards development (October meeting)

Recommendations to NEC-SGP on how to develop and assess standards

Secretariat

NEC-SGP reviews evidence and accept recommendations on methodology of standards development (October meeting)

Council acceptance of methodology, timeline and cost

Project Manager

NEC-SGP recommends methodology, timeline and budget required to RACGP Council

NEC-SGP (5)

RACGP Council ratifies methodology of standards development process and approves budget (November meeting)

2 Reviewing the evidence base and the current standards

Review the evidence base for current standards

Ensure new edition includes material that is supported by latest evidence

1/09/09 - 31/12/09

Literature search of each criteria in current edition

Senior Project Officer

NEC-SGP decide on inclusion or exclusion of current criteria based on evidence presented (January 2010 meeting)

Recommendations to NEC-SGP on latest trends relating to each criteria’s relevance to the next edition

Secretariat

Decide on membership of subcommittees and chairs; methodology and tools; new material (e.g. e-health, governance, IC, present material)

NEC-SGP (5)

Review the current standards

All feedback on current edition is considered

1/10/09 - 31/01/10

Review feedback collected by RACGP since release of current edition

Project Manager

Secretariat provides analysis of feedback to subcommittees

New material for possible inclusion in revised edition is identified

Collection of feedback from members and stakeholders via focus groups, online surveys and written submissions

Secretariat

Analysis of feedback

Recommendations to NEC-SGP and subcommittees on material identified from formal feedback

3 New material development

Develop new material for new standards

Material included in new edition is evidence based

1/11/09 - 31/03/10

New material is matched with evidence - literature search relating to new material

Senior Project Officer

Secretariat draft new material

New criteria, indicators and explanatory material are drafted

Senior Project Officer

Sub committees recommend changes to drafts, approve draft of new material and make recommendations to NEC-SGP

Project Manager

NEC-SGP accept recommendations of sub committees

Senior Project Officer

NEC-SGP approve draft of new material for trial

Working groups

NEC-SGP recommend draft to RACGP Council for endorsement for trial

Secretariat

4 Develop field trial methodology

Develop field trial methodology

Prepare field trial of revised standards

1/1/10 - 28/2/10

Methodology of field trial of new material is described

Senior Project Officer

Secretariat design trial methodology and tools

Trial tools are developed

GP research expert

Subcommittee approve to methodology and tools

Secretariat

Subcommittees recommend methodology and tools to NEC-SGP

NEC-SGP (5)

NEC-SGP approve trial methodology (April meeting)

NEC-SGP recommend methodology and tools to RACGP Council (April meeting)

5 Drafting, trials and feedback about the new standards

Seek feedback, trial and refine draft new standards

Ensuring new edition of standards meet needs of RACGP and stakeholders

1/4/10 - 31/8/10

First line feedback groups provide an early indication of the feasibility and acceptability of proposed explanatory material and indicators to the working parties

Accreditation agencies (participant numbers not specified)

Subcommittees approve changes and recommend penultimate version to NEC-SGP

On line survey for participation by all interested general practice professionals

Project Manager

NEC-SGP approves penultimate draft (July meeting)

Written submissions

Senior Project Officer

NEC-SGP recommends final Standards to RACGP Council for endorsement

Focus groups

Secretariat

RACGP Council endorses Standards (August meeting)

Trialling of draft standards through field trial (only new Criterion to be tested)

NEC-SGP (5)

6 Standard document revision and production

Format, produce and launch new standards

New edition of standards are produced and released to the sector

1/9/10 - 4/10/10

Ensure formatting of new standards is correct

Project Manager

NEC-SGP members, secretariat and key stakeholders present at launch

New standards are launched during ASC 2010

Senior Project Officer

Online version is made available

Publications/ IT

Complimentary copies sent to key stakeholders

Secretariat

Over 100 individuals substantively participated in the review process, with an additional 30 providing periodic input and feedback. Participants were drawn from general practice stakeholders, including: healthcare professional associations; primary healthcare services; accreditation agencies; government agencies; and public health organisations. Their expertise spanned the fields of: project management; standards development and writing; primary healthcare practice; quality and safety improvement methodologies; accreditation implementation and surveying; and research methodologies.

The review and development process was shaped by five factors. First, identifying and delivering upon the requirements of the RACGP, and stakeholders associated with the standards, was reported as necessary for the credibility of the product. Second, identifying and communicating resource and time restrictions to participants, and observers within the sector, was required to enable the review to be completed as expected. Managing expectations and employing an effective communication strategy reinforced a collaborative approach and facilitated broad stakeholder engagement with the review; these being the third and fourth issues identified as essential for a positive development process leading to the acceptance of revised standards. Finally, the review project had to deliver a well structured, clearly written, evidence based, high quality document that was consistent with previous editions. One significant improvement suggestion emerged from the evaluation: a majority of participants agreed that consideration could be given to altering the standards revision process to conduct periodic reviews and progressive updates.

Discussion

This study provides the first case-study evidence about processes invoked for the development and revision of accreditation standards [2], and lays the foundation for further work in this area [3],[4]. The research reveals that the revision of accreditation standards is a major undertaking requiring considerable resources and expertise, drawn from a broad range of stakeholders. Industry acceptance of the standards produced was found to be related to a collaborative, inclusive process, grounded by clinical evidence and process reviews, which promoted stakeholder participation. These findings support previously reported, non-empirical assessments, of how to approach the task [1]. For other accrediting bodies the study provides three things: insight into a difficult and challenging process; encouragement to investigate and make public their own experiences; and, a template and structure to follow to undertake such forensic examinations.

The project was completed through the combined efforts, or distributed leadership [19], of more than 100 people over a 12 month period. Key influences on the review process were: project requirements and stakeholder expectations; resource and time restrictions; collaborative team approach; stakeholder engagement; and the product required. The revision process necessitated the delicate balancing of these issues to maintain cohesion and continued participation between diverse and distributed stakeholders over an extended time period. Methodological rigour, as recognised by the six phases, was applied by the RACGP in the development, piloting and revision of materials. The commitment and effort of agency staff and committee members, who efficiently used resources with strict time constraints, enabled the efficient completion of the project. RACGP evaluation showed that stakeholder acceptance of the revision process and the revised standards produced was based on their perception of a transparent, inclusive and rigorous process implemented by the College [1].

Conclusion

The revision of accreditation standards requires collaboration from a diverse range of professionals, with considerable resources and expertise. The collaborative, inclusive process employed engaged stakeholders and promoted the acceptance of the revised standards by the sector.

Authors’ contributions

DG, RH, JW and JB conceived of the study, and participated in its design and coordination and helped to draft the manuscript. AD participated in the data collection and analysis, and along with AH and MC assisted in the revising of the manuscript. All authors read and approved the final manuscript.

Declarations

Acknowledgements

This research was supported under Australian Research Council’s Linkage Projects funding scheme (project number LP100200586). The RACGP staff and members of the National Standing Committee - Standards for General Practice are thanked for access to documentation and contribution to this study. We also recognise our national and international collaborators for their contributions to the ACCREDIT project.

Authors’ Affiliations

(1)
Centre for Clinical Governance Research, Australian Institute of Health Innovation, Faculty of Medicine, University of New South Wales
(2)
Chair National Standing Committee - Standards for General Practice, The Royal Australian College of General Practitioners
(3)
Centre for Health Systems and Patient Safety Research, Australian Institute of Health Innovation, University of New South Wales

References

  1. McGlynn E, Shekelle P, Hussey P: Developing, Disseminating and Assessing Standards for the National Health Service: An Assessment of Current Status and Opportunities for Improvement. RAND Health; 2008.Google Scholar
  2. Greenfield D, Pawsey M, Hinchcliff R, Moldovan M, Braithwaite J: The standard of healthcare accreditation standards: a review of empirical research underpinning their development and impact. BMC Health Serv Res 2012, 12:329..Google Scholar
  3. Greenfield D, Braithwaite J: Health sector accreditation research: a systematic review. Int J Qual Health Care. 2008, 20 (3): 172-183. 10.1093/intqhc/mzn005.View ArticlePubMedGoogle Scholar
  4. Hinchcliff R, Greenfield D, Moldovan M, Pawsey M, Mumford V, Westbrook J, Braithwaite J: Narrative synthesis of health service accreditation literature. BMJ Qual Saf. 2012, 21: 979-991. 10.1136/bmjqs-2012-000852.View ArticlePubMedGoogle Scholar
  5. National Safety and Quality Health Service Standards. 2011, Commonwealth of Australia, Sydney, AustraliaGoogle Scholar
  6. Access to Standards. [], [www.accreditation.ca/review-our-standards]
  7. Braithwaite J, Shaw C, Moldovan M, Greenfield D, Hinchcliff R, Mumford V, Kristensen M, Westbrook J, Nicklin W, Fortune T, Whittaker S: Comparison of health service accreditation in low- and middle-income countries with those in higher income countries: a cross sectional study. Int J Qual Health Care. 2012, 24 (6): 568-577. 10.1093/intqhc/mzs064.View ArticlePubMedGoogle Scholar
  8. Liamputtong P: Qualitative Research Methods. 2009, Oxford University Press, Melbourne, 3Google Scholar
  9. Creswell J: Qualitative Inquiry and Research Design: Choosing Among Five Approaches. 2007, Sage Publications, LondonGoogle Scholar
  10. Greenfield D, Pawsey M, Braithwaite J: Accreditation: A Global Regulatory Mechanism to Promote Quality and Safety. Continuous Quality Improvement in Health Care. Edited by: Sollecito W, Johnson J. 2013, Jones and Barlett Learning, New York, 513-531. 4Google Scholar
  11. Greenfield D, Pawsey M, Braithwaite J: The role and impact of accreditation on the healthcare revolution. O papel e o impacto da acreditação na revolução da atenção à saúde. Acreditação. 2012, 1 (2): 1-14.Google Scholar
  12. Braithwaite J, Westbrook J, Johnston B, Clark S, Brandon M, Banks M, Hughes C, Greenfield D, Pawsey M, Corbett A, Georgiou A, Callen J, Øvretveit J, Pope C, Suñol R, Shaw C, Debono D, Westbrook M, Hinchcliff R, Moldovan M: Strengthening organizational performance through accreditation research: the ACCREDIT project. BMC Res Notes 2011, 4:390..Google Scholar
  13. Greenfield D, Hinchcliff R, Moldovan M, Mumford V, Pawsey M, Westbrook JJB, Braithwaite J: A multi-method research investigation of consumer involvement in Australian health service accreditation programs: the ACCREDIT-SCI study protocol. BMJ Open 2012, 2:329..Google Scholar
  14. Hinchcliff R, Greenfield D, Moldovan M, Pawsey M, Mumford V, Westbrook JJB, Braithwaite J: Evaluation of current Australian health services accreditation processes (ACCREDIT-CAP): a protocol for a mixed method research project. BMJ Open 2012, 0:e001726..Google Scholar
  15. Mumford V, Greenfield D, Hinchcliff R, Moldovan M, Forde K, Westbrook J, Braithwaite J: Economic evaluation of Australian acute care accreditation (ACCREDIT-CBA [acute]): study protocol for a mixed-method research project. BMJ Open 2013, 3:e002381..Google Scholar
  16. Braithwaite J, Greenfield D, Westbrook J, Pawsey M, Westbrook M, Gibberd R, Naylor J, Nathan S, Robinson M, Runciman B, Jackson M, Travaglia J, Johnston B, Yen D, McDonald H, Low L, Redman S, Johnson B, Corbett A, Hennessy D, Clark J, Lancaster J: Health service accreditation as a predictor of clinical and organisational performance: a blinded, random, stratified study. Qual Saf Health Care. 2010, 19 (1): 14-21. 10.1136/qshc.2009.033928.View ArticlePubMedGoogle Scholar
  17. Greenfield D, Braithwaite J: Developing the evidence base for accreditation of healthcare organizations: a call for transparency and innovation. Qual Saf Health Care. 2009, 18: 162-163. 10.1136/qshc.2009.032359.View ArticlePubMedGoogle Scholar
  18. Greenfield D, Braithwaite J, Pawsey M: Health care accreditation surveyor styles typology. Int J Health Care Qual Assur. 2008, 21 (5): 435-443. 10.1108/09526860810890422.View ArticlePubMedGoogle Scholar
  19. Greenfield D, Braithwaite J, Pawsey M, Johnson B, Robinson M: Distributed leadership to mobilise capacity for accreditation research. J Health Organ Manage. 2009, 23 (2): 255-267. 10.1108/14777260910960975.View ArticleGoogle Scholar
  20. Greenfield D, Hinchcliff R, Westbrook M, Jones D, Low L, Johnston B, Banks M, Pawsey M, Moldovan M, Westbrook J, Braithwaite J: An empirical test of accreditation patient journey surveys: randomised trial. Int J Qual Health Care. 2012, 24 (5): 495-500. 10.1093/intqhc/mzs035.View ArticlePubMedGoogle Scholar
  21. Greenfield D, Moldovan M, Westbrook M, Jones D, Low L, Johnston B, Clark S, Banks M, Pawsey M, Hinchcliff R, Westbrook J, Braithwaite J: An empirical test of short notice surveys in two accreditation programs. Int J Qual Health Care. 2012, 24 (1): 65-71. 10.1093/intqhc/mzr074.View ArticlePubMedGoogle Scholar
  22. Greenfield D, Pawsey M, Braithwaite J: What motivates health professionals to engage in the accreditation of healthcare organizations?. Int J Qual Health Care. 2011, 23 (1): 8-14. 10.1093/intqhc/mzq069.View ArticlePubMedGoogle Scholar
  23. Greenfield D, Pawsey M, Naylor J, Braithwaite J: Are accreditation surveys reliable?. Int J Health Care Qual Assur. 2009, 22 (2): 105-116. 10.1108/09526860910944601.View ArticlePubMedGoogle Scholar
  24. Greenfield D, Pawsey M, Naylor J, Braithwaite J: Researching the reliability of accreditation survey teams: lessons learnt when things went awry. Health Inform Manage. 2013, 42 (1): 4-10.View ArticleGoogle Scholar
  25. Australia’s Health 2012. 2012, Australian Institute of Health and Welfare, CanberraGoogle Scholar
  26. Australian General Practice Accreditation Limited and Quality in Practice: Australian General Practice Accreditation Limited and Quality in Practice Pty Ltd Annual Report. In . Brisbane: 2012.Google Scholar

Copyright

© Greenfield et al.; licensee BioMed Central Ltd. 2014

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Advertisement