This article has Open Peer Review reports available.
Using qualitative mixed methods to study small health care organizations while maximising trustworthiness and authenticity
© Phillips et al.; licensee BioMed Central Ltd. 2014
Received: 19 May 2014
Accepted: 24 October 2014
Published: 19 November 2014
The primary health care sector delivers the majority of health care in western countries through small, community-based organizations. However, research into these healthcare organizations is limited by the time constraints and pressure facing them, and the concern by staff that research is peripheral to their work. We developed Q-RARA-Qualitative Rapid Appraisal, Rigorous Analysis-to study small, primary health care organizations in a way that is efficient, acceptable to participants and methodologically rigorous.
Q-RARA comprises a site visit, semi-structured interviews, structured and unstructured observations, photographs, floor plans, and social scanning data. Data were collected over the course of one day per site and the qualitative analysis was integrated and iterative.
We found Q-RARA to be acceptable to participants and effective in collecting data on organizational function in multiple sites without disrupting the practice, while maintaining a balance between speed and trustworthiness.
The Q-RARA approach is capable of providing a richly textured, rigorous understanding of the processes of the primary care practice while also allowing researchers to develop an organizational perspective. For these reasons the approach is recommended for use in small-scale organizations both within and outside the primary health care sector.
The primary health care sector delivers the majority of health care in western countries through small community-based organizations. In Australia, the vast majority of primary care is delivered through general practice. These small, busy organizations, composed of shifting configurations of staff, vary according to their local contexts, their size, their modes of funding, and the types of services they provide. To understand the processes that underpin the delivery of quality primary care, we need to have a nuanced idea of how individuals work within these organizations. But the methodological toolkit to collect and analyse data on staff and organizational function and activities is rather limited. Much research into primary care has relied on quantitative, or single-method qualitative research. Although the importance of qualitative research methods in primary care has been emphasized since the mid-1990s, spearheaded by the British Medical Journal -, studies typically focus on interviews or focus groups as the primary data source. Single-method studies do not capture the richness and variety of organizational functioning, the ways that staff members interact and use their time, and the impact of the spatial environment of the organization on their work.
In this paper we present an overview of a mixed-methods approach to researching small-scale primary care organizations that is rapid and rigorous (Q-RARA - Qualitative Rapid Appraisal, Rigorous Approach). We developed this method to study nurses in general practice in Australia, at a time when nurses were moving into general practice in large numbers . We wished to understand the impacts of individual features of the practice, of the nurses themselves, of the other staff and of professional culture on the nurses’ roles and work. Ethnography offered one route to study emerging practices and actions within a small organizational culture, but we were concerned that the length of time to conduct organizational ethnography would mean that we would only be able to study a few sites. Q-RARA is a methodological design that is fast, flexible, and capable of producing a wide range of data from different perspectives with minimal disruption of the organization. Since conducting the Australian General Practice Nurses Study we have modified and streamlined the research design. We believe that Q-RARA is likely to be useful for research into other small scale organizations, and for bounded areas of large health organizations.
Several reasons have been put forward to explain the reliance on single-method studies, mainly interviews, focus groups and surveys in primary care research. As a rule, general practices are small businesses with few staff members all working under strict time constraints and pressure . Because of the competing demands for time in general practice, staff may view research as peripheral to the purpose of the organization . Under these circumstances, researchers may choose methods that can be conducted out of work hours or away from the organization.
An additional challenge for researchers is the diversity of primary care organizations in terms of structure, funding, and function within and between countries . Such plurality may make inference problematic if research is based on a limited number of field-sites.
Q-RARA draws on two main influences: rapid appraisal and qualitative mixed methods research design (or QUAL-qual methods). The approach takes into account the need to minimize the impact of conducting research in small organizations, while maximizing the capacity to produce rich, detailed contextual findings. In this article we present: (1) an overview of the background to the approach - rapid appraisal and qualitative mixed methods design (2) the approach itself, and (3) a critical discussion of the broader literature regarding mixed methods design and issues of rigor. We identify the strengths and weaknesses of the approach, as well as its potential contribution to researching primary care organizations by extending the use of a qualitative mixed methods design.
Antecedents of Q-RARA
Rapid appraisal was pioneered in the 1960s in the field of rural studies , but not used systematically in health-related fields until the 1980s when rapid social science data were collected by trained  and, more controversially, untrained  researchers, on illness profiles, the understanding of disease terms, and health practices. A parallel endeavour had existed for some time in public health, where communicable disease epidemiologists used quantitative rapid appraisal methods to investigate disease outbreaks .
Rapid appraisal has since been used in many other settings, including humanitarian crises, and its variations are now largely referred to under the banner of Rapid Evaluation and Assessment Methods (REAM) . REAM is an umbrella term that offers little detail about the processes involved in implementing a rapid appraisal approach, particularly using qualitative mixed methods. The epistemological underpinnings of REAM range from realist, objectivist epistemology (as in studies used to provide quick assessments of program performance ) to constructionism (in studies used to assess the impacts of policy  or roles ). The epistemological diversity of these studies reflects an under-theorization of methodology in this research area.
Qualitative mixed methods
The other antecedent for our method is mixed methods research. Mixed methods research is research that is informed by and situated along a spectrum of quantitative through to qualitative paradigms, and generally refers to the combination of quantitative and qualitative methods carried out “for the broad purposes of breadth and depth of understanding and corroboration” . It is not new in general practice, having first been advocated over thirty years ago ,. One of the caveats of this method in primary care, however, is that it can prove time-consuming and overwhelming for small institutions .
A particular strength of mixed methods research is the formal integration of individual methods at some point in the research process . Although integration produces a more detailed, richer understanding of the phenomena of interest, in health care research integration is frequently neglected . Furthermore, one of the pitfalls in mixed methods research is the conflation of integration with triangulation ,. In his influential book The Research Act, Denzin  popularized the concept of triangulation as “the combination of methodologies in the study of the same phenomenon” (p. 291). The many variants, processes and critical debate surrounding triangulation , are beyond the purpose and scope of this article. We acknowledge the necessity to engage with the epistemological assumptions of the relative methods employed in any given mixed methods study so that issues of commensurability are addressed.
Morse  challenged qualitative researchers to consider “if, when and how” the use of two methods from the same paradigm, referred to as QUAL-qual methods, can be considered mixed methods. QUAL-qual denotes a core project and a supplementary project whereby the latter cannot be a stand-alone project. According to Morse  the data types, levels of analysis, or participant perspectives of the core and supplementary components need “to be handled differently and to be kept apart” (p 491).
Q-RARA attempts to marry the rapidity and limited intrusiveness of rapid appraisal methods with the rigour and integration of mixed methods research. In this paper, we subject the quality and rigor of Q-RARA to scrutiny through an assessment of its performance against Lincoln and Guba’s well-established frameworks of trustworthiness and authenticity .
The research setting
The research approach was developed as the main mechanism to conduct the Australian General Practice Nurses Study. The methods are presented in detail here for the first time. We focus in our discussion of the methods on the ability to collect multiple forms of data, and the challenges of synthesis and quality assurance for QUAL-qual studies.
The AGPNS was conducted from 2005 to 2008. During this period there was a nationwide, government-led promotion of practice nurses in Australia including the provision of direct incentives, newly funded Medicare items, and the nursing workforce more than doubled to nearly 9,000 . This changing landscape of nursing resulted in a number of questions about the interplay of micro, meso and macro determinants on the role of nurses in primary care practice.
How do nurses operate within the structure of general practice?
What are the local, individual and structural factors that determine the role development of nurses in different general practice settings?
What contribution do practice nurses make to the safety and quality of general practice? and
How might the development of new models of practice nursing be facilitated?
Type and quantity of data collected from 25 family practices using the rapid QUAL-qual method
Interviews with nurses
36 Mean, 41 minutes
Explored life history, working roles, understanding of teamwork, experiences of a GP nurse, interactions with others, notions of quality practice
Interviews with doctors
24 Mean, 27 minutes
Explored history of practice and working life within it, roles of nurses within the practice, views of potential nurse roles
Interviews with practice managers
22 Mean, 26.5 minutes
Explored history of practice and working life within it, roles of nurses within the practice, views of potential nurse roles
Observation of nurse activity
34 nurses; 51 hours of observation
2 separate hour-long structured observation of a nurse’s activities
Photographs of nurse-identified important working sites
35 nurses; 205 photographs
Photographs taken of important working sites identified by nurses within the Practice
Maps of practice layout
7 hand-drawn & 18 printed floor plans
These plans located the nurse’s station and other key sites identified by nurses or observed by researcher
Field notes taken by researcher after each visit
Summary of staff numbers & working hours
Questionnaire filled out by practice manager
Details collected for each practice included: RRMA classification, distance from nearest acute hospital and community based services, number of regional general practices, allied health service availability, population data, and regional SES indicators such as unemployment rates
Recruitment and consent
Recruitment occurred via the then Divisions of General Practice in Victoria, NSW and the ACT, as the research team were based in Victoria and the ACT. Informed consent was obtained from both the practice as a whole for the mapping and overall observational aspects (signed by the practice principal after a briefing of the practice), and by individuals for the interviews and for the nurse observation components.
This research was approved by the Human Research Ethics Committees of the Australian National University and the Royal Australian College of General Practitioners.
The impact of the research conducted on the practices was minimized by the allocation of only one researcher to each research site and the completion of onsite data collection within one day. The two field researchers had disciplinary backgrounds in social science and community development and were not health professionals: a deliberate choice to optimize “naïve” or “fresh” observations of nurses in practices that were not obscured by prior professional experience. The field researchers scheduled the first observation early in the day because it demonstrated respect for the working hours of the practice and contributed to the development of rapport with staff. The background discussions prior to the visit and the daylong presence of a researcher meant that they were often invited to sit in on organizational activities, such as practice meetings or lunch breaks in the tearoom. The inclusion of the researchers in these ways demonstrated that the approach was acceptable to the research participants and contributed to the quality of the field notes.
In-depth interviews were conducted with practice staff to generate participant accounts and to assay different perspectives on the nurses’ roles. Practice nurses provided insight into their lived experience, practice managers were able to reflect on the administrative and managerial aspects of the nurses’ work, and general practitioners recounted their experiences both before and after they had decided to employ a nurse, and the impact of this decision.
Example of completed structured observation
TIME 9.15-10.15 am
Informal chat with research assistant
Receptionist chat about problem with another staff member
Opening mail, sorting for GPs, updating records on computer
Drug rep arrives to do a free check on blood pressure monitor
Steriliser beeping to signal end of cleaning cycle. She attends to it
Checked blood pressure on elderly patient
Shows me internal mail system
Receptionist inquiry about arrival time of another nurse
Continues to open mail
Conversation with drug rep about next visit
Talks about other nurse’s involvement in multi-site collaborative care program
GP checks his pigeon hole; inquires about our project
Continues to open mail
Immunization of 8-week old baby
Chat about immunization and paperwork for recall system. Keeping track of lapsed immunizations. Her role as educator for parents about immunizations
Introduces GP to me
Takes patient file to GP. Enters data onto computer to be added to immunization register
GP ducks in. Chat to pathology courier
Enters data on immunization onto computer
Immunization of 8-week old baby
Clears plastic dishes for immunization and puts them away
Adult male of penicillin injection. 2 years old for asthma medication. Education of mother on how to use spacer device
After a two-day training workshop, and in tandem with a chief investigator, each field researcher trialled Q-RARA to check the quality of their interview and observational skills. The concordance rate between the field researcher and the chief investigators for the time-motion studies was between 94% and 96%. As each site was visited, the data were double checked by other researchers, and the field researchers were able to debrief on the process of data collection. Many practice nurses reflected upon their activities to the researcher, resulting in a slowed-down version of the nurses’ activities over an hour. Even so, the number of activities and contacts undertaken by nurses was very high, with an average of 28 discrete activities per hour (range 6 to 60). In addition to activities, nurses sometimes moved rapidly between the consulting rooms responding to doctors, as well as communicating with patients in the waiting room. One nurse had 36 direct brief contacts with staff and patients in the hour, an average of one every 100 seconds.
Spatial and social contextual data
The floor plan was obtained or drawn by the observer, with patient areas and nurse workspaces marked in, and nurses were asked to identify key work areas to be photographed. The observer also collected written material from the family practice, such as practice information leaflets, and practice websites supplemented the other material. Finally, each practice was subject to social contextual mapping, which involved obtaining local demographic and contextual information about the availability of health services. Details collected for each practice included: geographical classification, distance from nearest acute hospital and community based services, number of regional family practices, allied health service availability, population data, and regional socio-economic indicators such as unemployment rates.
The researchers involved with the AGPNS had a range of disciplinary backgrounds including nursing, general practice, sociology and health policy, and they met regularly over twelve months to analyse the data and generate the findings. All data that were related to one general practice were combined to make a case. Analysis occurred in three stages: (1) All like data, such as interview data, were coded, (2) data that related to an individual general practice were examined as an intra-case study, and, (3) each practice was compared with all other practices via an inter-case study.
Interviews and field-notes were coded using NVivo 7.0 software (QSR International) by two researchers and two research assistants, employing a coding framework developed concurrently with the interview schedules. This framework was then continually revised throughout the analysis in response to research undertaken using emergent design.
Observational data were coded using a framework that distinguished tasks as administrative, clinical, servicing (e.g. stocking and sterilizing), or communicative (e.g. brief contacts with colleagues), and were then presented as pictograms. Each pictogram captured four distinct elements of the nurse’s work world: time (when), tasks (what), people (who) and locations (where), and the relationships between these elements. Producing each pictogram was an integral part of the analytic process . Our pictograms functioned both as a taxonomy-an elaborated list of all the meaningful elements within the sociocultural context of a practice nurse’s work-and as a concept chart that illustrated how the key concepts of time, tasks, people and locations relate to each other temporally, processually and locationally . The raw observational data were also coded into activity type and tasks funded or not funded through existing Medicare arrangements.
A researcher coded the practice floor plans and photographs using the principle that the floor plans functioned as a set for a theatrical performance, while the photographs gave insights into the way the set was used . The floor plans were classified first into different zoned areas (unrestricted public areas, restricted public areas, staff-only areas, and places of clinician/patient interaction). We then catalogued the ways in which nurses moved through the practice to describe the itineraries of nurses across different spaces, comparing these with the ways other staff and patients used these spaces.
Intra-case analysis. The purpose of this level of research was synoptic, because we sought coherences across the different data types to make a finding about a particular practice, and a nurse’s role in that practice. For example, comments about space in the interviews were matched to the floor plans and photos of nurse workplaces, and were considered in the light of the time and motion studies and researcher field-notes. The intra-case analyses were returned for comment to the participant practices to establish the credibility of our generated findings .
Inter-case analysis. Two team members from different disciplinary backgrounds working independently analysed in detail the data for each emergent finding, and subsequently presented their analyses to the rest of the team for discussion. A third team member was appointed to assist with the synthesis of data and identify discrepancies in the analysis; this prevented superficial synthesis of the data, or discordant analysis. The iterative approach of the multidisciplinary team drawing on a number of data sources was able to generate a more trustworthy interpretation than would have been achieved otherwise. An example of an evolving interpretation of data from the study from a range of disciplinary perspectives is presented in Case Study 1. In Case Study 2 we set out an example of integration of like analysis, and intracase and intercase analyses.
Case Study 1: Evolving Insights into the Social Geography of Nurses’ Stations in General Practice
The maps of general practices showed that nurses’ stations were located either in treatment rooms or in retrofitted areas like alcoves and areas off the reception desk. There were very few nurses who had their own dedicated consultation rooms.
Initial interpretation. Our initial interpretation of this allocation of social space was that it reinforced the marginalization of nurses in general practice. Although it was recognized that urban practices had to retrofit their spaces to incorporate nurses, doctors on the research team felt that not having their own rooms indicated that nurses were marginalized. This view was supported on the online discussion board by some nurses.
Second interpretation. The sociologist wondered if the location of nurses’ rooms reflected more than professional hierarchies. She noted that nurses’ stations were usually spatially central in the practice, and co-located with the patient bed upon which the most acutely ill patients would be located. The field researchers’ notes recorded numerous instances of nurses citing the treatment room as their ideal professional locale. Review of data showed that the treatment room and the waiting rooms were the two patient-centred spaces, and that nurses visited these the most.
Revision of second interpretation. Field researcher reflections noted the reluctance of nurses to enter general practitioner spaces, except to find things. However, observation data and their reflections indicated that a reverse recognition was occurring with doctors reluctant to enter the treatment room without permission if it was occupied by the nurse.
Final interpretation. Locating the nurse’s station in the treatment room (provided the treatment room was large enough) was not seen as inappropriate or demeaning by most nurses, but rather acted as a way of reinforcing their status as parallel clinicians within the practice.
Case Study 2: Interpretation and Synthesis of Like, Intra-Case and Inter-Case Data
Intra-case synthesis (Synthesised from interviews with nurses, practice manager and doctor, social scanning and RA notes). The nurse was new to general practice, and described herself as increasingly aware of the exigencies of small business life; hence, she had arranged for a visit from the drug rep in return for servicing the electronic blood pressure monitor. She and the GP had a collegiate relationship. He had directly recruited her from hospital to “do the things I don’t know about” (he instanced immunisations and wound dressings). The manager and the nurse had different ideas of the nursing role, with the manager wanting her to provide more services that would meet a need in the community and generate income (e. g. chronic disease clinics). The nurse described a number of educative activities - infection control, using guidelines - as key roles. The RA notes recorded that as the nurse took her photographs she commented rather ruefully on her work being in the “unseen places”.
Inter-case synthesis Comparisons across all practices were produced on lines of thought emerging from individual practices about nurses’ roles and their determinants. This case study produced as key lines of thought: “educator role”, “contribution to quality enhancement”, and “recognition”. The nurse’s workload was evolving towards education in addition to clinical work, without her employer recognising it. The drivers of connectivity identified in this study (central treatment room, permission to move across spaces, and multiple brief communications) are well-illustrated in this example. The case also illustrates the need to problematise the connectivity role. The amount of time devoted to e-connectivity may prevent the nurse from fully engaging in other work to expand her role.
Performance of rapid QUAL-qual Method against Guba and Lincoln’s trustworthiness criteria
Application in research
Extent to which findings accurately portray respondents’ constructions. Involves the following:
Prolonged engagement in targeted site to build rapport and trust between evaluators and setting members and provide evaluators with a deeper understanding of the relevant culture.
Prolonged engagement: One day site visits precluded prolonged engagement.
Persistent observation of site to provide sufficient understanding.
Persistent observation: Although it is recognized that persistent observation was not carried out for this study, the researchers attempted to respond to this criterion through repeated observation periods.
Peer debriefing: Extensive discussions of data and preliminary findings with one or more peers to refine thinking.
Peer debriefing: RAs provided field notes and reflections for each site visit and were able to debrief with Research Manager (SH) as often as necessary. Furthermore, the questions in the qualitative interview schedule were clarified and streamlined in response to feedback from a research assistant.
Negative case analysis: The constant reworking of hypotheses in light of disconfirming evidence.
Negative case analysis: In most practices, leadership was vested in the general practitioner, and nurses were relative newcomers to the practice. Specific analytical attention looking for difference was paid to one practice where the nurse was the senior clinician who had worked longest in the practice as a negative case.
Progressive subjectivity: Researchers identify and articulate any biases they hold, examine how their understandings shift during the project, and attend to how these biases might affect interpretations.
Progressive subjectivity: Assumptions were regularly challenged during fortnightly analysis meetings (see Case Study 1).
Member checks involve sharing and checking findings and interpretations with the people from whom the data were collected.
Member checks: Summaries of our research were returned to each practice for verification. We also presented our data at general practice and nursing conferences, and posted evolving data on a website developed for the project, inviting feedback from readers on the blog who were practice nurses on the summary de-identified findings and our interpretations. The Reference Group, formed at the commencement of the study, met over the course of the project and gave their feedback on topics specifically raised with them.
Researchers describe features of targeted context in detail and suggest additional contexts to which findings might be generalized.
Extensive background and case information included in final report.
Concerned with stability over time in researchers and methods. Assessed by means of a dependability audit, which involves reviewing project records to determine the extent to which project procedures and changes are documented.
This team included clinicians, academics and individuals engaged in organization policy and advocacy, who assisted in recruitment and in ensuring that the understanding of the project by the field sites was consistent. Regular meetings were held with all team members to monitor adherence to project procedures and to document changes in protocols.
The three chief investigators met regularly in person and via telephone, and a summary of the decisions made were routinely produced.
Extent to which findings are grounded in the data. Assessed by means of confirmability audits, which involve reviewing research records to determine if findings can be traced to data and data to original sources.
Kept all case-summary, substantive theme and pattern analysis documents.
Data and themes in all non-public documents were linked to subject IDs.
At regular team meetings to discuss the ongoing analysis, members were encouraged to look for the “black swans”, that is, evidence that might contradict the finding under discussion.
Performance of rapid QUAL-qual Method against Guba and Lincoln’s Authenticity Criteria
Application in research
Extent to which different stakeholder perspectives are elicited and taken into account. Involves identifying all stakeholders, soliciting their perspectives, and engaging in open negotiations with them around recommendations and future actions.
We interviewed people holding a range of roles in each practice: practice nurses, a general practitioner, the practice manager and a receptionist.
The Reference Group was particularly valuable in the final phases of the write-up, through the advice they gave on structuring the recommendations.
Extent to which stakeholders’ perceptions of the world have been improved or expanded.
The research gave “voice” to the participants by publishing and presenting information that they knew, but was not well understood or recognized more broadly.
Extent to which individuals have developed a better understanding of other stakeholders’ experiences and perspectives.
This work was distributed in many forms (peer review journals, conference, trade press articles etc.) to both nurses and doctors, and gave support to the notion that nurses play multiple functions, some under-recognized, by general practitioners and nurses.
Extent to which the research elicits action and change.
There is evidence at the macro level, that previously unnoticed element of practice nursing was the extent to which nurses were “educators” and yet “doctors tended not to recognize nurses’ educator [role] ... within the practice.” However, General Practice Education and Training, the national body responsible for preparing doctors for general practice, has now funded trials of practice nurses training general practice registrars.
Extent to which stakeholders feel empowered by the evaluation and by the ability to influence the actions taken.
The findings (e.g. six roles of nurses ) were taken up at multiple levels by nurses from the Chief Nurse in Department of Health and Ageing to individual practice nurses. Spin-offs have included further development of some of the under-recognized roles such as the nurse as educator role through specific project funding .
One of the clear challenges in assessing our approach against Guba and Lincoln’s criteria for trustworthiness was that they maintain that credible data are developed through prolonged engagement. Our method is predicated on short-term, intense engagement and cannot be compared with ethnography, which involves long-term (prolonged) engagement.
Despite their short period in the practice, research assistants were allowed access to the backstage of general practice. They were invited to observe practice meetings, accompanied nurses on their rounds when they left the practice, and followed her as she moved across all the spaces in the practice, including into doctor’s rooms. We considered that within a short-term engagement we had achieved a sense of engagement that was sufficient to meet the goals of the larger project in terms of data collection, but that this time frame may not be suitable for some other types of projects.
Against other criteria for the domain of trustworthiness the method performed well as outlined in Table 3.
The method was able to meet all the criteria contained within the domain of authenticity, as outlined in Table 4. The ability of this method to generate authentic data reflects, in part, its roots in rapid appraisal, a method that is pragmatic and purposive. On the other hand, the fact that the method also performed well against the trustworthiness criteria in general reflects the formalization of the mixed methods approach to frame rapid appraisal. An instructive comparison with our approach is research into the impact of financial incentives on clinical autonomy and internal motivation on family physicians in the UK . This ethnographic study developed detailed case studies of five British family practices at a time of structural policy change, including the introduction of the National Service Frameworks and the new General Medical Services contract. The study involved in-depth, long-term contact with each of the practices. The data from this study also scored highly on both trustworthiness and authenticity criteria, but the research itself was time-consuming and the generalizability of the data in some instances was limited.
The approach to using Q-RARA, developed through the AGPNS, demonstrated three main features: (1) a high degree of acceptability in the field, (2) the operationalization of a modified QUAL-qual method, and (3) quality and rigor. First, the approach was well received by the research participants, particularly the practice nurses who believed that the interest of the researchers validated their work. In common with traditional rapid appraisal methods, Q-RARA in general practices was carried out with minimal disruption to the research site while gathering a large quantity and range of data. Feedback from the nurses collected at the time and after leaving the site, indicated that Q-RARA did not disrupt their work unduly beyond the issues described above. It also has relevance for research in other small organizations, or can be employed, as Murray  has demonstrated, as a mechanism for public involvement in the collection of data about social and health needs in primary care.
Second, in terms of the modification of the QUAL-qual method, the original definition of mixed method design proposed by Morse and Neihaus , p. 9 stated that it “consists of a complete method (i.e. the core component), plus one (or more) incomplete method(s) (i.e. the supplementary component[s]) that cannot be published alone, within a single study”. Morse  went on to specify that the data types, level of analysis or participant perspectives must be sufficiently different that they need to be handled separately prior to integration. The stipulation that one of the qualitative methods must be seen as supplemental might imply an inadequacy in the primary method, and consequently, that the need for supplemental strategies arises from “a lack of clarity on the conceptual framework of the study” , p. 281. In contrast, the qualitative methods used in the AGPNS were driven by the research questions and conceived prior to the site visits. Our study is not alone in this regard with many other studies in the broader health sector employing more than one complete, qualitative method -.
In our approach we modified the QUAL-qual method by adding more than one standard qualitative method central to the study; both the in-depth interviews and the structured observation were core components. We suggest that the successful operationalization of this extended version of the QUAL-qual method may require alternate wording for the definition of mixed qualitative method design that addresses the concerns outlined above: QUAL-qual method design comprises several different qualitative methods, the choice of which is driven by the researchers’ epistemological position and theoretical perspective, and the research objectives. One or more of the methods may be able to generate sufficiently coherent and convincing findings to be published alone, but the authenticity and trustworthiness of the findings are increased by the planned use of additional research strategies that are insufficiently robust to generate defensible findings alone. The data types, level of analysis, or participant perspectives must be sufficiently different to warrant separate handling prior to integration. However, the findings generated by the different methods must be combined at some point in the research process.
The modifications to using the QUAL-qual method that were made in our approach are a development from existing rapid appraisal studies in primary care in that they directly engage with epistemology and integration, unlike other studies-such as Murray et al. , where the main research objective appears to be to advocate for the relevance of rapid appraisal methods in primary care, specifically interviews and focus groups or quantitative rapid appraisal methods, rather than a deeper engagement about method. Further to this, in studies where there is an exclusive qualitative mixed methods design, such as Manthorpe et al.  it is typical for details about individual methods to be described but a discussion of integration to be absent.
Third, the use of exploratory mixed methods provided a range of data that would have been missing had we limited ourselves to the original definition proposed by Morse ,. These diverse data, in conjunction with the multidisciplinary team and an iterative approach to data analysis allowed us to engage in a dialogue with the data and produce authentic and trustworthy findings. The location of the nurse’s station as a way of reinforcing her centrality in the general practice provides an example of an evolving authentic understanding that would not have been otherwise achieved. The dialogic engagement with the data also enabled us to articulate previously unrecognized elements of the social world, that of the practice nurses’ role as agent of connectivity. The QUAL-qual method also revealed the nuances of social life, particularly in the way that they relate to differences in power and authority. For instance, it became apparent that the general practitioners were unaware of the range of tasks undertaken by practice nurses and their corresponding skill set .
The conduct of the AGPNS and development of our approach, like any field-based method, can be derailed by major events. A site visit had to be curtailed because the general practice principal collapsed and had to be taken to hospital. Nevertheless, the field researcher was able to observe and comment on the way the organization functioned in a crisis. The most difficult component of data collection was securing an interview with the general practitioner due to time pressures on their work. Consequently, some interviews with GPs were conducted by telephone after the practice visit. In small general practices, GPs have the most difficulty disposing of their time freely. Our results suggest that if this method had focused on those with least flexible time schedules in the organization, we would have had to allow more time or additional participant-responsive methods, such as video, or specific attention to informal meeting sites, such as tea-rooms.
The intensive nature of the day-long visits to practices could be draining for the field researchers, who were required to collect a great deal of data in a concentrated period of time, without disrupting the practice or losing rapport with practice staff. The two field researchers travelled very long distances to visit the sites, and were initially over-scheduled. Sufficient time between site visits is needed to collate field-notes, to recover from the trip and to prepare for the next one. Consequently, the 25 site visits were undertaken over a period of four months, equivalent to one site visit by each researcher every 10 days.
We believe that the one-day visit design is defensible even though it means that intra-clinic variation in workload across the week was not captured. All staff were asked if the observation day was typical for them, and how they structured their working week. Interviews were able to flesh out and extend the insights gained through observation; thus although there would have been differences in the observational data on different days, these did not change the broad categories of role expressions and determinants that were identified in each practice.
We noted that the structured observation tool could be improved by using more advanced technology. Nurses could be trained to use personal digital assistants or smart phone applications to record their own time use patterns. This may result in more valid data on the time use of nurses  but at the loss of reflections-in-action, which the field researchers then used to refine the focus of their interview schedule. Alternatively, the field researchers could continue to record the data using tablet computers.
Focus groups were not included in our version of Q-RARA because of the very small size of the health care teams, and concerns that setting up a focus group would take away too many people from the workplace. We have subsequently used them in another study using Q-RARA where we were interested in the interaction between patients and volunteer-leaders in a falls prevention program .
The structured observations were also limited because the researcher was not permitted to observe patient-nurse interactions. This meant that our analysis under-represented clinical care aspects of the nurse’s role. However, the fact that the nurse-patient interactions were not observed made the presence of the researcher more acceptable to the family practice itself. A method of this nature, where the researcher is only present for a short period, does not allow the researcher to develop trusting relationships with patients as, for example, an ethnographer may be able to do. To observe patient-nurse interactions would have imposed the burden of explaining the study to successive patients upon the reception and nursing staff, and slowed down the work of the practice.
The dominance of interviews in qualitative general practice research typically drives researchers to explore one cadre within the practice, such as patients or healthcare providers. Consequently, there is a need for methods with proven utility that enable researchers to undertake studies of the organization itself and consider the health policy environment. We adapted and formalized REAM within a mixed-methods approach by taking the broad area of REAM and QUAL-qual methods, integrated these into a practical approach that is efficient, acceptable and rigorous and successfully trialled the resultant rapid QUAL-qual method in primary health care settings, an area that has seen little field-based research. We found our Q-RARA approach to be acceptable to participants and effective in collecting data on organizational function in multiple sites without disrupting the practice, or requiring the researcher to have a long-term presence in the practice. It was also able to strike a balance between speed and trustworthiness. Q-RARA seems capable of providing a richly textured rigorous understanding of the processes of the primary care practice while also allowing researchers to develop an organizational perspective. We have since used Q-RARA to explore community-based falls prevention activities, incorporating both movement-mapping and focus groups with patients.
For these reasons we believe the approach has merit for studying small-scale organizations both within and outside the primary health care sector, and bounded areas of larger health institutions.
CBP developed the methodology, CBP KD JH wrote the manuscript, CBP KD SH CP undertook the research and analysed the data, KD developed the quality assessment framework. All authors read and approved the manuscript.
This research was funded by the Australian Government Department of Health and Ageing, through the Australian Primary Health Care Research Institute (APHCRI). We are grateful to staff of the participating practices and to Hannah Walker and Daniel Sybaczynskyj for their research assistance. The opinions expressed in this paper are not the opinions of APHCRI or the Department of Health and Ageing.
- Mays N, Pope C: Qualitative research: observational methods in health care settings. Br Med J. 1995, 311: 182-184. 10.1136/bmj.311.6998.182.View ArticleGoogle Scholar
- Mays N, Pope C: Rigour and qualitative research. Br Med J. 1995, 311: 109-112. 10.1136/bmj.311.6997.109.View ArticleGoogle Scholar
- Pope C, Mays N: Reaching the parts other methods cannot reach: an introduction to qualitative methods in health and health services research. Br Med J. 1995, 311: 42-45. 10.1136/bmj.311.6996.42.View ArticleGoogle Scholar
- Phillips CB, Pearce CM, Hall S, Kljakovic M, Sibbald B, Dwan K, Porritt J, Yates R: Enhancing care, improving quality: the six roles of the general practice nurse. Med J Aust. 2009, 191 (2): 92-97.PubMedGoogle Scholar
- Baron RJ: What’s keeping us so busy in primary care? A snapshot from one practice. N Engl J Med. 2010, 362: 1632-1636. 10.1056/NEJMon0910793.View ArticlePubMedGoogle Scholar
- Rosemann T, Szecsenyi J: General practitioners’ attitudes towards research in primary care: qualitative results of a cross sectional study. BMC Fam Pract 2004, 5:31..Google Scholar
- Berman P: Organization of ambulatory care provision: a critical determinant of health system performance in developing countries. Bull World Health Organ. 2000, 78: 791-802.PubMedPubMed CentralGoogle Scholar
- Beebe J: Basic concepts and techniques of rapid appraisal. Hum Organ. 1995, 54: 42-51.View ArticleGoogle Scholar
- Anthropological Research: Process and Application. 1992, State University of New York Press, AlbanyGoogle Scholar
- Manderson L, Aaby P: An epidemic in the field: rapid assessment procedures and health research. Soc Sci Med. 1992, 35: 839-850. 10.1016/0277-9536(92)90098-B.View ArticlePubMedGoogle Scholar
- Goodman RA, Buehler JW, Koplan JP: The epidemiologic field investigation: science and judgment in public health practice. Am J Epidemiol. 1990, 132: 9-16.PubMedGoogle Scholar
- McNall M, Foster-Fishman PG: Methods of rapid evaluation, assessment, and appraisal. Am J Eval. 2007, 28: 151-168. 10.1177/1098214007300895.View ArticleGoogle Scholar
- Anker M, Guidotti RJ, Orzeszyna S, Sapirie SA, Thuriaux MC: Rapid evaluation methods (REM) of health services performance: methodological observations. Bull World Health Organ. 1993, 71: 15-21.PubMedPubMed CentralGoogle Scholar
- Harvey HD, Fleming P, Patterson M: A rapid appraisal method for reviewing the effectiveness of workplace smoking policies in large and medium sized organizations. J R Soc Promot Health. 2001, 121: 50-55. 10.1177/146642400112100111.View ArticlePubMedGoogle Scholar
- Nyanzi S, Manneh H, Walraven G: Traditional birth attendants in rural Gambia: beyond health to social cohesion. Afr J Reprod Health. 2007, 11: 43-56. 10.2307/30032487.View ArticlePubMedGoogle Scholar
- Johnson RB, Onwuegbuzie AJ, Turner LA: Toward a definition of mixed methods research. J Mixed Methods Res. 2007, 1: 112-133. 10.1177/1558689806298224.View ArticleGoogle Scholar
- Blake RL: Integrating quantitative and qualitative methods in family research. Fam Syst Med. 1989, 7: 411-427. 10.1037/h0089788.View ArticleGoogle Scholar
- Stange KC, Zyzanski SJ: Integrating qualitative and quantitative research methods. Fam Med. 1989, 21: 448-451.PubMedGoogle Scholar
- Stange KC, Miller WL, McWhinney I: Developing the knowledge base of family practice. Fam Med. 2001, 33: 286-297.PubMedGoogle Scholar
- Creswell JW, Fetters MD, Ivankova NV: Designing a mixed methods study in primary care. Ann Fam Med. 2004, 2: 7-12. 10.1370/afm.104.View ArticlePubMedPubMed CentralGoogle Scholar
- O’Cathain A, Murphy E, Nicholl J: The quality of mixed methods studies in health services research. J Health Serv Res Policy. 2008, 13: 92-98. 10.1258/jhsrp.2007.007074.View ArticlePubMedGoogle Scholar
- Docherty B: Using triangulation in health-care research . Prof Nurse. 2002, 16: 926-927.Google Scholar
- Foss C, Ellefsen B: The value of combining qualitative and quantitative approaches in nursing research by means of method triangulation. J Adv Nurs. 2002, 40: 242-248. 10.1046/j.1365-2648.2002.02366.x.View ArticlePubMedGoogle Scholar
- Denzin NK: The Research Act: A Theoretical Introduction to Sociological Methods. 1978, McGraw-Hill, New YorkGoogle Scholar
- Blaikie NW: A critique of the use of triangulation in social research. Qual Quantity. 1991, 25: 115-136. 10.1007/BF00145701.View ArticleGoogle Scholar
- Morse J: Mixing qualitative methods. Qual Health Res. 2009, 19: 1523-1524. 10.1177/1049732309349360.View ArticlePubMedGoogle Scholar
- Morse J: Simultaneous and sequential qualitative mixed method designs. Qual Inq. 2010, 16: 483-491. 10.1177/1077800410364741.View ArticleGoogle Scholar
- Lincoln YS, Guba EG: Naturalistic Inquiry. 1985, Sage Publications, Beverly Hills, CalifGoogle Scholar
- National Practice Nurse Workforce Survey Report 2009. 2009, AGPN, CanberraGoogle Scholar
- Pearce C, Hall S, Phillips CB, Dwan K, Yates R, Sibbald B: A spatial analysis of the expanding roles of nurses in general practice. BMC Nurs 2012, 11:13..Google Scholar
- Pearce C, Hall SH, Phillips CB: When policy meets the personal: general practice nurses in Australia . J Health Serv Res Policy. 2010, 15 (Supplement 2): 26-34. 10.1258/jhsrp.2009.009099.View ArticlePubMedGoogle Scholar
- Pearce C, Phillips C, Hall S, Sibbald B, Porritt J, Yates R, Dwan K, Kljakovic M: Contributions from the lifeworld: quality, caring and the general practice nurse. Qual Saf Health Care. 2009, 17 (1): 5-13.Google Scholar
- Pearce C, Phillips CB, Hall S, Sibbald B, Porritt J, Yates R, Dawn K, Kljakovic M: Following the funding trail: financing, nurses and teamwork in Australian general practice. BMC Health Serv Res 2011, 11:38..Google Scholar
- Phillips CB, Hall S: Nursing and the wise organization: techne and phronesis in Australian general practice. Nurs Inq. 2013, 20 (2): 121-132. 10.1111/j.1440-1800.2012.00601.x.View ArticlePubMedGoogle Scholar
- Phillips CB, Dwan K, Pearce C, Hall S, Porritt J, Yates R, Sibbald B: Time to talk, time to see: changing microeconomies of professional practice among nurses and doctors in Australian general practice. Contemp Nurse. 2007, 26: 137-144. 10.5172/conu.2007.26.1.136.View ArticleGoogle Scholar
- Harper D: Choosing a qualitative research method. Qualitative Research Methods in Mental Health and Psychotherapy: A Guide for Students and Practitioners. Edited by: Harper D, Thompson AR. 2011, Wiley-Blackwell, Hoboken, NJ, 10.1002/9781119973249.View ArticleGoogle Scholar
- Willig C: Perspectives on the epistemological bases for qualitative research. The APA Handbook of Research Methods in Psychology. Edited by: Cooper H. 2012, American Psychological Association, Washington, DCGoogle Scholar
- Lofland J, Lofland JH: Analyzing Social Settings: A Guide to Qualitative Observation and Analysis. 1995, Wadsworth, Belmont, CAGoogle Scholar
- Emmison M: The conceptualization and analysis of visual data. Qualitative Research: Theory, Method and Practice. Edited by: Silverman D. 2004, Sage, London, 246-265.Google Scholar
- Coast City Country General Practice Training Program: A Resource Kit for Practice Nurses who Teach. In; 2014.  Accessed 19 May 2014., [http://nursingkit.ccctraining.org/index.htm]
- McDonald R, Harrison S, Checkland K, Campbell SM, Roland M: Impact of financial incentives on clinical autonomy and internal motivation in primary care: ethnographic study. Br Med J 2007, 334:1357..Google Scholar
- Murray SA: Experiences with “rapid appraisal” in primary care: involving the public in assessing health needs, orientating staff, and educating medical students. Br Med J. 1999, 318: 440-444. 10.1136/bmj.318.7181.440.View ArticleGoogle Scholar
- Morse J, Niehaus L: Principles and Procedures of Mixed Methods Design. 2009, Left Coast Press, Walnut Creek, CAGoogle Scholar
- Nepal VP: On mixing qualitative methods. Qual Health Res. 2010, 20: 281-281. 10.1177/1049732309355717.View ArticlePubMedGoogle Scholar
- Checkland K: Management in general practice: the challenge of the new General Medical Services contract. Br J Gen Pract. 2004, 54: 734-739.PubMedPubMed CentralGoogle Scholar
- Nugus P, Greenfield D, Travaglia J, Braithwaite J: The politics of action research: “if you don’t like the way things are going, get off the bus”. Soc Sci Med. 2012, 75: 1946-1953. 10.1016/j.socscimed.2012.06.024.View ArticlePubMedGoogle Scholar
- Scott C, Walker J, White P, Lewith G: Forging convictions: the effects of active participation in a clinical trial. Soc Sci Med. 2011, 72: 2041-2048. 10.1016/j.socscimed.2011.04.021.View ArticlePubMedGoogle Scholar
- Murray SA, Tapson J, Turnbull L, McCallum J, Little A: Listening to local voices: adapting rapid appraisal to assess health and social needs in general practice. Br Med J. 1994, 1994 (308): 698-700. 10.1136/bmj.308.6930.698.View ArticleGoogle Scholar
- Manthorpe J, Clough R, Cornes M, Bright L, Moriarty J, Iliffe S: Four years on: the impact of the National Service Framework for older people on the experiences, expectations and views of older people. Age Ageing. 2007, 36: 501-507. 10.1093/ageing/afm078.View ArticlePubMedGoogle Scholar
- Bolsin S, Colson M, Patrick A, Freestone L, Creati B, Bent P: Factors contributing to successful incident reporting in anaesthesia. Br J Anaesth. 2011, 107: 473-474. 10.1093/bja/aer243.View ArticlePubMedGoogle Scholar
- Hall SJ, Phillips CB, Dubois L, Follett N, Pancaningtyas N: Preventing Falls, Promoting Health, Engaging Community. 2010, ANU Medical School, CanberraGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.