- Research article
- Open Access
- Open Peer Review
Sustainability of health information systems: a three-country qualitative study in southern Africa
BMC Health Services Researchvolume 17, Article number: 23 (2017)
Health information systems are central to strong health systems. They assist with patient and program management, quality improvement, disease surveillance, and strategic use of information. Many donors have worked to improve health information systems, particularly by supporting the introduction of electronic health information systems (EHIS), which are considered more responsive and more efficient than older, paper-based systems. As many donor-driven programs are increasing their focus on country ownership, sustainability of these investments is a key concern. This analysis explores the potential sustainability of EHIS investments in Malawi, Zambia and Zimbabwe, originally supported by the United States President’s Emergency Plan for AIDS Relief (PEPFAR).
Using a framework based on sustainability theories from the health systems literature, this analysis employs a qualitative case study methodology to highlight factors that may increase the likelihood that donor-supported initiatives will continue after the original support is modified or ends.
Findings highlight commonalities around possible determinants of sustainability. The study found that there is great optimism about the potential for EHIS, but the perceived risks may result in hesitancy to transition completely and parallel use of paper-based systems. Full stakeholder engagement is likely to be crucial for sustainability, as well as integration with other activities within the health system and those funded by development partners. The literature suggests that a sustainable system has clearly-defined goals around which stakeholders can rally, but this has not been achieved in the systems studied. The study also found that technical resource constraints – affecting system usage, maintenance, upgrades and repairs – may limit EHIS sustainability even if these other pillars were addressed.
The sustainability of EHIS faces many challenges, which could be addressed through systems’ technical design, stakeholder coordination, and the building of organizational capacity to maintain and enhance such systems. All of this requires time and attention, but is likely to enhance long-term outcomes.
Sustainability is a crucial aspect of a program’s life cycle: do activities and benefits continue after original support ends, and what aspects of a program’s design and activities help ensure such longevity? In the realm of global health, donor financing has driven the development and expansion of many initiatives, particularly in the fight against HIV/AIDS  – for example, the President’s Emergency Plan for AIDS Relief (PEPFAR), which is “the largest [development fund] by any nation to combat a single disease internationally”  and has committed over $65 billion to the HIV/AIDS pandemic since its inception in 2003 . There are numerous other significant sources of funding for HIV/AIDS programs, including other bilateral agencies, the Global Fund to Fight AIDS, Tuberculosis and Malaria, and private donors such as the Bill and Melinda Gates Foundation. However, recent evidence indicates that such donor funding has stagnated .
Health information systems (HIS)
The term health information system(s) (HIS) refers to the collection, storage, management, processing and transmission of information within the health sector , and includes things such as: district-level routine information systems, disease surveillance systems, laboratory systems, and/or human resource management information. Electronic HIS can allow for improved timeliness, use, legibility, and data quality, as well as for easier transmission between facilities. Health information systems are critical for a strong health system. They are used to improve disease surveillance, facilitate the strategic use of information, manage patients and programs, and increase service quality through more efficient and efficacious care [6, 7]. Internationally, there are increasing demands for accountability and transparency, so accurate and timely data are important for resource allocation and to monitor and evaluate initiatives’ effectiveness [5, 8].
Sustainability is the capacity to maintain program services after the end of financial, managerial, and technical assistance from external donors . Table 1 presents factors hypothesized to affect sustainability, as described in the literature, and this was the conceptual framework for this study [10–13]. Program-specific factors include the degree to which project goals are clearly specified and able to show results, the perceived effectiveness of the program in achieving these results, the availability of financing for the program, and the emphasis on training within the program. Relevant organizational factors include the flexibility to modify implementation to meet local needs and conditions, the participatory nature of donor-client and donor-community interactions, the existence of an effective project champion, the extent to which the program is integrated into the host institution and activities, and the institutional strength of the implementing agency. Contextual factors include the presence or absence of concurrent donor projects (either those that compete with the program or those that complement it), the receptivity of the community to participation, and the political, economic and cultural characteristics surrounding the project. Similar determinants have emerged as important for EHIS – for example, political commitment to the system, human resource and infrastructural constraints, physical and socioeconomic environment, alongside global determinants such as donor role, the technological environment, and institutional issues such as the project environment and knowledge management practices [14–17].
This study uses the above-described sustainability framework to inform a case study about the potential sustainability of electronic health information systems (EHIS). The study focuses on work in three sub-Saharan African countries supported by the Centers for Disease Control and Prevention (CDC) under PEPFAR. This article complements a broader emerging literature about the implementation, effectiveness and impact of EHIS in low-resource settings [18–20] including in Latin America [21–23], Asia [24–26] and sub-Saharan Africa [27, 28]. Few studies have emphasized the sustainability of EHIS, particularly with a multi-country comparison and in the context of declining donor support, as is presented here (with some noteworthy exceptions [21, 29]).
This was a qualitative study based on interviews with major stakeholders involved in ongoing donor-funded projects for strengthening HIV care through EHIS (as identified through collaborative discussions with in-country experts familiar with the local EHIS) in Malawi, Zambia and Zimbabwe. The interview guide (attached as an Additional file 1) was developed specifically for this study, and used open-ended questions to probe for details about the constructs shown in Table 1. The interview guide was informed by the sustainability framework, and questions were developed to probe different aspects of these constructs.
Study systems were selected among EHIS projects funded by PEPFAR via CDC, and were chosen to represent systems that had successfully achieved broad national or near-national implementation, and had sufficient tenure of operation to have generated multiple years of sustained exposure among stakeholders. The Zambian and Malawian EHIS are electronic medical record systems, designed and implemented primarily to manage clinical encounters, but also to inform a national data reporting system. The Zimbabwean human resources information system was designed and implemented at the national and provincial levels. A description of the study sites can be found in Table 2.
Project researchers from CDC and from Harvard T.H. Chan School of Public Health traveled to the study sites between July and November 2013. All data collectors had training and experience in conducting qualitative research, including data collection and analysis. The team held in-depth discussions with stakeholders from the government, health facilities and implementing partners. Example interviewee types included employees at government ministries, clinical and data/clerical staff at health facilities, and those involved “upstream” in the EHIS (such as software developers, managers, advisors and board members). These respondents were identified by experts in the three countries, and were purposively selected to ensure knowledge of the system and to offer a broad range of experiences with the system. The objective of these interviews was to document features of the EHIS and to identify progress toward country sustainability as per the sustainability framework (i.e., constructs in Table 1). In total, 58 interviews were conducted. Interviews were semi-structured, one-on-one with key informants, based on a standard study protocol, and lasted on average one hour. All participants were given a consent form and were asked to provide oral consent before beginning the interviews. Ethical approval for this study was obtained from CDC, the Harvard T.H. Chan School of Public Health, and host country government offices (Malawi, Zambia and Zimbabwe).
Interview notes were written by hand. These data were typed up, and were independently analyzed by the interviewers as subsequent interviews were still taking place. This allowed for initial analysis, for any emergent themes to be further evaluated, and served as a method of organizing interview transcripts. Additionally, by beginning to analyze transcripts while data collection is ongoing, researcher(s) can become grounded in the data and additional interviews can be modified to focus on better understanding weak themes and to validate unclear or new ideas brought forward in prior interviews .
Patterns were identified from the interview texts, first by inductive analysis for emergent themes, which were then grouped into the constructs of the sustainability framework. This initial coding and sorting of results was done independently by the 3 interviewers [CM, AS, CB] and then this group examined and discussed the preliminary thematic sets of results, particularly focusing on commonalities across all study countries and areas of difference, for example by system type. The narrative synthesis for each construct was written and then checked by revisiting the original interview text to ensure data were not de-contextualized, and that verbatim quotes were interpreted correctly based on their original context. There was also a member checking process, whereby the main results were discussed with the study participants to confirm information received by the interviewers. Other secondary data sources (i.e. program planning documents, progress reports) were also used to triangulate and ensure validity of the results.
This section details the main study findings as identified across the set of three case studies according to the sustainability framework. Findings are also summarized in Table 3. All findings are presented anonymously as stated in the research protocol and consent form.
Consensus around program vision was found to be a particular challenge for the two systems that involve a range of activities and a diverse set of users. For these EHIS, few users were able to articulate the full breadth of the systems’ stated goals and metrics, and stakeholders frequently perceived only those system goals that coincided with their own work duties. For example, service providers emphasized clinical applications, administrators focused on planning and resource allocation, and those involved in monitoring and evaluation prioritized indicator reporting. Meanwhile, many respondents with a more global understanding of the system indicated that it was unclear whose perceptions of effectiveness matter most, while others highlighted the complexity of responding to many goals with a single system. One respondent noted that “if we do not concentrate on what we have right now… we will spread ourselves so thin that we will have no effect.” These two EHIS were also perceived to lack explicit methods to evaluate progress towards program objectives. Combined with the compartmentalized nature of Ministries of Health, which tended to isolate health information within a single unit, respondents noted that the lack of monitored progress exacerbated the perception that systems’ roles and impacts were limited.
The importance of perceived effectiveness was a universal finding in the case studies. As one respondent noted, there is a “limit of faith” for the EHIS, and respondents in each of the three countries expressed concerns that system use would stop if users did not see tangible benefits. However, there was an apparent recognition that even promising systems may face early setbacks and challenges; this is partly the nature of information technology deployment and software development. Respondents clearly vocalized frustration with the systems that were being replaced – paper-based information systems were viewed as antiquated and burdensome – and expressed optimism that each of the three systems has potential to significantly improve management and delivery of health services. However, concerns regarding EHIS reliability led to the maintenance of parallel paper systems well after EHIS introduction. Maintenance of duplicate systems was perceived to increase workload and ultimately limit the perceived and actual effectiveness of the EHIS. Reflecting on setbacks that had resulted in the failure of previous systems, one respondent noted that similar issues continue to frustrate current efforts. He succinctly captured how optimism combines with concern, “it can change the way that … decisions are being made” in the country, but added “[I] hope someone will not tire [from these problems] along the way.”
Nearly all respondents acknowledged concerns about financing. Each of the systems relies (to a differing extent) on contributions from external sources, and there was a near-universal perception that the systems would not exist without substantial external financing. While external financing presents opportunities, this reliance was seen to carry risks. Donor-sponsorship comes with deliverables and some respondents voiced concerns that donors use their influence on the content and deployment of EHIS to meet their own priorities and deadlines. These factors may limit perceived effectiveness as well as championship and integration (see below). On the other hand, dependable donor funding was seen as a way to lessen pressure around securing additional funding, thereby enabling focus on the EHIS primary objectives. In one setting where the EHIS was seeking diversification of funding outside PEPFAR, stakeholders were nervous about the possible “mission creep” and the reporting requirements that would likely accompany any additional financing.
The study systems include notable examples of low-cost technologies that innovate around resource constraints, including alternative approaches to power and network connectivity issues (e.g., locally-adapted low-power screens). However, many stakeholders alluded to the high cost of even relatively inexpensive EHIS: “how could [country] justify paying for a computer system if we can’t even pay for medicines?” Another respondent from a different study country stated, “if donor funding were to leave I don’t see the government able to pay that. They can’t even afford our own salaries. Without you people, we are done.” Nonetheless, there is evidence of increasing national financial commitment to operating the systems in all three cases. Examples include paying for consumables such as printer ink, contributing to supporting infrastructure for the EHIS, and introducing a new cadre of information technology officers to assist in troubleshooting and maintenance of EHIS.
Respondents perceived training to bridge multiple aspects of sustainability. For example, institutional capacity is built through trainings. Also, large-scale training efforts are seen to increase exposure and buy-in to an EHIS, which may foster a champion and ease the system’s integration into core workplace activities. Perceived effectiveness may be strengthened if users have knowledge and tools to make system information useful to them. Some respondents noted the potential gains of training a variety of partners.
All three EHIS projects have conducted extensive training activities. Different models were described, from formal certification procedures to on-the-job peer-based trainings. Respondents viewed training as essential to building and bolstering the EHIS program, but reported challenges including limited baseline computer literacy, difficulty meeting the need for training (due both to rapid system scale-up and high turnover of users), the costs associated with travel and per diems for trainers and participants, the opportunity cost of lost clinical time, and possible knowledge dilution within peer-training models. Despite these challenges, the importance of such capacity-building was mentioned repeatedly. Trainings reportedly help build computer competency and comfort. It was reported that many trainees had never used a computer before – and in one illustrative example, a respondent spoke about using a computer for the first time during EHIS training, and her subsequent pride about then training peers on the system.
However, ongoing capacity gaps were reported by users of each of the three systems. In particular, stakeholders at all levels described an ongoing need for training that expands technical knowledge. Critically, all three study systems are run by implementing organizations outside of the government and these organizations remain the first responder for most troubleshooting. This was seem as perhaps diminishing system confidence and sustainability, particularly for users in remote areas where technical support may be slow to arrive. However, successful efforts to address this issue were identified: for example, one system trains “super-users” to locally resolve some technical issues, to decrease system downtime. In two of the study countries, the governments have made administrative commitments to taking on some technical support responsibilities – but in both instances capacity gaps remain, and the necessary financial and technical resources are not yet deployed.
It is essential to train those who will be entering data into the EHIS, but multiple respondents commented on the importance of also training managers and Ministry administrators about data use and quality because, to quote one respondent, “appreciation [of the system] is lacking among the senior management.”
Locally-modified EHIS – developed to meet local needs and with early and extensive engagement of in-country partners – may find greater traction and increased sustainability. To the extent that a consensus is reached early and effectively, this adaptation period was seen as helping to ensure the standardization of information and harmonization across stakeholders. To quote a respondent, “[the EHIS] is sustainable but only if we listen to the users first… Listen to them and then engineer.” A respondent in another country noted that there are “smart guys doing the software development, but they need provider input too.”
In all of the countries visited, stakeholders provided system and software feedback, although there were differences in the speed and uptake of this feedback. Notably, software deployment mechanisms varied – and in the countries where upgrades were done in-person by employees of the managing EHIS organization (which is time- and resource-consuming, relative to web-based downloads of the software which can be done at each site, for example, by trained “super-users” as described above) users expressed more frustration with the speed at which they saw system improvements.
Importantly, some respondents saw the newly-introduced EHIS as a donor priority at odds with local needs. To quote one respondent, “we are always skeptical about these sorts of projects… Programs come and distort the system, then we have to start over. . . But we try to be optimistic.” In another country, a program technician noted, “Right now, we worry that, without us going on and demanding the report, [the EHIS] won’t be there.” This overriding skepticism could reportedly undermine the transition away from locally-developed, long-standing paper-based information systems.
Despite these challenges, all sites reported strong working relationships with the donor agencies that support the EHIS, and participatory, collaborative efforts for planning and reporting. Such interactions also sometimes include donor membership on EHIS-relevant working groups and committees within the Ministry of Health. However, many stakeholders mentioned a desire for greater feedback from donors about organizational performance of the EHIS, emphasizing again the importance of a clear focus and measurable outcomes.
Many respondents noted weaker ties for donor-community interactions, where the “community” of an EHIS comprises its users (health workers, data clerks, and managers). In one study country, stakeholders perceived that end users saw the EHIS “as a black box” and might wonder “what’s the government up to?” Few respondents clearly acknowledged the donor source or donor role in the EHIS, and instead noted activities and policies as initiated by the Ministry of Health. These findings may reflect efforts to build buy-in and to institutionalize the systems within facility operations.
Identifying the ideal moment to engage system users was highlighted as a particular challenge in the study countries. Respondents felt that reaching out too soon led to unrealistic expectations – and if early iterations of the system could not meet these expectations, frustration and decreased buy-in. Several respondents recalled instances of systems that had strong early engagement but ultimately failed due to delays in procurement or in program updates. One respondent noted a history of challenges with system use, remarking that “we still have low confidence in the system… even where it is working well, people have doubts.” In another country, the current EHIS is a new iteration of earlier software that, in the words of a respondent, “died a natural death for lack of funds.” This early experience had a lasting impression, and the respondent now requires paper-based data collection in addition to the electronic entries, “in case this thing crashes.”
The EHIS in this study faced challenges with project champions. Respondents noted the importance of high-level engagement and commitment to new systems, especially ones as complex as EHIS—but respondents in all three countries expressed concern that such engagement is lacking. Respondents in one country noted “in meetings, you feel like the government is saying all the right things, but they are still not taking over,” and only with government ownership will they “feel totally responsible if the system fails.”
Championing may also occur “bottom-up.” In one instance the clinical staff at a large urban hospital helped co-develop the EHIS to respond to their needs, and the resulting system was thoroughly integrated to their continuum of care, resulting in widespread use by service providers. A respondent in a different study country noted that “facilities are the best champions. Once they commit, they work to make the system their own, to make it work for them, and to push it forward.”
The need for smooth integration within the broader health information system was highlighted often. In particular, stakeholders expressed a desire to complement ongoing data collection activities across levels of the health system, from the central Ministry of Health to facilities. Many countries have existing information systems for collecting health data, and respondents noted the need to build synergistic rather than duplicative systems. They saw parallel systems as adding workload, reinforcing multiple record-keeping mechanisms (e.g. paper- and computer-based), and detracting from the perceived efficacy of the EHIS.
In all three countries respondents wanted the systems to incorporate information from, and inform decision-making across, different departments. In one study country, facilities using the system throughout patient interactions reported decreased workload and increased care quality, thus increasing buy-in and fostering sustainability. In this same country other sites have not well-integrated the EHIS – one respondent noted that “at the talking level, there is talk” about integration but little actual progress to date. When integration is not achieved, the software essentially operates as a standalone data collection system entirely outside the care experience.
However, integration carries risks. This was highlighted by the many stakeholders who were concerned with spillover effects from potential discontinuation of an EHIS that had become fully embedded in the daily workflow. According to one respondent (at a facility where paper records were not well-maintained) the EHIS was embedded to the point that a “sudden stop would be a disaster.” Integration across systems was similarly seen as risky: if broader information systems are dependent on the EHIS, then a glitch in one component could have far-reaching effects on the country’s capacity to aggregate sector-wide data. In the words of one respondent, “if we keep [this EHIS] completely separate [from other reporting systems] then it can be easily divorced.”
Almost all study stakeholders emphasized the crucial role of institutional capacity of the relevant partner institutions. This need for capacity was highlighted across a spectrum of activities including developing the software, adopting and implementing the system at the facility level, and setting EHIS as a national priority. Computer literacy and comfort are essential for the successful deployment of EHIS. Respondents perceived that simpler and less technically-complex systems may face fewer capacity constraints, although many noted that this latter issue was rapidly losing importance due to overall increases in computer literacy. Software development and technical support were seen as more substantial challenges to systems’ sustainability. Concerns linked to these issues were raised in each of the three countries visited. The high cost and relative scarcity of strong software and information technology developers and support was a universal concern, and countries were forced to rely on donor support to finance these critical inputs to EHIS development and maintenance.
Concurrent projects/donor-supported activities
EHIS support did not occur in isolation in any of the countries visited. In all cases activities were strengthened by support for complementary projects – including broader EHIS activities as well as training on the use of data for decision-making. Importantly, the study EHIS was the nationally agreed-upon standard system in all three countries. This carried both opportunities and challenges. The lack of competing activities enabled easier standardization and harmonization across sites. This also eased portability of system knowledge and skills when trained users change job locations within the health system. However, early adoption of a single system reportedly introduced challenges in meeting local needs when sufficient consensus was not reached ex-ante. For example, when the primary system in one country failed to provide all data needed by the system’s many stakeholders, individual organizations began to lobby for the introduction of alternative electronic medical record systems. Notably, despite the official emphasis on a single system, the study team encountered a number of alternative data and information endeavors in this country, sometimes sponsored by key implementing partners of the study EHIS. While these other activities may have increased the availability of hardware and general computer literacy, as well as fostered a broader environment of data collection and use, there were cited instances of other software solutions “competing” with the study EHIS. This competition introduces additional challenges, notably the increased workload for health workers faced with multiple systems and weakened data compatibility.
The community of system users universally expressed receptivity to participation, often greeting the new EHIS with great excitement. EHIS were perceived to be a tangible tool for health system strengthening at the facility level and beyond. However, as noted above, this early excitement carries risk. Some stakeholders discussed facing increased pressure to add new sites, risking deployment of the EHIS at a faster rate than could reasonably be supported within current organizational structures and budgets. Additionally, some respondents emphasized the need to achieve buy-in at all levels, noting cases in which managers were excited about the EHIS but staff were not equally enthusiastic about adding the system to their workload. The new EHIS would create more work for already-overtaxed health workers since most sites continued to maintain paper-based records. It was mentioned several times that any users with initial hesitation around the system had them allayed once they saw tangible benefits of EHIS use – and when these benefits were not obvious, system use waned.
Political, economic and cultural characteristics
The three study sites have differing political, economic, and cultural characteristics. Nonetheless there are some commonalities in their health systems that may affect EHIS sustainability. First, the majority of care is delivered through the public sector so the Ministry of Health has a large role in planning and delivering services. Second, all are highly reliant on donor funding, particularly for the HIV sector. This was acknowledged by many respondents with challenges as discussed earlier. Third, the region has a health workforce shortage. Many stakeholders cited this as a challenge in EHIS implementation.
There were discussions about how broad contextual factors are likely to positively impact sustainability of an EHIS. Importantly, many stakeholders noted recent shifts in information technology. Familiarity with software has been cited as generally enabling EHIS implementation  and computer literacy has been rapidly increasing in southern Africa. There are expanded training opportunities at the university and certificate levels, and greater computer literacy and comfort in general. In two instances local clinical education programs added specialized training on the EHIS to the standard curriculum. Respondents also noted a broader push for electronic recordkeeping across all sectors. Attitudes towards data were also changing, with increasing focus on how data can be used to inform decision-making, and on the importance of protecting information and using it appropriately.
This study aimed to better understand how to build a promising environment for EHIS investments by exploring constructs from health systems sustainability frameworks. Many studies of health program sustainability have looked at traditional clinical or public health services, but EHIS have unique characteristics – from technical complexity to interactions with broader societal and technological trends – that may differentially affect program sustainability, especially in resource-poor settings.
These case studies highlight the importance of aligning perspectives across partners and across levels of the health system. Such harmonization will improve EHIS sustainability by aligning incentives and setting manageable expectations while more robustly integrating EHIS activities into the health system. Agreement on system vision and goals, and on how to measure these, was frequently noted by participants to be a crucial step in introducing a successful and sustainable EHIS. Accordingly, clarity and concordance on goals also enabled championing whether top-down or bottom-up; this has been found to be important in other instances of EHIS implementation as well .
Attaining such universal agreement on the mission and role of an EHIS may be an important goal, but it is a difficult one to achieve . Different partners may enable this in a variety of ways. Donors can encourage participatory and consensus-building activities to set comprehensive project goals and metrics. Governments should also take an active role in such processes, engaging stakeholders from different sectors, including clinical areas, monitoring and evaluation, and planning departments. Finally, EHIS organizations can strive for transparent strategic planning and monitoring processes. Other studies have cited institutionalizing EHIS and other information systems as an important aspect of sustainability , citing the risk of such projects remaining in “pilot mode” if they do not have a long-term, multi-partner implementation commitment .
System users can play a central role in project design, and their inclusion may help improve sustainability. A main objective of EHIS is to reduce burden on health workers. The additional burden of parallel paper- and computer-based systems can be, and has been, a key challenge to the introduction and implementation of these systems in these three countries, and has been cited elsewhere in the literature [24, 28]. Managers and policymakers in all three countries emphasized the systems’ potential for simplifying and accelerating data collection. However, those tasked with operating the systems frequently expressed frustration that the dual management of paper and electronic records resulted in duplication, rather than elimination, of efforts. Frustrations were especially high among health workers who had been promised that EHIS would streamline their workflow. Abandoning paper in favor of electronic recordkeeping requires a leap of faith, and EHIS implementers should request such a switch only when the system can reasonably handle this demand. Nonetheless, such a transition should be a near-term goal even if it can only be achieved incrementally by specific modules within the software or in only some geographic areas of a country. Respondents from each of the three countries emphasized the fact that system users must perceive tangible benefits if we expect them to use, let alone rely upon, the EHIS. The sooner this can happen, the more reasonable it is to move the health system towards computer-only information systems. Perceived usefulness has emerged in the broader literature as an important determinant of EHIS success .
The dynamic nature of the health workforce also emerged as an important issue. The low level of computer literacy within the health workforce is an important constraint to fully effective EHIS deployment. High staff turnover exacerbates this challenge, resulting in constant pressure for training on new EHIS systems. EHIS projects can mitigate the effects of this on system sustainability by giving health workers portable skills, for example, via improved pre-service trainings and standardized software and use protocols throughout the health system. This would reduce the need for constant re-training as health workers move around within the system. It is also crucial to increase public sector salaries to avoid losing trained staff (from EHIS developers to downstream health workers) to private sector employers also in need of their skills. Computer literacy is likely a waning challenge as more youth enter the workforce already familiar with computing, and EHIS trainings may need to adapt their focus from basic skills-building to improved use of technology for providing efficient and high-quality care. EHIS training should be viewed as having an important spillover effect in general increased computer comfort. Human resources has also been cited as a critical factor for successful EHIS implementation and sustainability in other studies from low-resource settings [18, 20, 22].
Lastly and particularly notably, this study highlights the importance of understanding sustainability as a concept that reaches beyond financial stability. There are many other essential determinants of programs’ likely perpetuation and donors should invest in activities that support these additional pillars of sustainability.
This assessment has some limitations. First, the three selected cases are a small sample, and were purposively selected for their successful broad national or near-national EHIS implementation and adoption. Therefore, these cases likely reflect stronger, more effective implementations of EHIS projects than average. Conversely, this arguably makes the experiences of this set of selected systems more important than more limited implementations of EHIS projects, given the high stakes for the health system if such widely-implemented systems are not sustained. All were funded by the same organization (PEPFAR). This eliminates variability in donor characteristics, which might have affected outcomes, but may limit the external validity and generalizability of the results. Additionally this assessment interviewed a limited number of stakeholders per country. It should be noted, however, that interview responses were highly consistent across countries. As a qualitative exercise, there is no statistical test on the internal validity of these results and no way to control for omitted variables or other sources of bias. The diverse study team participated in different ways throughout the research process (including study design, data collection and analysis) which may have helped mitigate this problem, but it cannot be eliminated completely. Lastly, these systems are undergoing many changes and this research captured only a “snapshot” of these efforts. An ideal study design would include future assessments including after a change or termination in funding, to empirically evaluate the predictions generated here.
Our goal was not to assess the merit of EHIS investments, which is a separate and important discussion that should be informed by national priorities as well as data from impact evaluations and cost-effectiveness analyses [31, 32]. These have been acknowledged to be largely lacking in the literature around EHIS . We encourage further study on these topics, as well as on other important EHIS-related issues such as security, data confidentiality, and the appropriateness of EHIS investments by donors given many competing program priorities and the potential for misalignment of needs among national and international stakeholders.
A broad range of stakeholders confirmed the importance of a number of sustainability determinants, both in guided interview questions corresponding to the framework, and in open-ended questions to elicit their own unprompted perceptions of critical determinants. These findings underscore the importance of creating an enabling environment for program sustainability, including by fostering communication between stakeholders for aligning perspectives and agreeing on a system’s goals, engaging users in the design and implementation process, and taking a broad view of sustainability that looks beyond financial dimensions to other important determinants. This will be critical for donor-led investments such as EHIS in low-resource settings. Achieving sustainability is a resource-intense endeavor, but will be necessary to ensure the long-term success of these programs and to see improved health outcomes.
Centers for Disease Control and Prevention
Electronic health information systems
Electronic medical record system
President’s Emergency Plan for AIDS Relief
Prevention of mother to child transmission
Ravishankar N, Gubbins P, Cooley RJ, Leach-Kemon K, Michaud CM, Jamison DT, Murray CJL. Financing of global health: tracking development assistance for health from 1990 to 2007. Lancet. 2009;373(9681):2113–24.
Office of the US Global AIDS Coordinator. The U.S. President’s emergency plan for AIDS relief: 2009. In: US Department of State, editor. Annual report to congress on PEPFAR program results. 2010.
PEPFAR Funding. [http://www.pepfar.gov/documents/organization/252516.pdf]. Accessed 29 Dec 2016.
Efficient and Sustainable HIV Responses: Case studies on country progress. [http://www.unaids.org/sites/default/files/media_asset/JC2450_case-studies-country-progress_en_0.pdf]. Accessed 29 Dec 2016.
World Health Organization, Health Metrics Network. Country health information systems: a review of the current situation and trends. 2011.
Design and implementation of health information systems. [http://apps.who.int/iris/bitstream/10665/42289/1/9241561998.pdf]. Accessed 29 Dec 2016.
Fryatt R. Investing in health. Lancet. 2014;383(9921):950.
Aqil A, Lippeveld T, Hozumi D. PRISM framework: a paradigm shift for designing, strengthening and evaluating routine health information systems. Health Policy Plan. 2009;24(3):217–28.
Shediac-Rizkallah MC, Bone LR. Planning for the sustainability of community-based health programs: conceptual frameworks and future directions for research, practice and policy. Health Educ Res. 1998;13(1):87–108.
Bossert TJ. Can they get along without us? Sustainability of donor-supported health projects in Central America and Africa. Soc Sci Med. 1990;30(9):1015–23.
Stirman SW, Kimberly J, Cook N, Calloway A, Castro F, Charns M. The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research. Implement Sci. 2012;7(17):1–19.
Gruen RL, Elliott JH, Nolan ML, Lawton PD, Parkhill A, McLaren CJ, Lavis JN. Sustainability science: an integrated approach for health-programme planning. Lancet. 2008;372(9649):1579–89.
Scheirer MA. Is sustainability possible? A review and commentary on empirical studies of program sustainability. Am J Eval. 2005;26(3):320–47.
Ludwick DA, Doucette J. Adopting electronic medical records in primary care: lessons learned from health information systems implementation experience in seven countries. Int J Med Inform. 2009;78(1):22–31.
Gordon AN, Hinson RE. Towards a sustainable framework for computer based health information systems (CHIS) for least developed countries (LDCs). Int J Health Care Qual Assur. 2007;20(6):532–44.
Garde S, Hovenga EJ, Gränz J, Foozonkhah S, Heard S. Managing archetypes for sustainable and semantically interoperable electronic health records. Electron J Health Inform. 2007;2(2):9.
Luna D, Almerares A, Mayan JC, González Bernaldo de Quirós F, Otero C. Health informatics in developing countries: going beyond pilot practices to sustainable implementations: a review of the current challenges. Healthcare Inform Res. 2014;20(1):3–10.
Fritz F, Tilahun B, Dugas M. Success criteria for electronic medical record implementations in low-resource settings: a systematic review. J Am Med Inform Assoc. 2015;22(2):479–88.
Tamrat T, Kachnowski S. Special delivery: an analysis of mHealth in maternal and newborn health programs and their outcomes around the world. Matern Child Health J. 2012;16(5):1092–101.
Oluoch T, Santas X, Kwaro D, Were M, Biondich P, Bailey C, Abu-Hanna A, de Keizer N. The effect of electronic medical record-based clinical decision support on HIV care in resource-constrained settings: a systematic review. Int J Med Inform. 2012;81(10):e83–92.
Farach N, Faba G, Julian S, Mejía F, Cabieses B, D’Agostino M, Cortinois AA. Stories from the field: the use of information and communication technologies to address the health needs of underserved populations in Latin America and the Caribbean. JMIR Public Health and Surveill. 2015;1(1), e1.
Fraser HS, Thomas D, Tomaylla J, Garcia N, Lecca L, Murray M, Becerra MC. Adaptation of a web-based, open source electronic medical record system platform to support a large study of tuberculosis epidemiology. BMC Med Inform Decis Mak. 2012;12(1):125.
Graven M, Allen P, Smith I, MacDonald NE. Decline in mortality with the Belize integrated patient-centred country wide health information system (BHIS) with embedded program management. Int J Med Inform. 2013;82(10):954–63.
Gera R, Muthusamy N, Bahulekar A, Sharma A, Singh P, Sekhar A, Singh V. An in-depth assessment of India’s Mother and Child Tracking System (MCTS) in Rajasthan and Uttar Pradesh. BMC Health Serv Res. 2015;15(1):315.
Wang J-Y, Ho H-Y, Chen J-D, Chai S, Tai C-J, Chen Y-F. Attitudes toward inter-hospital electronic patient record exchange: discrepancies among physicians, medical record staff, and patients. BMC Health Serv Res. 2015;15(1):264.
Ismail NI, Abdullah NH, Shamsudin A, Ariffin NAN. Implementation differences of hospital information system (HIS) in Malaysian public hospitals. Technology. 2013;20(21):22.
Cline GB, Luiz JM. Information technology systems in public sector health facilities in developing countries: the case of South Africa. BMC Med Inform Decis Mak. 2013;13(1):13.
Tilahun B, Fritz F. Comprehensive evaluation of electronic medical record system use and user satisfaction at five low-resource setting hospitals in Ethiopia. JMIR Med Inform. 2015;3(2):e22.
Kimaro H, Nhampossa J. The challenges of sustainability of health information systems in developing countries: comparative case studies of Mozambique and Tanzania. J Health Inform Dev Ctries. 2007;1:1.
Charmaz K. Grounded theory: objectivist and constructivist methods. In: Denzin NK, Lincoln YS, editors. Handbook of qualitative research. Thousand Oaks: Sage Publications; 2000. p. 509–36.
Were MC, Meslin EM. Ethics of implementing Electronic Health Records in developing countries: points to consider. In: AMIA Annual Symposium Proceedings: 2011. Washington DC: American Medical Informatics Association; 2011: 1499.
Driessen J, Cioffi M, Alide N, Landis-Lewis Z, Gamadzi G, Gadabu OJ, Douglas G. Modeling return on investment for an electronic medical record system in Lilongwe, Malawi. J Am Med Inform Assoc. 2013;20(4):743–8.
Fritz F, Kebede M, Tilahun B. The need for cost-benefit analyses of eHealth in low and middle-income countries. Stud Health Technol Inform. 2014;216:981.
This work could not have been completed without the support and instrumental assistance of staff from all three country CDC offices, the Ministries of Health and implementing organizations, including EHIS developers, in Malawi, Zambia and Zimbabwe. We are most grateful to all interviewees and study participants, as well as to Andrew Mitchell and Callae Snively for research assistance in early study phases.
This work was supported by the President’s Emergency Plan for AIDS Relief (PEPFAR) through the Centers for Disease Control and Prevention (CDC) under the terms of project number (200-2009-32280/0007). The funding body had no role in the design of the study, or collection, analysis, and/or interpretation of data, or in writing the manuscript. The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.
Availability of data and materials
These data may be identifiable so cannot be shared.
CM contributed to overall study design, participated in the qualitative research, and drafted the manuscript. AS contributed to overall study design, participated in the qualitative research, and contributed to the manuscript. CB participated in the qualitative research, and contributed to the manuscript. DG, PHK, NN and ZB facilitated site selection, coordinated in-country research procedures, and contributed to the manuscript. MSL and TB designed the original research project, led site selection, and provided feedback on the manuscript. All authors read and approved the final manuscript.
At the time of this study, AS, DG, PHK, NN, ZM and MSL were all employees of CDC.
The authors declare that they have no competing interests.
Consent to publish
Ethics approval and consent to participate
Ethical approval for this study was obtained from CDC, the Harvard T.H. Chan School of Public Health, and host country government offices (Malawi, Zambia and Zimbabwe). All participants were given a consent form and were asked to provide oral consent before beginning the interviews.
Health information system sustainability interview questions. (DOCX 18 kb)