- Open Access
- Open Peer Review
Qualitative assessment of innovations in healthcare provision
BMC Health Services Researchvolume 9, Article number: 50 (2009)
The triad of quality, innovation and economic restraint is as important in health care as it is in the business world. There are many proposals for the assessment of quality and of economic restraints in health care but only a few address assessment of innovations. We propose a strategy and new structures to standardize the description of health care innovations and to quantify them.
Strategy and structure are based on the assumption that in the early phase of an innovation only data on the feasibility and possibly on the efficacy or effectiveness of an innovation can be expected. From the patient's perspective, benefit resulting from an innovation can be confirmed only in a later phase of development. Early indicators of patient's benefit will be surrogate parameters which correlate only weakly with the desired endpoints. After the innovation has been in use, there will be more evidence on correlations between surrogate parameters and the desired endpoints to provide evidence of the patient benefit. From an administrative perspective, this evidence can be considered in decisions about public financing. Different criteria are proposed for the assessment of innovations in prevention, diagnosis and therapy. For decisions on public financing a public fund for innovations may be helpful. Depending on the phase of innovation risk sharing models are proposed between manufacturers, private insurers and public funding.
Potential for patient benefit is always uncertain during early stages of innovations. This uncertainty decreases with increasing information on the effects of the innovation. Information about an innovation can be quantified, categorized and integrated into rational economic decisions.
The triad of quality, innovation, and economic restraint is as important in the provision of healthcare as it is in the business world. In economics, an innovation refers not only to products (like therapies and diagnostic measures), but also to procedures and the development of new markets . Transferring this understanding of innovation to a publicly-financed system is risky because forces that govern the free market are often not feasible in a publicly-financed system. We do not have a good understanding of how systems innovations can systematically, fairly, and transparently be transferred to healthcare provision. A Canadian proposal describes innovation in the healthcare system as the replacement of a previous method with a new approach that provides sustainable and competitive benefit. Benefit is sustainable and competitive if it keeps the organization economically healthy, introduces a rare or singular characteristic, and contains a learning process which enables the organization to achieve a leading position in the market .
According to the Veterans Affairs Health System, quality cannot be separated from innovation. Research results are immediately integrated into everyday clinical practice to either improve patient care or to increase the efficiency of the system . Also the British National Institute for Health and Clinical Excellence (NICE) is confronted with a similar problem. It has to make well-balanced decisions among quality, innovation, and value for money ). In Germany we are in the stage of developing a similar system. A rapid transfer of innovation to the daily provision of healthcare must be combined with a critical test of its validity. If a new method does not fulfill expectations, a search for a better method to replace it is likely. However, these examples are exceptions. In many countries practices, once introduced, may get continuing application without ongoing evaluation. In the United States as well as in Germany, the Food and Drug Administration (FDA) in the U.S. and the corresponding authority in Germany requires that pharmaceutical firms maintain phase IV monitoring studies. However, the concern in these evaluations is in both countries surveillance for harm, rather than continuing evaluation of efficacy. Lack of efficacy may be considered as economic harm.
Access to provision of healthcare innovations is ultimately regulated by reimbursement policies. This dichotomous decision to fund or not fund a particular innovation is problematic because of the benefit of the intervention is often uncertain. Such uncertainty creates two risks in these decisions – either healthcare services of little benefit are publicly financed or access to highly beneficial healthcare services is denied. It is difficult to exclude publicly-financed services which have been supported and promoting untested innovations is associated with far-reaching risks. We believe it is necessary to have regulation which minimizes both risks, the undesired financing of services of little benefit and the undesired curbing of innovations. Uncoupling consumers from medical progress and loss of the stimulating effect on competition are among the far-reaching risks of curbing innovation.
To reduce uncertainty in decision-making about public support of innovative treatments, standardized cost-benefit and cost-utility assessments of healthcare services should be required whenever possible. The methodical drawbacks of these procedures are well known, but are accepted because they still improve the decision process. Since this discussion, including the value assessment of new medications, is ongoing in Germany, many parties have suggested accepting the internationally recognized standard (with its known deficiencies) while simultaneously working on improved acceptable solutions [5, 6].
Comparing innovations is challenging because the benefit of different innovations cannot be evaluated with identical criteria. The evaluation of preventive measures  and of diagnostic measures  requires considerations which are too complex and too far removed from an acceptable solution to be applied in a standardized way. These limitations should be taken into consideration to avoid overestimating the practical relevance of criteria which we apply to assess innovations. We illustrate this with examples from prevention and diagnosis.
A preventive measure will be considered valuable if the achievement of the desired endpoint (like avoiding late consequences of a metabolic syndrome) can actually be confirmed. The benefit will be valued less highly if it can be proved that surrogate markers closely correlated with the endpoint (like a significant reduction in body mass index) is attained and maintained. If, however, there are no data describing the chances or attaining the endpoint, the benefit of this innovation will be considered small.
Diagnostic measures also require a discriminating evaluation because the benefit to the patient deriving from the diagnosis depends on a successful intervention for the problem that has been diagnosed. A number of hurdles lie between a correct diagnosis and a successful resolution of the problem in everyday clinical practice, like deducing the correct diagnostic result, correctly managing the accompanying illnesses, and achieving high patient compliance. In everyday practice it is rarely recognized that each of these hurdles must be successfully overcome to reach the desired goal. Intermediate goals are extremely important, but are not sufficient to serve as the basis of a benefit-oriented allocation of resources .
The solution may lie in a compromise – dividing the assessment of innovations into two parts, the evaluation of less and of more complex problems. Less complex problems could be evaluated by statements upon which agreement can be reached and which are considered to be supported by scientific theory and evidence. Complex problems cannot be solved by descriptions and explanations because there is no way to get the unequivocal evidence. They can only be solved by norms such as recommendations, guidelines or laws. These norms cannot be prescribed by scientists. Norms require deliberation beyond the scientific process and must include the public and policy makers. We refer to this process as democratic authorization. We as scientists shall herein restrict ourselves to suggestions for solving simpler problems, which can be supported by true, scientific statements.
We propose qualitative assessment of innovations in healthcare provision, although we are well aware that an evaluation of innovations, just as evaluation of the benefit of healthcare services, concerns normative, but not "true", statements and requires democratic authorization.
Qualitative Instead of Dichotomous Assessment of Innovations
Clinical decisions on innovations are problematic because we are uncertain whether the assumed advantages will materialize and the expected disadvantages will remain within the set limits. The data that justify the decisions are often unavailable before most medicines are licensed.
The risk of making incorrect decisions can be reduced if the final decision can be delayed until relevant data are available and a provisional decision based on existing information is offered. The better the available data the less will be the risk. The decisive factor is information that is based on the results and not on structures or processes which, in the final analysis, describe the quality of patient care. This point of view, which contains the three components, is shared by Portuguese authors , e.g., in "Inovacão, Evidencia e Mondo Real" (Innovation, Evidence, and the Real World), which describes the innovative idea, the data which confirm its feasibility, and the result obtained under everyday conditions.
We propose that new public agencies are required to oversee the funding of innovations. Current strategies have serious limitations. For example, the United States Food and Drug Administration primarily focussed on safety. Although they do require some continuing monitoring of pharmaceutical products, the FDA does not provide enough information for the comparison of relative effectiveness. Further, the FDA does not consider cost-effectiveness or many components of patient reported wellness.
One concern with contemporary approaches is that conflicts of interest are common. For example, members of committees that create guidelines often receive financial support from companies that make the products they are evaluating [11, 12].
We propose national agencies similar to the National Institute for Health and Clinical Excellence (NICE). These agencies would focus on clinical effectiveness and would require specific components in their evaluation. The agencies would provide clinical information and outcomes study evaluations long after products are licensed. Components of the evaluations would include clinical outcome studies, evaluations of the relevance of the outcome measures, and cost-utility assessments. Based on these data the agency would also make prioritization recommendations but no effective decisions for publicly supported programs. The effective decisions about public health care services have to be made by an agency with democratic authorization. Guidelines would be available to private payers who could make their own decisions about whether or not to apply the guidelines.
The public agencies would draw on peer review panels of experts and would have very strict conflict of interest disclosure requirements.
Components of Decision Process Used by Agencies
These three components can be allocated to two dimensions which we believe are appropriate to classify innovations. The two dimensions concern the positive perspective and the uncertainty about expected risks and benefits (Fig. 1).
The positive perspective, expressing the power of motivation [13–15], is the most important criterion of innovation. An innovation is attractive if it offers the prospective of a solution to a hitherto unsolved problem. The greater the belief that the new innovation will offer benefit, the greater will be the demand for the promised solution. Not infrequently however, the need and desire to offer a new therapeutic principle will motivate the use of interventions that are not accompanied by convincing data.
On several occasions it is impossible to separate placebo effects  on one hand from the desired increase in motivation i.e. directing the patient towards the attainment of her/his goal . A simple experiment performed at the Massachusetts Institute of Technology  helps illustrate this point. Students were randomly assigned to be informed that they had received an expensive painkiller or a cheaper version of the same painkiller. All of the subjects then received the same level of electric shock. Those told they had received the expensive medication perceived the shock as considerably weaker than their peers who believed they had taken the cheaper pill. Since both groups had actually been given placebos, this experiment confirms that the reality we perceive is substantially influenced by information and value judgments (e.g., expensive = better). The ethical evaluation of this experiment could influence its acceptance. We got the impression that colleagues who considered this experiment to be ethically problematic evaluated the information gained as less important than colleagues who accepted the ethical problem involved in this experiment. Those colleagues who accepted the terms of the experiment assume that the gain in information about the power of the placebo effect compensates the ethical problem because many incorrect conclusions could be avoided by this information, and risks to patients could thereby be reduced.
In summary, interest in an innovation increases the chances that the innovation will be evaluated positively. Rational considerations predict that the information, which induces a positive perspective, may not be evidence-based and may often lack validity. Neither the doctor who generates the positive perspective nor the patient who requests the positive perspective will be motivated to seek objective information that may discredit their faith in the innovation. Skepticism about the value of the innovation would destroy its power to mediate the positive perspective.
The second dimension of innovation is related to uncertainty of the results. In principle, it is relatively irrelevant whether the uncertainty is due to insufficient validity of the data or variability of the results. The important point is that only few reliable data are available at the time when a decision to either accept or reject an innovation must be made. In this case an attempt is made to use the few reliable data to support a statement that represents the best knowledge at the time the decision is being made
Description of Steps in Innovation
Since innovations in prevention, diagnosis, and therapy cannot be evaluated according to identical criteria, for the sake of clarity we are listing them separately, even if that inevitably leads to repetition. The individual steps in innovation differ in the information available to support its benefit and are different in preventive (Table 1), diagnostic (Table 2) and therapeutic innovations (Table 3).
In the first step of innovation, only data on the efficacy and on the feasibility of a measure would be required.
The second step would require confirmation of the benefit of the innovation with surrogate parameters which may only weakly correlate with the desired endpoints.
The third step, would require a high correlation between the surrogate parameters and the desired endpoints, i.e. evidence that the benefit of the innovation is sufficiently probable to justify public financing.
Justification for public financing of an innovation would require the steps outlines in Table 4. This evidence would be reviewed by the impartial public agency described above.
Assessment and support of innovations increases the efficiency of a healthcare system because resources would be directed toward those innovations that have the highest probability of producing benefit. This leads to a desired redistribution of the available resources. When evaluating innovations, the break even point, which is considered appropriate by society, must be identified. This is the threshold that identifies levels of benefit that the public is willing to pay for .
The program would require the creation of a special set aside fund to finance higher risk innovations. Partial financing could provide important information about the willingness of the insured to pay, and a comparative value judgment could be obtained. However, this information can only be gained at the expense of at least temporary, unequal access to this innovation. It is also possible to obtain information concerning willingness to pay on the part of healthcare providers. A risk-sharing model oriented to the expected successes of the innovation could be applied here. For reasons of fairness in decision making, either both forms of information gain or neither should be accepted.
In the interest of competition, the extent of financing cannot be determined prospectively. It will depend on the amount of available financial resources, the quality of the innovation, the intensity of the demand, and the willingness of the public to pay.
The steps in innovation describe the phases of information gain concerning the benefit of an innovation. In the beginning of these phases the only data available concern the effectiveness and the safety of the innovation. Data on benefit are typically not available. Safety and efficacy can justify licensure of an innovation but may not be enough to endorse public funding. Whether or not the innovation should be financed from public funds can only be decided when its benefit can be confirmed by a continuous gain in information. The goal of the innovation scale is to bridge this interval of indecision. It should ensure that promising innovations are available to all citizens, but remain publicly affordable.
Whether we approve of competition or not, stakeholders will decide which innovation prevails when the service of a different provider on the international market is better and more economical than that which the domestic provider can offer. Stakeholders can be the patients or purchasers of goods and services . Technical and commercial services will be evaluated according to different quality criteria than services which place high demands on communication and the supply of specialist competence. It is, therefore, appropriate to work out criteria to assess the value of healthcare services and to place these in relation to costs . Teleradiology and telepathology are services for which it should be relatively easy to develop uniform quality criteria for innovations.
It will be considerably more difficult to determine which methods should be used to decide whether something new should be recognized as an innovation. One recent debate considered whether the NICE Health Technology Assessment of therapy of the attention deficit/hyperactivity disorder (ADHD) met the demands of a well-balanced evaluation because NICE's economic model is dominated by cost differences [20, 21]. This decision is relatively simple in comparison to the questions of why quantitative research is preferred to qualitative research and the techniques are subject to a hierarchic value judgment . We know from experience that cultural differences, whether evidence based or not, are reality to patients, families and physicians. This is also true for innovations and the differences in recognizing them in different cultural circles. We need more discussion of criteria applied in different cultures to make decisions concerning the allocation of resources for innovations.
Finally, even a qualitative assessment of innovative healthcare services cannot conceal that successful reorganization of the system, which is often necessary for the implementation of innovations, involves the coordination and management of a complex process and not merely the realization of an isolated project .
Quantitative assessment of innovations may be a helpful step for economic decisions about innovations
We propose a three step model based on the best available information at different developmental stages of the innovation evaluation process
Decision about reimbursement should be based on reviews by an impartial outcomes assessment agency
A public fund should be established to support evidence-based innovations.
An example is presented to illustrate how risk sharing models between manufacturers, private insurers, and the public fund can be adapted as new data on efficiency and effectiveness become available.
Schumpeter JA: Business Cycles: A Theoretical, Historical, and Statistical Analysis of the Capitalist Process. 1939, McGraw-Hill Book Company, Inc., New York and London, 1
Macdonald M: Quality, Innovation, and Cost Containment. Can Nurse. 1993, 89: 15-17.
Kizer KW, Demakis JG, Feussner JR: Reinventing VA Health Care – Systematizing Quality improvement and Quality innovation. Medical Care. 2000, 38: 1-7.
Pearson SD, Rawlins MD: Quality, Innovation, and Value for Money. NICE and the British National Health Service. JAMA. 2005, 294: 2618-2622.
Jönsson B: IQWiG: an opportunity lost?. Eur J Health Econ. 2008, 9: 205-207.
Krauth C, John J, Aidelsburger P, Brüggenjürgen B, Hansmeier T, Hessel F, Kohlmann T, Moock J, Rothgang H, Schweikert B, Seitz R, Wasem J, AG Methoden der gesundheitsökonomischen Evaluation: Comments by the Working Group for Methods of Economic Evaluation in Health Care (AG MEG) to IQWiG's Draft Guidelines "Methods for Assessment of the Relation of Benefits to Costs in the German Statutory Health Care System [Abstract in English]. Gesundheitswesen. 2008, 70: 1-16.
Porzsolt F, Kirner A, Kaplan RM: Predictors of Successful Cancer prevention Programs. Recent Results Cancer Res. 2009, 181: 19-31.
Stengel D, Porzsolt F: Efficacy, Effectiveness and Efficiency of Diagnostic Technology. Optimizing Health – Improving the Value of Healthcare Delivery. Edited by: Porzsolt F, Kaplan RM. 2006, Springer, New York, 217-231.
Battalio RC, Kagel JH, Jiranyakul K: Testing between alternative models of choice under uncertainty: Some initial results. J Risk Uncertainty. 1990, 3: 25-50.
Uva MS: Innovation, evidence and the real world. A surgeon's perspective on the treatment of three-vessel or left main coronary artery disease with drug-eluting stents. Rev Port Cardiol. 2007, 26: 1199-1211.
Choudhry NK, Stelfox HAT, Detsky AS: Relationship between authors of clinical practice guidelines and the pharmaceutical industry. JAMA. 2002, 287: 612-617.
Ridker PM, Torres J: Reported outcomes in major cardiovascular clinical trials funded by for-profit and not-for-profit organizations: 2000–2005. JAMA. 2006, 295: 2270-2274.
Barrett B, Rakel D, Chewning B, Marchand L, Rabago D, Brown R, Scheder J, Schmidt R, Gern JE, Bone K, Thomas G, Barlow S, Bobula J: Rationale and methods for a trial assessing placebo, echinacea, and doctor-patient interaction in the common cold. Explore (NY). 2007, 3: 561-572.
Choi BC, Pak AW: Multidisciplinarity, interdisciplinarity, and transdisciplinarity in health research, services, education and policy: 2. Promotors, barriers, and strategies of enhancement. Clin Invest Med. 2007, 30 (6): E224-E232.
Knight KM, McGowan L, Dickens C, Bundy C: A systematic review of motivational interviewing in physical health care settings. Br J Health Psychol. 2006, 11: 319-332.
Waber RL, Shiv B, Carmon C, Ariely D: Commercial features of placebo and therapeutic efficacy. JAMA. 2008, 299: 1016-1017.
Woolf SH, Johnson RE: The Break-Even Point: When Medical Advances Are Less Important Than Improving the Fidelity With Which They Are Delivered. Ann Fam Med. 2005, 3: 545-552.
Sing SN, Wachter RM: Perspectives on Medical Outsourcing and Telemedicine – Rough Edges in a Flat World?. N Engl J Med. 2008, 358: 1622-1627.
Porter ME, Teisberg EO: How physicians can change the future of health care. JAMA. 2007, 297: 1103-1111.
Schlander M: Has NICE got it right? An international perspective considering the case of Technology Appraisal No. 98 by the National Institute for Health and Clinical Excellence (NICE). Curr Med Res Opin. 2008, 24: 951-966.
Schlander M: The NICE ADHD health technology assessment: A review and critique. Child Adolesc Psychiatry Ment Health. 2008, 2: 1-9.
Lambert H: Accounting for EBM: Notions of evidence in medicine. Soc Sci Med. 2006, 62 (11): 2633-2645.
Wang MC, Hyun JK, Harrison MI, Shortell SM, Fraser I: Redesigning health systems for quality: lessons from emerging practices. Jt Comm J Qual Patient Saf. 2006, 32 (11): 599-611.
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6963/9/50/prepub
We did not receive external funding for this project.
The authors declare that they have no competing interests.
Any of the authors contributed his essential part to this common product.
Authors’ original submitted files for images
Below are the links to the authors’ original submitted files for images.