- Research article
- Open Access
- Open Peer Review
Assessment of nursing home residents in Europe: the Services and Health for Elderly in Long TERm care (SHELTER) study
BMC Health Services Researchvolume 12, Article number: 5 (2012)
Aims of the present study are the following: 1. to describe the rationale and methodology of the Services and Health for Elderly in Long TERm care (SHELTER) study, a project funded by the European Union, aimed at implementing the interRAI instrument for Long Term Care Facilities (interRAI LTCF) as a tool to assess and gather uniform information about nursing home (NH) residents across different health systems in European countries; 2. to present the results about the test-retest and inter-rater reliability of the interRAI LTCF instrument translated into the languages of participating countries; 3 to illustrate the characteristics of NH residents at study entry.
A 12 months prospective cohort study was conducted in 57 NH in 7 EU countries (Czech Republic, England, Finland, France, Germany, Italy, The Netherlands) and 1 non EU country (Israel). Weighted kappa coefficients were used to evaluate the reliability of interRAI LTCF items.
Mean age of 4156 residents entering the study was 83.4 ± 9.4 years, 73% were female. ADL disability and cognitive impairment was observed in 81.3% and 68.0% of residents, respectively. Clinical complexity of residents was confirmed by a high prevalence of behavioral symptoms (27.5% of residents), falls (18.6%), pressure ulcers (10.4%), pain (36.0%) and urinary incontinence (73.5%). Overall, 197 of the 198 the items tested met or exceeded standard cut-offs for acceptable test-retest and inter-rater reliability after translation into the target languages.
The interRAI LTCF appears to be a reliable instrument. It enables the creation of databases that can be used to govern the provision of long-term care across different health systems in Europe, to answer relevant research and policy questions and to compare characteristics of NH residents across countries, languages and cultures.
About 20% of the population with functional limitations aged 65 and over living in European countries receives long-term care in an institution and about 30% receives formal care at home [1, 2]. However, the remaining 50% of individuals receive no formal care and either rely on informal care or receive no care at all. In the next decades, the population with functional limitations is projected to increase by about 120% and the population receiving formal care in institutions will rise by about 130% on average .
This rapidly increasing population with chronic disabling conditions will require long term support. It is of crucial importance to understand the health and social care needs of these people and to develop outcome measures for establishing effective and efficient programs and structures. Unlike epidemiological studies on acute hospital care and primary care activity, in long term care there are no sources of comparable data to inform regional or national policies. The issue is seldom the disease diagnosis but rather the functional and social consequences of health conditions [4, 5].
In the past decade, the AgeD in the HOme Care (ADHOC) study contributed to the creation of a cross-national database on community care for older people, enabling the possibility of examining quality, outcomes, and trends in home care in Europe [6, 7]. Similar evidence on institutional care (nursing home - NH) is lacking in Europe [8, 9]. One of the greatest obstacles is the absence of a validated methodology for routinely use in Europe . This need has been clearly identified by the European Union (EU) in its 7th Framework programme . Ideally, such a methodology would support the development of eligibility criteria, ensure standards of services, construct quality indicators, and result in a more efficient resource allocation to grant equal access to services to a rapidly increasing number of individuals.
In the past, a prototype of the current InterRAI instrument for Long Term Care Facilities (interRAI LTCF) - the so called Minimun Data Set - was developed as a comprehensive assessment tool to determine care needs, provide outcome information of NH residents, develop eligibility criteria, support monitoring of services delivery, construct quality measures, and result in a more efficient resource allocation . The new instrument (the interRAI LTCF) is a component of a suite of instruments built around a common set of assessment items relevant to a wide range of health care settings. The common items have identical definitions, observation time frames, and response codes and make these instruments an ideal tool to compare characteristics of older adults across settings .
In the US, the systematic use of the interRAI LTCF has led to the creation of a database that has proved of unprecedented value to the study of residents and services characteristics . The possible implementation of such an assessment methodology has never been considered in Europe, other than for research purposes in few countries.
The Services and Health for Elderly in Long TERm care (SHELTER) study addresses the aims of the 7th Framework Programme of the European Union and proposes to validate the interRAI LTCF as an instrument to assess the care needs and provision of care to residents of NH in Europe. The SHELTER study assesses validity and reliability of the InterRAI LTCF instrument when translated in languages of participating EU countries. In addition, the SHELTER study tests the implementation of the instrument on a large scale, leading to the creation of a unique database. These previously unavailable data will support measurements of residents outcomes, development of eligibility criteria, monitoring of services delivery and analysis of quality of care in a European NH setting. This manuscript describes the rationale and methodology of the SHELTER study, highlights the reliability of InterRAI LTCF instrument and illustrates the characteristics of NH residents in the SHELTER database. In addition, it describes a potential application of InterRAI instruments when used on a large scale by illustrating how they enable the comparison of the characteristics of NH residents in the SHELTER with those of older adults in home care in the ADHOC study.
The SHELTER study is a project funded by the Seventh Framework Programme of the European Union. The study was performed in 7 EU countries (Czech Republic, England, Finland, France, Germany, Italy, The Netherlands) and 1 non EU country (Israel). Several steps were followed in the study:
1. Applicability of the interRAI LTCF to languages of participating countries
The original version of the interRAI LTCF was translated from English version into 7 languages of participating countries (Czech, Dutch, Finnish, French, German, Hebrew, Italian) following several steps: forward translation, reconciliation, back translation, back translation review, cognitive debriefing, review of cognitive debriefing results and proof-reading. This procedure is in line with international recommendations [15, 16]. Following this step, in each country, a team of nurses, native speakers of the target languages, were asked to review items included in the translated versions of the interRAI LTCF and to judge whether they reflect the content they are intended to have. The team was composed by eight nurses in most countries, with the only exception of Germany where only five nurses were in the team.
2. Training of assessors
Assessors responsible for data collection were trained following a previously validated procedure . In each country, 2-day courses were organised to educate assessors about the concepts of comprehensive geriatric assessment and multidisciplinary teamwork and to train them to the use of interRAI LTCF. In the first part of the course main gerontological and geriatric problems were presented through classes, seminars and conferences. The concept of comprehensive geriatric assessment and multidisciplinary teamwork were introduced and multidimensional assessment tools were presented. In the second part of the course participants learned how to perform the assessment using the interRAI LTCF, including the specific forms and appropriate response codes, and to develop care planning.
Assessors were trained to use a variety of information sources, such as direct observation, interviews with the person under care, family, friends, or formal service providers, and review clinical records, both medical and nursing. The assessors were ordinary clinical staff, external research staff, or a mixture of both. Most assessors were nurses, but other professionals were also used. In line with interRAI's standard approach to coding, they were all instructed to exercise their best clinical judgment in order to record observations based on their evaluation of the most accurate information source.
3. Residents enrolment and follow up
In each country, study partners identified a sample of NH willing to participate to the study. This sample must not be considered randomly selected and is not intended to be representative of all NH in each country. The number of participating facilities in each country is presented in Figure 1.
Older adults residing in participating NH at the beginning of the study and those admitted in the 3 months enrolment period following the initiation of the study were assessed using the interRAI LTCF. In line with previous epidemiological research conducted using interRAI instruments in European populations [6, 18], enrolment of a target number of 4000 residents was planned. Participants enrolled in the study were re-assessed at 6 and 12 months if still in the facility. If no longer in the facility, reason (death, hospitalization, discharge at home or in another institution) and date of death or discharge were recorded.
Ethical approval for the study was obtained in all countries according to local regulations. Residents were invited to take part in the study and were free to decline participation. Consent was obtained with assurance of data confidentiality.
A random sample of residents was assessed on two separate occasions within a 2 days period to calculate test-retest reliability and inter-rater reliability. To assess test-retest reliability, the interRAI LTCF was administered to the same residents by the same individual assessors on two different occasions. To assess inter-rater reliability, two independent assessors administered the interRAI LTCF to the same residents independently at different times. Assessors were blinded to others' results and they were not permitted to discuss the case with each other, nor were they permitted to exchange information; however, they were both able to access the persons chart when completing their assessment.
5. Analytical Approach
Data from the baseline assessment for the entire study sample are reported. Multi-item summary scales embedded in the interRAI LTCF were used to measure residents characteristics. To evaluate functional status, the seven point MDS Activities of Daily Living (ADL) Hierarchy scale was used . The ADL Hierarchy scale groups activities of daily living according to the stage of the disablement process in which they occur. Early loss ADL's are assigned lower scores than late loss ADL's. The ADL Hierarchy Scale ranges from 0 (no impairment) to 6 (total dependence). ADL disability was categorized as follows: assistance required (ADL Hierarchy Scale score 2 to 4) and dependence (ADL Hierarchy Scale score ≥ 5). The cognitive performance scale (CPS) was used to assess cognitive status . The CPS combines information on memory impairment, level of consciousness, and executive function, with scores ranging from 0 (intact) to 6 (very severe impairment). The CPS has excellent comparability with the Mini Mental State Examination (MMSE) a score ≥ 2 being equivalent to a diagnosis of dementia or a MMSE score of ≤ 19 . Cognitive impairment was categorized as follows: moderate (CPS score 2 to 4) and severe (CPS score ≥ 5). The MDS Depression Rating Scale was used to assess the presence of depressive symptoms and a score ≥ 3 was used to diagnose depression . For all these scales, lower numbers represent less impairment.
The reliability of the categorical interRAI LTCF items was evaluated with weighted kappa coefficients using Fleiss-Cohen weights . According to Landis and Koch, kappa values below 0.40 should be considered poor, between 0.41 to 0.60 should be considered moderate, 0.61 to 0.80 should be considered substantial, and above 0.81 should be considered almost perfect . For continuous variables Pearson's correlation coefficients were presented.
6. AdHOC study sample
In order to describe potential application of InterRAI instruments when applied on a large scale characteristics of NH residents in the SHELTER study were compared with those of older adults in home care in the ADHOC study, a project funded by the Fifth Framework Programme of the EU, collecting data on a random sample of 4007 elderly patients admitted to the home care programs in urban areas of 11 European countries, from 2001 to 2003 . The main objective of the AD-HOC study was to identify a model of Home Care for the elderly through the analysis of the structural and organisational characteristics of Home Care Services in different countries of Europe, along with the clinical and functional characteristics of their clients. A trained staff was selected to assess home care patients entering the study by using the InterRAI instrument for Home Care (InterRAI HC). This instrument contains over 350 data elements including socio-demographic variables, numerous clinical items about both physical and cognitive status, as well as all clinical diagnoses. All items are coded using the same assessment approach as for the InterRAI LTCF; namely, assessors used all sources of information and then exercised clinical judgement as to the most appropriate answer based on standardized coding guidelines provided in the instrument's training manual. Most items permit the use of multiple information sources including personal interviews, review of the chart, direct observation of the person, communication with informal caregivers, and use of clinical communication between health care staff. Both InterRAI LTCF and InterRAI HC assessments include a set of common core data elements (e.g. outcomes assessing cognitive or functional status), as well as specialized items unique to that service setting.
In line with previous studies, to compare characteristics of home care patients collected by InterRAI instruments, CPS score and ADL hierarchy scale score are presented . These variables were shown to be strongly related to health outcomes of NH residents [25, 26].
A total number of 4156 residents was enrolled in the study, 500 from Czech Republic, 507 from England, 484 from Finland, 493 from France, 496 from Germany, 580 from Israel, 548 from Italy and 548 from the Netherlands. Table 1 summarizes the principal characteristics of NH residents enrolled in the study. Mean age was above 80 years and women represented approximately 3/4 of the sample. Disability was common, with more than 80% of participating residents requiring assistance or being dependent in ADL and cognitive impairment was present in more than 2/3 of the sample with 30% of residents being classified as severely impaired. Elevated level of "clinical" complexity of residents was confirmed by the high prevalence of urinary incontinence (73.5%), pain (36.0%), depression (32.0%), behavioral symptoms (27.5%), falls (18.6%) and pressure ulcers (10.4%).
Test-retest and Inter-rater reliability
Overall, reliability of 201 InterRAI LTCF items was tested. Only 3 items (walking speed, height and weight) were continuous variables. Table 2 presents average weighted kappa values for categorical items in different assessment areas.
Test-retest reliability was evaluated in 380 residents by 48 assessors (nurses: 42; MD: 5, psychologist: 1) in 17 facilities. This sample was composed of 52 dual assessments from Italy, 50 from Czech Republic, Germany and Israel, 48 from England and France, 43 from Finland and 39 from the Netherlands. As shown in table 2 test retest reliability of InterRAI LTCF items in each of the assessment areas ranged from 0.75 to 0.92. Average weighted kappa for the 198 categorical items was 0.83. None of the single items in the InterRAI LTCF presented with a kappa value below 0.40. Pearson's correlation coefficients for items based on continuous variables were 0.96 for walking speed, 0.92 for height and 0.96 for weight.
Inter-rater reliability was evaluated in 404 residents by 76 assessors (nurses: 68; MD: 4; psychologists: 4) in 23 facilities. This sample was composed of 60 dual assessments from Israel, 59 from Italy, 50 from England, 49 from France and Czech Republic, 48 from Germany and Finland and 41 from the Netherlands. As shown in Table 2 inter-rater reliability of items in the assessment areas ranged from 0.64 to 0.91. Average weighted kappa for the 198 categorical items was 0.74. Only one single item had a kappa value below 0.40 ('Fluid output exceeds input' included in the Oral/Nutritional Status assessment area). Pearson's correlation coefficients for items based on continuous variables were 0.88 for walking speed, 0.99 for height and 0.89 for weight.
Comparison of SHELTER and ADHOC samples
Figure 2 compares cognitive and physical functional characteristics (as measured by the CPS scale score and the ADL hierarchy scale score, respectively) of NH residents participating in the SHELTER study and home care recipients participating in the ADHOC study. Both these samples were assessed with InterRAI instruments.
NH residents in the SHELTER project showed higher cognitive and physical impairment than home care recipients in the ADHOC study. In all the countries participating in both studies, a higher level of cognitive and physical impairment was seen in NH residents compared with home care recipients. Interestingly, home care recipients in Italy and France had levels of cognitive and physical impairment similar to the NH residents in the Netherlands, Czech Republic and Germany.
The SHELTER study shows that the interRAI LTCF 1. is a reliable instrument which can be used to assess characteristics of NH residents; 2. can be used to compare characteristics of NH residents across Europe; 3. may contribute to the collection of information and to the creation of large data sets.
InterRAI LTCF was designed to assist clinicians in assessing the care needs of NH residents and providing a comprehensive view of the needs and strengths of the NH population, a view that is essential to the development of appropriate plans of care and allocation of resources . It shares with other interRAI instruments a common set of assessment items that are considered to be important in all care settings . This methodology may improve continuity of care and provide the essential basis for the most effective use of information technology . The use of these standardized instruments gives an opportunity to physicians, therapists, nurses, families, advocates, administrators, and public payers to track changes in an older adult's status across settings and over time. Completion of a full assessment based on InterRAI instruments may take between 20 and 90 minutes depending on setting and assessor experience with the use of the instrument: in particular it has been estimated that between 60 and 90 minutes are necessary to complete an initial InterRAI LTCF assessment [17, 29]. Each of these instruments has been developed to be rooted in care planning through a triggering system that enables the identification of the person's problems, but also provides information for monitoring quality of care, case-mix systems, and eligibility screeners.
Virtually all of the items tested met or exceeded standard cut-offs for acceptable test-retest and inter-rater reliability after translation into target languages and a substantial proportion of items showed excellent reliability. These results demonstrate that when implemented by appropriately trained and motivated staff the interRAI LTCF can provide high quality assessment as part of normal clinical practice. Similar reliability levels were found in previous studies performed using interRAI instruments in different settings, suggesting that these instruments, when appropriately translated and validated can be applied in different settings and geographic regions [30–33].
In a recent study Hirdes and coll. tested inter-rater reliability of InterRAI LTCF in 246 nursing home residents, finding kappa values ranging from 0.63 and 0.96. Interestingly, in line with the present study, the items with the best mean kappa values were continence (k = 0.90), physical functioning (k = 0.87) and medications (0.91) . The reliability of InterRAI instruments was also excellent when tested in other settings, including home care (average weighted kappa of items in the instrument = 0.74) , acute care hospitals (k = 0.82)  palliative care (k = 0.80)  and postacute care (k = 0.73) .
To assess inter-rater reliability, in line with previous publications [30–33], we have chosen to have independent assessors rating different patients in different facilities and countries, rather than having all study assessors in different countries evaluating the same patient (i.e. by the use of video recordings or clinical vignettes). This approach was chosen in order to minimize potential problems related to the creation of standardized assessment material in different study languages (video recordings or clinical vignettes).
The use of a standardized methodology enables comparisons of older adults across regions and settings. As shown in the manuscript (Figure 2), data of home care recipients collected by interRAI community care assessment instrument (interRAI HC) in the ADHOC study can be directly compared with those of NH residents in the SHELTER study. The SHELTER sample is not meant to be nationally representative and therefore residents characteristics can not be generalized to all NH residents in European countries. However, the data presented indicate the potential of interRAI instruments when applied on a large scale.
The International Classification of Function, Disability and Health (ICF) is intended to provide comparisons of physical, mental and social function, the factors that are the expression of impact of chronic disease . However to date, the absence of reliable assessment systems that can deliver ICF classification has meant that there has been no means of generating population Ievel ICF data. SHELTER has shown that the interRAI LTCF is a reliable tool in EU NH. ADHOC demonstrated the same for EU recipients of community care. Future research should explore the extent to which ICF classifications can be created from these assessment systems to enable the generation of internationally comparable epidemiological data in our aging populations characterized by the increasing prevalence of chronic diseases .
Systematic implementation of interRAI LTCF can contribute to the collection of information and to the creation of large data sets. InterRAI instrument based data sets provide a resource to support the analysis of crucial issues pertinent to the clinical care of very old, complex, chronically ill patients living in different settings, especially in NH and in the community. As these individuals are generally excluded from clinical trials, physicians and the health personnel have to rely their best judgment when making decisions. Linking reliable and valid outcome measures constructed with the interRAI LTCF items to the interventions received--pharmacological and nonpharmacological-- opens a new field of outcome oriented geriatric research relevant to scientists, clinicians, and administrators .
The data sets have become an important resource for filling knowledge gaps in these understudied populations. Older adults with multiple chronic conditions are still systematically excluded from clinical trials that inform the evidence base for medicine. Nowadays, these clinically relevant data sets are commonly used to conduct observational studies which, when appropriately performed, have proved as valid and useful as randomized clinical studies [37–40]. The creation of a SHELTER database containing data from the interRAI LTCF assessment and information on medications may enabled the exploration of a breadth of clinical questions regarding syndromes such as falls, pain, prevalent medical conditions and geriatric syndromes. Data in the SHELTER database may be useful to identify prognostic factors, and assess patients outcomes and quality indicators Results based these analyses may provide guidance to develop programs to improve quality of care in NH.
In conclusion, the SHELTER study validates the interRAI LTCF as a methodology that can be used not only to assist clinicians in performing a comprehensive assessment of NH patients, but also to compare characteristics of older adults across countries and to collect information and create databases that can be used to answer relevant questions in the field of long term care. This new methodology can be applied on a large scale to analyze the provision of long-term care across European health systems.
Czech Republic: Fialova D, Madlova P, Vlachova M, Ramadhani Chamradova MK. Germany: Denkinger M, Lukas A, Rissman U. England: Huang Y. Finland: Alanen HM, Björkman M, Hosia-Randell H, Elomaa-Sulkava U, Muurinen S, Noro A, Pitkälä K. France: Pasek J, Dequesne M, Armand MT, Berrabah MM, Logé O, Le Dastumer MD, Dugny G, Champvert P. Israel: Epstein S, Levi S, Ben-Israel S. Italy: Danese P, Antocicco M, Fusco D, Menculini G, Borroni G. The Netherlands: van der Roest H
Demographic change in the euro area: projections and consequences. European Central Bank Monthly Bulletin. 2006, 49-64.
Giannakouris K: Ageing characterises the demographic perspectives of the European societies. Eurostat Statistics in focus 72. 2008, 1-11.
Pickard L, Comas-Herrera A, Costa-Font J, Gori C, di Maio A, Patxot C, Pozzi A, Rothgang H, Wittenberg R: Modelling an entitlement to long-term care services for older people in Europe: projections for long-term care expenditure to 2050. Journal of European Social Policy. 2007, 17: 33-48.
Tolson D, Rolland Y, Andrieu S, Aquino JP, Beard J, Benetos A, Berrut G, Coll-Planas L, Dong B, Forette F, Franco A, Franzoni S, Salvà A, Swagerty D, Trabucchi M, Vellas B, Volicer L, Morley JE, The International Association of Gerontology and Geriatrics/World Health Organization/Society Française de Gérontologie et de Gériatrie Task Force: International association of gerontology and geriatrics: a global agenda for clinical research and quality of care in nursing homes. J Am Med Dir Assoc. 2011, 12: 184-9.
Rolland Y, Abellan van Kan G, Hermabessiere S, Gerard S, Guyonnet Gillette S, Vellas B: Descriptive study of nursing home residents from the REHPA network. J Nutr Health Aging. 2009, 13: 679-83.
Carpenter I, Gambassi G, Topinkova E, Schroll M, Finne-Soveri H, Henrard JC, Garms-Homolova V, Jonsson P, Frijters D, Ljunggren G, Sørbye LW, Wagner C, Onder G, Pedone C, Bernabei R: Community care in Europe. The Aged in Home Care project (AdHOC). Aging Clin Exp Res. 2004, 16: 259-69.
Onder G, Liperoti R, Soldato M, Carpenter I, Steel K, Bernabei R, Landi F: Case management and risk of nursing home admission for older adults in home care: results of the AgeD in HOme Care Study. J Am Geriatr Soc. 2007, 55: 439-44.
OECD Long-Term Care for Older People. 2005, Paris: OECD
Driest PF: Long term care in Europe. Edited by: Hassink J, van Dijk M. 2006, "Farming for health", Springer:101-6
Lundsgaard J: Consumer Direction and Choice in Long-Term Care for Older Persons, Including Payments for Informal Care: How Can it Help Improve Care Outcomes, Employment and Fiscal Sustainability?. OECD Health Working Papers. 2005, 20, OECD Publishing
European Commission: Cordis. [http://cordis.europa.eu/fp7/health/home_en.html]
Mor V: A comprehensive clinical assessment tool to inform policy and practice: applications of the minimum data set. Med Care. 2004, 42 (4 Suppl): III50-9.
Bernabei R, Landi F, Onder G, Liperoti R, Gambassi G: Second and third generation assessment instruments: the birth of standardization in geriatric care. J Gerontol A Biol Sci Med Sci. 2008, 63: 308-13.
Gambassi G, Landi F, Peng L, Brostrup-Jensen C, Calore K, Hiris J, Lipsitz L, Mor V, Bernabei R: Validity of diagnostic and drug data in standardized nursing home resident assessments: potential for geriatric pharmacoepidemiology. SAGE Study Group. Systematic Assessment of Geriatric drug use via Epidemiology. Med Care. 1998, 36: 167-79.
Wild D, Grove A, Martin M, Eremenco S, McElroy S, Verjee-Lorenz A, Erikson P, ISPOR Task Force for Translation and Cultural Adaptation: Principles of Good Practice for the Translation and Cultural Adaptation Process for Patient-Reported Outcomes (PRO) Measures: report of the ISPOR Task Force for Translation and Cultural Adaptation. Value Health. 2005, 8: 94-104.
Guillemin F: Cross-cultural adaptation and validation of health status measures. Scand J Rheumatol. 1995, 24: 61-3.
Landi F, Sgadari A, Zuccala G, Pahor M, Carbonin P, Bernabei R: A brief training program on resident assessment instrument improves motivation of nursing home staff. J Nutr Health Aging. 1999, 3: 24-8.
Landi F, Lattanzio F, Gambassi G, Zuccalà G, Sgadari A, Panfilo M, Ruffilli MP, Bernabei R: A model for integrated home care of frail older patients: the Silver Network project. SILVERNET-HC Study Group. Aging (Milano). 1999, 11: 262-72.
Morris JN, Fries BE, Morris SA: Scaling ADL's within the MDS. J Gerontol. 1999, 4: M546-53.
Morris JN, Fries BE, Mehr DR, Hawes C, Phillips C, Mor V, Lipsitz LA: The MDS Cognitive Performance Scale. Journal of Gerontology. 1994, 49: 174-182.
Landi F, Tua E, Onder G, Carrara B, Sgadari A, Rinaldi C, Gambassi G, Lattanzio F, Bernabei R, SILVERNET-HC Study Group of Bergamo: Minimum data set for home care: a valid instrument to assess frail older people living in the community. Med Care. 2000, 38: 1184-90.
Burrows AB, Morris JN, Simon SE, Hirdes JP, Phillips C: Development of a minimum data set-based depression rating scale for use in nursing homes. Age Ageing. 2000, 29: 165-72.
Fleiss JL: Statistical methods for rates and proportions. 1981, New York: John Wiley & Sons
Landis JR, Koch GG: The measurement of observer agreement for categorical data. Biometrics. 1977, 33: 159-174.
Lee JS, Chau PP, Hui E, Chan F, Woo J: Survival prediction in nursing home residents using the Minimum Data Set subscales: ADL Self-Performance Hierarchy, Cognitive Performance and the Changes in Health, End-stage disease and Symptoms and Signs scales. Eur J Public Health. 2009, 19: 308-12.
Hjaltadóttir I, Hallberg IR, Ekwall AK, Nyberg P: Predicting mortality of residents at admission to nursing home: a longitudinal cohort study. BMC Health Serv Res. 2011, 11: 86.
Gray LC, Berg K, Fries BE, Henrard JC, Hirdes JP, Steel K, Morris JN: Sharing clinical information across care settings: the birth of an integrated assessment system. BMC Health Serv Res. 2009, 9: 71.
Lattanzio F, Mussi C, Scafato E, Ruggiero C, Dell'Aquila G, Pedone C, Mammarella F, Galluzzo L, Salvioli G, Senin U, Carbonin PU, Bernabei R, Cherubini A, U.L.I.S.S.E. Study Group: Health care for older people in Italy: The U.L.I.S.S.E. Project (Un link informatico sui servizi sanitari esistenti per l'anziano - a computerized network on health care services for older people). J Nutr Health Aging. 2010, 14: 238-42.
Hirdes JP, Ljunggren G, Morris JN, Frijters DH, Finne Soveri H, Gray L, Björkgren M, Gilgen R: Reliability of the interRAI suite of assessment instruments: a 12-country study of an integrated health information system. BMC Health Serv Res. 2008, 8: 277.
Morris JN, Fries BE, Steel K, Ikegami N, Bernabei R, Carpenter GI, Gilgen R, Hirdes JP, Topinková E: Comprehensive clinical assessment in community setting: applicability of the MDS-HC. J Am Geriatr Soc. 1997, 45: 1017-24.
Wellens NI, Van Lancker A, Flamaing J, Gray L, Moons P, Verbeke G, Boonen S, Milisen K: Interrater reliability of the interRAI Acute Care (interRAI AC). Arch Gerontol Geriatr. 2011.
Steel K, Ljunggren G, Topinková E, Morris JN, Vitale C, Parzuchowski J, Nonemaker S, Frijters DH, Rabinowitz T, Murphy KM, Ribbe MW, Fries BE: The RAI-PC: an assessment instrument for palliative care in all settings. Am J Hosp Palliat Care. 2003, 20: 211-9.
Dahl TH: International classification of functioning, disability and health: an introduction and discussion of its potential impact on rehabilitation services and research. J Rehabil Med. 2002, 34: 201-4.
Berg K, Finne-Soveri H, Gray L, Henrard JC, Hirdes J, Ikegami N, Ljunggren G, Morris JN, Paquay L, Resnik L, Teare G: Relationship between interRAI HC and the ICF: opportunity for operationalizing the ICF. BMC Health Serv Res. 2009, 9: 47.
Bernabei R: International gerontology. Hazzard's Principles of Geriatric Medicine and Gerontology. 2008, New York: McGraw-Hill, 6
Bernabei R, Gambassi G, Lapane K, Sgadari A, Landi F, Gatsonis C, Lipsitz L, Mor V: Characteristics of the SAGE database: a new resource for research on outcomes in long-term care. SAGE (Systematic Assessment of Geriatric drug use via Epidemiology) Study Group. J Gerontol Med Sci. 1999, 54A: M25-M33.
Gambassi G, Landi F, Peng L, Brostrup-Jensen C, Calore K, Hiris J, Lipsitz L, Mor V, Bernabei R: Validity of diagnostic and drug data in standardized nursing home resident assessments: potential for geriatric pharmacoepidemiology. SAGE Study Group. Systematic Assessment of Geriatric drug use via Epidemiology. Med Care. 1998, 36: 167-179.
Williams JG, Cheung WY, Cohen DR, Hutchings HA, Longo MF, Russell IT: Can randomised trials rely on existing electronic data? A feasibility study to explore the value of routine data in health technology assessment. Health Technol Assess. 2003, 7: iii-v-x, 1-117
Vickers AJ, Scardino PT: The clinically-integrated randomized trial: proposed novel method for conducting large trials at low cost. Trials. 2009, 10: 14.
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6963/12/5/prepub
Ackowledgement and funding
The SHELTER study was funded by the EU 7th Framework Programme
IC, HFS, JG, DF, JCH, ET and RB are members of InterRAI.
GO, RB: conception and design, analysis and interpretation of data, preparation of the manuscript; IC, HFS, JG, DF, JCH, TN, ET: acquisition and interpretation of data, revision of the manuscript; MT, RL, FL: concept and design, acquisition and interpretation of data, revision of the manuscript. The final version of the manuscript was revised and approved by all authors.