Development and preliminary validation of a questionnaire to measure satisfaction with home care in Greece: an exploratory factor analysis of polychoric correlations

The elderly in the European Union constitute a significant segment of the society, which is expected to rise in the future [1]. In 2005 there were 79 million individuals over the age of 65 in Western and Central Europe. This figure will increase to 107 million in 2025 (+ 35%) and to 133 million in 2050 (+ 68%). The largest growth in this age group will be observed for those over 80 years (+ 180%). By 2030, in contrast, all other adult age groups will be declining in size. There is therefore a gradual shift from a society dominated by young individuals to a society in which the elderly will become the majority. The need for the provision of social services is exacerbated by the observed social exclusion, financial difficulties and health problems related to this particular age group. The World Health Organization has thus placed "good aging" fifth among its targets for the 21^st century.

This has led to the adoption of practices which aimed at the promotion of the well-being of the elderly on a social as well as on a physical level. Home care programs have been evolving in European countries for many decades based on the aforementioned and comprise the most important attempt to support the vulnerable social group of the elderly [2]. As one of the few social initiatives of local government in Greece ten years ago, legislation was enacted in the majority of municipalities for the elderly and people with disabilities. The program aimed at improving the quality of life of the enrollees. Its effectiveness nevertheless has never been formally assessed. This is not surprising since research on patient satisfaction by means of validated questionnaires is in fact very scarce in Greece [3, 4].

The program of providing home care services to the elderly and disabled begun in 1996 and quickly spread to most municipalities in Greece [5]. Seventy five percent of the necessary funds have been provided by the European Union and the rest through national financing [6]. By the beginning of 2005 there were 1084 home care units established serving 40,060 enrollees [7]. The general purpose of the program was to confront the social exclusion and institutionalization as well as to improve the quality of life of the elderly and the disabled. Priority has been given to individuals with an inability for self-care, who lived alone and had limited financial resources to improve their living standards and become independent.

In order to achieve these goals, social work, nursing care and life assistance services have been used [8]. The social worker is responsible, at an initial stage, for the needs assessment of the applicants, the scheduling of home visits and the determination of the range of provided services. Moreover, this employee is assigned with the task of co-operating with local parties, welfare services, voluntary organizations and the linking of the home care program with the participants and the local community. His/her role also includes provision of counseling and emotional support to the enrollees and their caregivers as well as admission to various institutions (e.g. hospitals, welfare, mental health units) when necessary. Nursing care is prevention-oriented. Information is provided regarding the illnesses and diseases enrollees are facing or are likely to face in the future. Nursing also addresses the enrollees' nutrition. Blood tests (including blood glucose tests), examination of vital signs and medication compliance are targeted towards the preservation of health, the self-care of enrollees and the establishment of a safe environment. Information is provided and the family and other caregivers are trained, so that they can handle problems more effectively. The aide workers' assistance includes help with personal hygiene, light housework, grocery shopping or other small purchases, payment of bills, meal preparation and other life-assisting activities. The interaction of all team members is often essential, especially for activities such as companionship, social mobilization of enrollees, emotional and psychological support and co-operation with caregivers.

In this study we developed and tested a questionnaire for measuring home care satisfaction in Greece. Although other questionnaires were available on an international level, there are significant differences in the various national settings that made the development of a customized instrument necessary. First, in several countries the specific home care services enjoyed by the participants depend on private payments or a person's contribution to a health insurance fund, whereas in Greece all services are provided free of charge by the municipalities. They are therefore determined solely by assessing relative needs. Second, in Greece the team which offers the home care services comprises mainly of a social worker, a nurse and two aide workers, whereas other specialties (medical doctors, psychologists, physiotherapists) encountered in other countries might or might not be included, depending on the specific municipality's policy. In addition, other services, such as meal preparation, might be available to all upon request abroad, yet only in certain programs and individuals in Greece. In general, there are important differences in the structure and mode of provision of home care services between countries, related to the methods of financing, as well as the cultural and social factors.

The aim of this study was to measure home care satisfaction by developing an appropriate questionnaire that covers the Greek home care provision. The tool devised was psychometrically tested and subsequently the level of satisfaction with home care services was measured. A novel concept in this work is the use of polychoric correlations in the factorial analysis and psychometric testing, which acknowledges the ordinal nature of the data, typically ignored in relevant prior research.

A questionnaire in Greek was developed to measure the satisfaction of enrollees with home care services. Items were phrased using a typical 5-point Likert scale with 1 indicating "strongly disagree" and 5 corresponding to "strongly agree". Positively and negatively worded items were interchanged to avoid positive biases [9]. Negative worded questions were therefore subsequently re-coded so that higher values would indicate higher levels of enrollee satisfaction, and vice versa.

Items were carefully selected after an extensive literature search and cognitive interviews with 15 program enrollees (of various ages, levels of education and incomes) and 15 employees (an equal number of social workers, nurses and aide workers) that were not included in the main study. Items were phrased so as to avoid vagueness and biases, double negatives, double-barreled questions and protests.

Satisfaction items reflect all the important aspects of the home care program, namely effectiveness, safety, respect, attention, participation in planning, correctness of planning, access, socialization and technical abilities of staff. The literature sought included theoretical papers, questionnaire validations and qualitative analyses [10–29]. Two questions included referred to general satisfaction from the provided home care services. Regarding socioeconomic and demographic variables, we recorded gender, age, educational level, income, family status, enrolment duration, visit frequency and informal caregivers [10]. Two more items were incorporated in order to exclude individuals that gave inconsistent responses. These had the same meaning as other questions but were differently phrased.

An important subsequent step in instrument development was pretesting, which took the form of cognitive interviews with elderly and disabled participating in the programs [30]. There were 10 such subjects from each group. These identified problems with the content and understanding of the questions, as well as additional causes of (dis)satisfaction, not captured by the instrument, thus enhancing reliability and content validity. Towards the same end, individuals pinpointed questions identified by the literature search that had to be removed from the instrument as being unimportant. The same persons were approached after a revision of the questionnaire and no additional issues were identified. Moreover, expert panel judgment was also exploited to ensure content validity and item clarity.

We initially screened the data by assessing (Spearman's) correlation coefficients between the items capturing satisfaction from specific characteristics of home care and the two items measuring overall satisfaction. Theory suggests that there might be a relationship apparent between overall satisfaction and satisfaction from individual services [9, 31]. Specifically, overall satisfaction emerges from customer's evaluations of service encounters, which in turn are evaluated on the basis of service attributes (e.g. response time of the nurse). The latter are evaluated on the basis of positive and negative events that occur during the encounter with the provider. In this sense, the overall evaluation of the provider might be a weighted average of all evaluations of service attributes, the weights being the importance assigned by the customer to each attribute.

In our context, however, one needs additional justification for excluding items from the survey, than simply observing insignificant correlations, for two reasons. First, we employed in this research two questions that have been used most frequently in practice for measuring overall satisfaction. It is nevertheless theoretically possible that they do not fully cover all the aspects of satisfaction from the service encounter, as perceived by customers. Second, a single item might have to move a great deal in order to have a significant effect on overall quality and there might be additional effects of the particular item when combined with other related questions. For these reasons, statistically insignificant correlations observed in this study only served as triggers for further investigation of specific items.

The questionnaire was administered to enrollees of home care programs, who were elderly and disabled in the age group of 65 to 90 years. An inclusion criterion to the study entailed that the subjects should have been participating in the program for at least two months, following Westra et al. [15]. The sample was approached from March 7 to April 3, 2006 with face-to-face interviews by trained interviewers. The study was approved by the Review Board of the Hellenic Open University. Subjects were told that their anonymity would be assured in the study. We excluded those who faced health disorders/problems that affected their mental state (e.g. Alzheimer's disease, severe stroke). Subsequently, five home care programs in Thessaloniki were randomly selected from a total of 36 programs that existed in municipalities of the city having more than 10,000 residents: Municipality of Stavroupoli (two home care programs), Municipality of Thermaikos, Municipality of Menemeni and Pefka Village. Although a consensus is lacking, samples with at least 200 persons and an observation-to-item ratio higher than 5:1 are generally acceptable in this type of research [32]. The interviewers visited the houses of all 319 enrollees in these programs that matched the inclusion criteria at the time of the study.

Summated scales were formed by exploratory factor analysis [33]. The Kaiser-Meyer-Olkin measure of sampling adequacy and Bartlett's test of sphericity were used to check whether the data were suitable for factor analysis. Values of KMO greater than 0.70 have been characterized as "middling" (and greater than 0.90 as "marvelous"). Bartlett's test of sphericity tests the null hypothesis that the correlation matrix is an identity matrix (i.e. that there is no relationship between the items) [34].

Although in many instances they yield similar results, factor analysis was preferred to principal components analysis, since the purpose of the study was to develop a theoretical construct, rather than merely perform data reduction [35]. The specific factor analyses employed unweighted least squares and varimax rotation. The unweighted least squares method was chosen because it is robust, can be used with small samples and also when the correlation matrix is non-positive definite. Factors whose eigenvalues exceeded unity were retained [36]. Following other researchers, we assigned an item to a factor - summated scale if its factor loading was greater than 0.50 and its differences with other factor loadings of the same item were greater than 0.20 [37, 38].

The main factor analysis carried out in this study acknowledges the fact that the 5-point Likert scale is in fact ordinal [39]. It was hence based on polychoric correlations. The polychoric correlation coefficient assumes that pairs of ordinal scores are in fact generated by latent bivariate normally distributed random variables. Measuring the association between ordinal variables entails the estimation of the product moment correlation between the corresponding normally distributed variables. Interestingly, a recent study that utilized a generalized polychoric coefficient found that it does not outperform the typical polychoric measure in cases where the sample size is smaller than 400 observations and the number of ordinal categories in the response scale is greater than three [40]. The polychoric correlation matrix was estimated using the two-stage procedure described in Jöreskog [41] and implemented in PRELIS v.2.8 [42]. Note that since the matrix was estimated in a pairwise fashion it was possible to be non-positive definite. An exploratory factor analysis was then performed entering the estimated polychoric correlation matrix into SPSS v.15.

Since prior research has mainly assumed that the Likert scale can be treated as an interval or ratio scale, we also performed, for comparative purposes, typical factor analyses based on Pearson correlations. In addition to unweighted least squares, we also extracted factors with principal axis factoring to account for the skewness observed in the data [43, 44].

Internal consistency reliability measures the extent to which all items within a scale are indeed capturing the same construct. Cronbach's alpha coefficients greater than 0.80 indicate high levels of internal consistency, whereas values less than 0.70 suggest that the researcher should attempt deleting individual items from the scales to examine whether consistency improves [45]. Due to the ordinal nature of the data, an ordinal standardised item alpha coefficient was also derived by inserting polychoric correlations into the simplified formula of the alpha coefficient [46, 47]. Note that prior research has examined the potential effect of skewness to this form of reliability coefficients and has found no impact of square root and log transformations on Cronbach's alpha coefficients [48]. A further criterion used to assess internal consistency reliability was inter-item correlations. Prior research has argued that items within a scale with lower correlations than 0.30 might not be sufficiently related to be used as measures of the same phenomenon, whereas items with correlations higher than 0.70 might be redundant [49]. We calculated Spearman's coefficients, thus taking into account potential skewness.

The degree to which scales are sensitive to external factors in successive measurements was assessed by examining test-retest reliability. The second administration occurred two weeks after the initial measurement. In this study, intraclass correlations were preferred to Pearson product moments coefficients because the test and retest variables originated from the same group of individuals and the former coefficients could detect systematic errors [50, 51]. There is no consensus regarding the appropriate standards for the value of the coefficients [52]. Nevertheless, a value of 0.70 is considered satisfactory in practice [53]. In addition, to account for possible skewness in the data, we also employed Kendall's Tau b, which is a distribution-free correlation measure.

Multi-trait analysis was then performed to assess internal consistency reliability, construct validity and within-scale item additivity. It is a psychometric technique applied when a questionnaire is hypothesized to capture several distinct theoretical constructs - concepts, each measured by several individual items (questions). The criteria assessed are [54]:

If criteria 1 and 2 are not met, the groupings should be reconsidered and if criteria 3 and 4 are not met, weighting of items while forming summated scales should be pursued.

In the multi-trait analysis conducted we acknowledged the ordinal nature of the data as well as the potential skewness of individual item scores. We therefore compared Pearson, Spearman, polychoric and polyserial correlation results, following prior research [55]. In the first case, we used a t-statistic for testing the significance of the difference between two dependent correlations [56]. Spearman's Rho is suitable for ordinal data since it is based on ranks of data. Note also that wherever a variable (multi-item scale) had more than 15 categories, it was treated as continuous. Therefore, a polyserial correlation was used for its association with other ordinal variables.

Construct validity was further assessed by examining inter-scale correlations. These had to be statistically significant and greater than 0.40 [57]. Furthermore, we looked at the correlations of the global satisfaction measure with the other scales, as well as the correlations of the latter with specific participant characteristics. To account for data ordinality and skewness we computed polychoric, polyserial, Spearman and Kendall's Tau b coefficients.

Ceiling and floor effects for the created multi-item scales were examined by assessing the percentage of responses falling into the highest and lowest possible satisfaction scores (i.e. strongly agree and strongly disagree). These are measures of the sensitivity of the tool. Finally, we computed the means and medians of the created scales in order to measure satisfaction with various aspects of home care as well as overall enrollee satisfaction. In fact, we linearly converted the scores into a 0-100 point scale.

The factor analysis was repeated with Pearson correlations. The variance explained was similarly high (62.97%) and - although there were some differences in the actual correlations used- the factors obtained with this approach were identical to those obtained with polychoric correlations. Hence these results are not presented here. Furthermore, given the skewed nature of the data documented in Table 1, analysis was also carried out with principal axis factoring. Results (not shown here) indicated that the factors identified were identical to those derived by unweighted least squares.

Multitrait scaling yielded satisfactory results. Table 5 shows that the first criterion of item internal consistency was met for three scales, since the Pearson, Spearman and polychoric correlations took values above 0.40. It was not satisfied, however, for the service planning scale. The second criterion of divergent validity was fully met for all scales. Regardless of which correlation coefficient we employed, in all cases the correlation of an item with its own scale was higher than its correlation with other summated scales. Differences were in fact significant (p < 0.05). With respect to the third criterion, for the two scales that contained more than two items, the ranges of the various correlation coefficient values were quite narrow. Similar values were also evident for the variances in the four summated scales, suggesting that the last criterion of the multitrait analysis was also satisfied.

Finally, Spearman's and Kendall's Tau correlations suggest some significant relationships of scales with respondents' characteristics that conform to our prior beliefs. As expected, net monthly income was negatively related to satisfaction with socio-economic change, staff skills & attitudes and service appropriateness. It was unrelated to service planning satisfaction. In addition, the duration of enrolment might be positively related to satisfaction with service appropriateness. These results however should be interpreted with some caution since polychoric and polyserial correlations lacked statistical significance (p > 0.05). These latter correlations also revealed an expected significant positive relation between visit frequency and satisfaction with service planning.

The percentages of observations falling into the highest satisfaction score category for the four summated scales were 7%, 15.4%, 35.3% and 3.5%, respectively. The respective floor effects were very limited and equal to 0.5%, 0.5%, 2% and 2.5%. Note that high ceiling effects are not uncommon in the literature [58], although some studies do find such effects to be more limited, that is lower than 10-12% [59].

Median (mean) satisfaction from socio-economic aspects of care, staff skills and attitudes, appropriateness of provided services and planning was 75.00 (72.17), 85.00 (81.27), 87.5 (81.03) and 60.00 (59.60), respectively. Overall satisfaction was 75.00 (74.63). This means that most enrollees were satisfied yet not very satisfied overall with provided services. In fact, 81.6% of all participants were to some extent satisfied with the home care programs, whilst 9% expressed some level of dissatisfaction. A point of concern, nevertheless, relates to the lower satisfaction with service planning documented.

A preliminary validation of a new psychometric tool for measuring satisfaction with home care was conducted with a sample of 201 individuals enrolled in five home care programs operating in the City of Thessaloniki. The questionnaire had 31 items and was expressed on a Likert scale. Although this scale has been typically considered in practice as interval it is in principle ordinal. We have therefore extended the usual research design to comparatively include ordinal methods, such as polychoric correlations.

Results from this exploratory analysis suggested that four multi-item factors are associated with home care satisfaction: socioeconomic change (4 items), staff skills and attitudes (5 items), appropriateness of provided services (2 items), and planning of services (2 items). An additional two-item scale was formed reflecting overall satisfaction with the programs. The first three multi-item scales had satisfactory internal consistency reliability, test-retest reliability and construct validity. An additional dimension suggested by factor analysis was "service planning" which has been found to have lower internal consistency reliability. Future research should further investigate whether the specific summated scale can be improved or legitimately ignored without sacrificing content validity.

There are differences as well as similarities between our scale dimensions and those found in other international studies. For instance, Reeder & Chen's confirmatory factor analysis lumped together three previously documented scales -namely professional/technical, interpersonal/trust and educational- into one unique dimension [13]. Using the same instrument, Laferriere found four dimensions of client satisfaction with home nursing care: technical quality of care, communication, personal relationships between client and provider, and service delivery [11]. Bear et al. instead reported two factors: service delivery and service sufficiency [20]. Netten et al. and Jones et al. mentioned carer quality - opinions towards carer, service quality and outcomes [27, 60]. Finally, Geron et al. classified items into the categories homemaker/health aide, care management service, home-delivered meal service and grocery service [22]. The reasons for the observed differences between the number and content of scales in the home care satisfaction literature have not been studied. In our research some dissimilarity with other findings was expected due to the differences in the Greek home care service provision outlined previously.

There is some evidence to suggest that satisfaction ratings with home care are influenced by monthly income and duration of enrolment in the program. Individuals with fewer own financial resources were in most need of home care services and hence reported higher satisfaction, and vice versa. Further, the more years an enrollee had been participating in the home care program, the more he trusted staff to carry out tasks for him and presumably did not allow aides to force him do things he disliked. Yet one should be cautious in uncritically adopting these associations and explanations since some correlation measures lacked statistical significance.

It is important to note that the results of this study were not influenced by the approach employed. That is, whether one assumed that the 5-point Likert scale was an interval scale or an ordinal one had in essence no significant effect in the study findings. Similar findings from ordinal and Pearsonian exploratory factor analyses have also been documented for the Big Five Questionnaire [39].

Finally, overall enrollees were satisfied with the home care programs. Higher levels of satisfaction were associated with the skills and attitudes of the staff and the suitability of services. Whereas lower levels were associated with the social and economic aspects of provided help and service planning. There are no existing studies in the Greek setting with which to compare the current findings. Although home care services in other countries differ, have been evaluated with other tools and refer to different populations, we simply note that findings such as ours are not uncommon. For instance, the mean satisfaction in this study was 74.63, whereas in a US study a mean of 74.1 was reported [22].

This study has certain limitations related both to the measurement of satisfaction and the validation of the questionnaire. Since the provision of home care programs in Greece is decentralized and information on the characteristics of the population of enrollees is not gathered by the central government, we were unable to investigate whether our sample was representative. The sample was in fact geographically restricted to the city of Thessaloniki. Moreover, there were no other studies in the Greek setting with which we could compare our findings. The summated scales "service planning" and "overall satisfaction" had low reliability values and "service appropriateness" a marginally acceptable Cronbach's alpha coefficient. The fact that these were all two-item scales might partly account for that. Nevertheless, we note that scales comprising of only two questions have often been found highly reliable in the Greek setting [4]. Finally, the measurement of satisfaction has been criticized for its weak conceptual foundation and potential biases due to acquiescent response set [61, 62].

Summing up, exploratory ordinal and traditional factor analyses alongside reliability and validity tests revealed three reliable and valid multi-item scales: socioeconomic change, staff skills and attitudes and appropriateness of services. A fourth dimension -service planning- had lower internal consistency reliability. Overall, enrollees were satisfied with the home care program. Future research should further validate the questionnaire with the use of confirmatory factor analysis and larger samples from other home care programs. Moreover, the role of service planning in the determination of home care satisfaction should be further assessed.