This article has Open Peer Review reports available.
Estimating recruitment rates for routine use of patient reported outcome measures and the impact on provider comparisons
© Hutchings et al.; licensee BioMed Central Ltd. 2014
Received: 12 November 2012
Accepted: 5 February 2014
Published: 11 February 2014
The routine use of patient reported outcome measures (PROMs) aims to compare providers as regards the clinical need of their patients and their outcome. Simple methods of estimating recruitment rates based on aggregated data may be inaccurate. Our objectives were to: use patient-level linked data to evaluate these estimates; produce revised estimates of national and providers’ recruitment rates; and explore whether or not recruitment bias exists.
Case study based on patients who were eligible to participate in the English National PROMs Programme for elective surgery (hip and knee replacement, groin hernia repair, varicose vein surgery) using data from pre-operative questionnaires and Hospital Episode Statistics. Data were linked to determine: the eligibility for including operations; eligibility of date of surgery; duplicate questionnaires; cancelled operations; correct assignment to provider. Influence of patient characteristics on recruitment rates were investigated.
National recruitment rates based on aggregated data over-estimated the true rate because of the inclusion of ineligible operations (from 1.9% - 7.0% depending on operation) and operations being cancelled (1.9% - 3.6%). Estimates of national recruitment rates using inclusion criteria based on patient-level linked data were lower than those based on simple methods (eg hip replacement was 73% rather than 78%).
Estimates of provider’s recruitment rates based on aggregated data were also adversely affected by attributing patients to the wrong provider (2.4% - 4.9%). Use of linked data eliminated all estimates of over 100% recruitment, though providers still showed a wide range of rates.
While the principal determinant of recruitment rates was the provider, some patients’ socio-demographic characteristics had an influence on non-recruitment: non-white (Adjusted Odds Ratio 1.25-1.67, depending on operation); most deprived socio-economic group (OR 1.11-1.23); aged over 75 years (OR 1.28-1.79). However, there was no statistically significant association between providers’ recruitment rates and patients’ pre-operative clinical need.
Accurate recruitment rates require the use of linked data to establish consistent inclusion criteria for numerators and denominators. Non-recruitment will bias comparisons of providers’ pre-operative case-mix and may bias comparisons of outcomes if unmeasured confounders are not evenly distributed between providers. It is important, therefore, to strive for high recruitment rates.
There is increasing interest in using patient reported outcome measures (PROMs) to assess the quality of providers of surgery, most notably in Sweden  and, more recently, in England . The National PROMs Programme that started in England in April 2009 covered four common operations and aimed to allow comparisons of providers in two principal ways: the appropriateness of patients undergoing surgery according to their preoperative health status and quality of life; and the outcome of care as judged by the change in health status and quality of life .
One particular concern is the extent to which incomplete recruitment of patients might bias comparisons of providers. This raises two methodological questions: are the estimates of recruitment rates accurate; and to what extent do those recruited differ from those not recruited?
As regards the first question, simple recruitment rates for each provider are routinely reported by the Health and Social Care Information Centre (HSCIC)  based on the number of completed pre-operative PROM questionnaires and the number of procedures undertaken according to a routine administrative data source, the Hospital Episode Statistics (HES). The shortcomings of this approach are evident in the number of providers who are reported as having more than 100% recruitment.
Potential causes of inaccuracies in the estimates of the numerator include: patients being included who underwent an ineligible procedure; despite having completed a pre-operative questionnaire, the procedure did not take place; the procedure was delayed such that it took place outside the time period being assessed; and patients being inadvertently asked to complete more than one pre-operative questionnaire, creating duplicate entries. Errors in the denominator can arise because patients are attributed to the wrong provider, a particular risk when operations are sub-contracted to another provider. To investigate the extent to which these problems occur, data on individual patients completing a pre-operative questionnaire need to be linked to their data in the routine administrative data. In this way it is possible to ensure the individual patients included in the numerator are the same individuals as in the denominator.
Even having corrected for such factors, recruitment rates for providers are likely to vary. And if recruitment is associated with patient characteristics that are, in turn, associated with the probability of the level of benefit patients will gain from the intervention, then comparisons of providers’ outcomes may be confounded. Given that the case-mix of providers is known to differ, the impact that any specific characteristic (eg age) has will also differ, reducing the external validity of comparisons of providers . It is therefore important to identify patient characteristics associated with non-recruitment.
Given these concerns about simple methods of estimating recruitment rates based on aggregated data, our three objectives were to: use patient-level linked data to evaluate these estimates; produce revised estimates of national and providers’ recruitment rates; and explore whether or not recruitment bias exists.
Analysis of secondary data from the routine use of PROMs in the English NHS for patients undergoing hip replacement, knee replacement, groin hernia repair, and varicose vein (VV) surgery were used to investigate the objectives. These procedures had been selected for the programme by the Department of Health for England as commonly occurring elective operations.
All elective patients funded by the NHS in England were eligible for inclusion if they underwent one of the following procedures: for hip surgery, unilateral, primary or revision surgery; total hip replacement, femoral head prosthesis only, acetabular cap prosthesis only. For knee surgery, total knee replacement or unicondylar surgery, unilateral, primary or revision. Groin hernia repairs included operations for inguinal, femoral and incisional hernias, unilateral, open or laparoscopic, primary or recurrent. For VV surgery, all modalities were eligible for inclusion: surgery, radiofrequency ablation, endovenous laser therapy, endoscopic perforator surgery, injection or foam sclerotherapy.
Two sources of data on patients undergoing surgery between October 2009 and September 2010 were used: pre-operative PROMs data for all patients who completed a questionnaire; and HES data on all patients whose episode included the relevant surgical procedures (based on 3-character OPCS codes). We received permission from the NHS Information Centre for use of the data. Analyses were restricted to NHS Trusts with at least 50 eligible HES episodes. Ethics committee approval was not necessary as this was secondary analysis of existing databases.
Linking patients who completed a pre-operative PROMs questionnaire to HES depended on the HSCIC having successfully attaching a HES_ID to each questionnaire. HES_IDs are patient identifiers based on the patient’s NHS Number and other identifiers. If the HSCIC is unable to identify a patient who completed a PROMs questionnaire in HES, they are not able to generate and attach a HESID.
Establishing accurate linked data
Step 1: Eliminating duplicate pre-operative questionnaires
HES data and pre-operative questionnaire data were used to check if a patient had inadvertently completed a questionnaire twice. The questionnaire with the best link to a hospital episode (by date and provider) was retained.
Step 2: Criteria for handling pre-operative questionnaires linked to more than one hospital episode
If links to more than one hospital episode were found, the one that most closely fitted in clinical terms and closest agreement of dates was used. Requirements for linkage were fairly unrestricted to maximise the success rate. Links were made even if:
the provider identified in HES was different to the one that supplied the questionnaire.
the date of surgery in HES was within 30 days before or 180 days after the date the questionnaire was completed.
the date of surgery recorded in the post-operative PROM questionnaire agreed with the date in HES.
Step 3: Identifying patients whose operation was cancelled
The PROMs database included a variable which indicates the stage the patient is at. This includes the option of ‘operation cancelled’.
Step 4: Excluding ineligible patients
If having linked the pre-operative questionnaires to a hospital episode, it was apparent the procedure was not eligible for inclusion, it was excluded. This arose most frequently when patients underwent resurfacing of their hip or knee joint (rather than joint replacement) and for those undergoing an incisional hernia repair in which the anatomical site was not stated. Patients were also excluded if the linked hospital episode fell outside the study period (October 2009-September 2010) even if the questionnare fell within it.
Step 5: Assigning eligibility to patients with unlinked pre-operative questionnaires
Having undertaken these steps, there remained some patients who had completed a questionnaire but could not be linked to HES (either because they had no HESID or no hospital episode could be found). As the eligibility of these questionnaires was uncertain, eligibility was assigned according to the proportion of eligible questionnaires found among patients who had been linked.
Investigating associations between patient characteristics and recruitment
Four socio-demographic characteristics of patients were available from HES data: age, sex, ethnicity, and deprivation measured using the Index of Multiple Deprivation (IMD) . The characteristics of patients for whom eligible hospital episodes were matched to a questionnaire were compared with those not matched to a questionnaire (taken as a proxy for non-recruitment). Multivariable logistic regression was used to examine the association between the characteristics and recruitment for each procedure. Providers were included as random effects (intercepts) to allow for differences in recruitment rates between providers.
While that approach is inevitably limited to those four patient characteristics, it is possible to investigate other factors indirectly by considering whether providers’ recruitment rates are associated with the health status of their patients. We explored the correlation between recruitment rates and mean disease-specific PROM scores (eg Oxford Hip Score) and mean EQ-5D scores for each of the operations. The impact of a 10% higher recruitment rate on the mean scores for each operation was also calculated.
Establishing accurate linked data
Number (%) of patients in HES who completed pre-operative questionnaires who were ineligible, of uncertain eligibility and were eligible for inclusion; and impact on estimated recruitment rate
Groin hernia repair
No patients in HES
No patients who completed pre-operative questionnaires
No (%) of patients who completed questionnaires of uncertain eligibility
6 231 (11.9)
7 366 (12.2)
7 161 (19.0)
1 530 ( 9.8)
No hospital episode
4 771 (9.1)
5 704 (9.4)
6 006 (15.9)
1 085 (7.0)
1 460 (2.8)
1 662 (2.8)
1 155 (3.1)
No (%) of patients who completed questionnaires who were ineligible
3 472 ( 6.7)
6 466 (10.7)
2 041 ( 5.4)
682 ( 4.4)
1 236 (2.4)
2 185 (3.6)
No (%) of patients who completed questionnaires who were eligible
42 480 (81.4)
46 879 (77.3)
28 515 (75.6)
13 376 (85.8)
National recruitment proportion:
Official (No of completed questionnaires/No patients in HES)
Best estimate (excluding ineligible)
Conservative estimate (excluding ineligible & uncertain eligible)
Patients who completed questionnaires but of uncertain eligibility
If questionnaires could not be linked to a HES episode, their eligibility for inclusion (ie they underwent the procedures being studied) was uncertain. The proportion that could not be linked varied from 9.8% to 19.0% by procedure (shown in row 3). For hip replacement this was true for 6 231 (11.9% of completed questionnaires). Our inability to link these patients arose mostly because no hospital episode could be found in HES and sometimes because they had no HES_ID.
Among those successfully linked, several reasons emerged as to why estimates of recruitment rates based on a simple comparison of numbers of questionnaires and hospital episodes rather than on linked data will be inaccurate.
Patients who completed questionnaires but were ineligible
There were three main reasons why patients were ineligible. First, having completed a questionnaire their operation was cancelled or postponed. This varied from 1.9% for VV surgery to 3.6% for knee replacement (Table 1). Second, some patients inadvertently completed two questionnaires though this occurred rarely, making up only 0.1-0.6% of all cases.
Third, some patients underwent operations that were not eligible for inclusion. This varied from 1.9% in VV surgery to 7.0% in knee surgery. For the two orthopaedic operations, this most frequently arose for patients undergoing resurfacing operations (75 hips, 3086 knees), bilateral operations (192 hips, 441 knees), non-elective operations (157 hips, 69 knees), non-NHS patients (247 hips, 159 knees) and the operation not being the initial reason they were admitted to hospital (204 hips, 206 knees).
Most (865) of the 1116 ineligible hernia operations were repairs of incisional hernias in which the site was not clearly coded as the groin. It is likely that these patients were in fact eligible but without the confirmatory site code, they couldn’t be included. Other reasons for ineligibility were 75 non-elective admissions, 56 non-NHS patients, and 113 who were initially admitted for another reason. Ineligibility of VV surgery patients was mostly due to the lack of a diagnostic code confirming the operation was carried out for VVs. These operations were probably eligible but, to be consistent with the denominator calculation, were excluded from the estimation of recruitment rate.
Ensuring operation within study period
The number of patients that had to be excluded for being outside the time period was balanced by a similar number who had been missed and had to be added (therefore not shown in Table 1). The numbers involved represented about 2% of all cases for hip and knee replacements and 1% for the other operations. The lower proportion for hernia repair and VV surgery was probably because more of those patients completed questionnaires on the day of surgery (rather than at pre-operative assessment clinics) thus minimising this particular risk.
Ensuring attribution to correct provider
While not affecting the national recruitment rate, a few eligible linked questionnaires (between 2.4% for groin hernia repair and 4.9% for knee replacements) were attributed to a different provider in HES than that indicated on their questionnaire. This was corrected when estimating providers’ rates.
Estimating recruitment rates
After removing pre-operative questionnaires that were ineligible, the national recruitment rate inevitably fell (indicated as Best estimate in Table 1). For hip replacements the rate (based simply on the number who completed a questionnaire divided by the number recorded in HES) fell from 78.4% to 73.1%; for knee replacement from 81.0 to 72.4%, for hernia repair from 54.7 to 51.8%; and for VV surgery from 44.7 to 42.7%.
If a very conservative approach were to be adopted and all patients of uncertain eligibility were assumed to be ineligible and were excluded, the recruitment rate would be even lower (eg 63.8% for hip replacement).
Identifying characteristics of patients not recruited
Proportion of non-recruited patients by socio-demographic characteristics and adjusted odds ratio (OR) for non-recruitment
Procedure (number not recruited/total number in HES)
Groin hernia repair
(24 071/66 598)
(27 982/74 827)
(40 418/68 886)
(21 508/34 909)
OR (95% CI)
OR (95% CI)
OR (95% CI)
OR (95% CI)
1.04 (1.00, 1.08)
1.03 (0.99, 1.06)
1.52 (1.43, 1.61)
0.99 (0.94, 1.04)
0.92 (0.87, 0.99)
0.85 (0.79, 0.91)
0.95 (0.91, 0.99)
1.14 (1.08, 1.22)
0.99 (0.93, 1.05)
0.88 (0.82, 0.94)
1.05 (1.01, 1.10)
1.26 (1.17, 1.36)
75 and over
1.28 (1.21, 1.36)
1.03 (0.97, 1.11)
1.37 (1.31, 1.44)
1.79 (1.61, 1.99)
1.49 (1.30, 1.70)
1.67 (1.55, 1.80)
1.44 (1.32, 1.56)
1.25 (1.12, 1.39)
1.01 (0.95, 1.08)
0.98 (0.92, 1.04)
1.01 (0.96, 1.07)
1.03 (0.95, 1.11)
Deprivation (IMD quintile)
1.03 (0.98, 1.08)
1.01 (0.96, 1.06)
1.04 (0.99, 1.09)
0.96 (0.88, 1.03)
1.08 (1.03, 1.14)
1.04 (0.99, 1.09)
1.05 (1.00, 1.11)
1.00 (0.92, 1.08)
1.17 (1.11, 1.24)
1.09 (1.04, 1.15)
1.08 (1.02, 1.14)
0.98 (0.90, 1.06)
1.20 (1.13, 1.28)
1.17 (1.10, 1.24)
1.23 (1.15, 1.30)
1.11 (1.02, 1.21)
Association between provider recruitment rate and pre-operative PROM scores
Mean score associated with a 10% higher recruitment rate
EQ5D index score
Disease specific score
Coefficient (95% CI)
Coefficient (95% CI)
-0.004 (-0.009, 0.000)
-0.12 (-0.26, 0.02)
-0.003 (-0.008, 0.002)
-0.09 (-0.23, 0.05)
Groin hernia repair
-0.002 (-0.005, 0.001)
Varicose vein surgery
0.002 (0.000, 0.005)
-0.15 (-0.28, -0.02)
Adopting a pragmatic approach, it is possible to establish a database of linked data (PROMs and HES) that encompasses 76-86% of patients. The process of doing so revealed several reasons why estimates of providers’ recruitment rates based on a simple comparison of aggregated counts of pre-operative questionnaires and hospital episodes rather than refined methods based on linked data will be inaccurate, the impact varying by operation: inclusion of ineligible operations (1.9% - 7.0%); attribution to wrong provider (2.4% - 4.9%); operation cancelled (1.9% - 3.6%).
Estimated national recruitment rates based on linked data and excluding ineligible questionnaires were lower: hip replacement was 73% (rather than 78%), knee replacement 72% (81%), hernia repair 52% (55%) and VV surgery 43% (45%). A more conservative approach in which those of uncertain eligibility are excluded inevitably reduces the estimates even further.
Use of these linked data eliminated providers with estimates of over 100% recruitment though the wide range of rates persisted (eg 6% - 96% for hip replacement for NHS Trusts). The new estimates showed that while over 80% recruitment is feasible for providers, this was achieved by only a quarter of them for hip and knee surgery and by only 2-4% for hernia repair and VV surgery. Many providers achieved less than 50% recruitment: a fifth of hip and knee surgery providers and about 60% of those providing hernia and VV surgery. These differences reflect the way providers have organised the recruitment of patients. Anecdotal evidence suggests that in the providers achieving the highest rates, one member of staff (usually a senior nurse) has taken personal responsibility for ensuring that patients are invited to participate.
While the principal determinant of recruitment rates was the provider, some patients’ socio-demographic characteristics had an influence. Non-white patients (Odds ratio 1.25-1.67) and those from the most deprived socio-economic group (OR 1.11-1.23) were less likely to be recruited in all four operations. This was also true for those aged over 75 years (OR 1.28-1.79) for three operations (not for knee replacement). Sex had little association with recruitment rate. There was no significant association between providers’ recruitment rates and the level of patients’ clinical need.
The method of establishing linked data required two assumptions. First, the best estimates of national recruitment rates assume that those of uncertain eligibility (because they could not be linked) were indeed eligible. This is justified as most linked patients were eligible (eg 92.4% of hip replacements).
Second, recruitment rates do not allow for patients who have more than one eligible hospital episode but only complete one pre-operative questionnaire. Providers have discretion in whether they offer another questionnaire to a patient having a second eligible operation. However, the number of such patients is so small as to have little impact on the recruitment rate.
These findings have two implications when using PROMs routinely to compare providers: how to calculate recruitment rates and the impact of non-recruitment on comparisons.
First, existing simple methods for estimating recruitment rates give inaccurate estimates. We have demonstrated how linked data can be used to establish consistent inclusion criteria and refine the estimation of recruitment rates for routine use of PROMs. The same principles and methods would also be applicable when estimating recruitment rates for other national clinical databases that use routine hospital administrative data to provide estimates of denominators.
Second, lower recruitment rates among non-whites, those over 75 years of age and those most deprived may reflect either that these patients are less likely to be asked to complete a questionnaire, less likely to agree to participate, or there is less success in linking their PROMs data to the routine administrative data. This will clearly bias comparisons of the pre-operative case-mix of providers. But will it bias comparisons of providers’ outcomes? If non-recruitment is random then it may not be a problem, even at low levels of recruitment. And even if non-recruitment is largely systematic it may pose no problem if the characteristics in question (confounders) are known, measured and adjusted for. The problem arises if there are other confounders (eg body mass index) that are not accounted for and the distribution of such factors differs between providers (which is possible). Therefore, given that it is impossible to be sure that all important confounders have been accounted for, it is safest to assume that comparisons of providers’ outcomes may be biased by non-recruitment. Clearly the higher the recruitment rate, the less likely the outcomes will be biased. It is therefore important to strive for high recruitment rates to enable pre-operative comparisons to be made.
We thank the Department of Health (Choice and PROMs) for funding.
- Rolfson O, Kärrholm J, Dahlberg LE, Garellick G: Patient-reported outcomes in the Swedish Hip Arthroplasty Register: results of a nationwide prospective observational study. J Bone Joint Surg (Br). 2011, 93: 867-875.View ArticleGoogle Scholar
- Black N: Patient reported outcome measures could help transform healthcare. BMJ. 2013, 346: 19-21. 10.1136/bmj.f19.View ArticleGoogle Scholar
- Guidance on the routine collection of Patient Reported Outcome Measures (PROMs) for the NHS in England. 2009, Department of Health, [http://webarchive.nationalarchives.gov.uk/20130107105354/http://www.dh.gov.uk/prod_consum_dh/groups/dh_digitalassets/@dh/@en/documents/digitalasset/dh_092625.pdf]
- Health & Social Care Information Centre: Finalised Patient Reported Outcome Measures (PROMs) in England -. 2009, (August 2011) http://www.hscic.gov.uk/proms, April to March 2010Google Scholar
- Britton A, McKee M, Black N, et al: Threats to applicability of randomised trials: exclusions and selective participation. J Health Serv Res Policy. 1999, 4: 112-121.PubMedGoogle Scholar
- Communities and Local Government: The English Indices of Deprivation 2007. 2007, London: Communities and Local Government PublicationsGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6963/14/66/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited.