Alberta CancerBridges development of a care plan evaluation measure

Background No standardized measures specifically assess cancer survivors’ and healthcare providers’ experience of Survivor Care Plans (scps). We sought to develop two care plan evaluation (cpe) measures, one for survivors (cpe-s) and one for healthcare providers (cpe-p), examine initial psychometric qualities in Alberta, and assess generalizability in Manitoba, Canada. Methods We developed the initial measures using convenience samples of breast (n = 35) and head and neck (n = 18) survivors who received scps at the end of active cancer-centre treatment. After assessing Alberta’s scp concordance with Institute of Medicine (iom) recommendations using a published coding scheme, we examined psychometric qualities for the cpe-s and cpe-p. We examined generalizability in Manitoba, Canada, with colorectal survivors discharged to primary care providers for follow-up (n = 75). Results We demonstrated acceptable internal consistency for the cpe-s and cpe-p subscales and total score after eliminating one item per subscale for cpe-s, two for cpe-p, resulting in revised scales with four 7-item and 6-item subscales, respectively. Subscale scores correlated highly indicating that for each measure the total score may be the most reliable and valid. We provide initial cpe-s discriminant, convergent, and predictive validity using the total score. Using the Manitoba sample, initial psychometrics similarly indicated good generalizability across differences in tumour groups, scp, and location. Conclusions We recommend the revised cpe-s and cpe-p for further use and development. Studies documenting the creation and standardization of scp evaluations are few, and we recommend further development of patient experience measures to improve both clinical practice and the specificity of research questions.


INTRODUCTION
For many cancer survivors, ending active treatment leads to uncertainty about their risk of recurrence and the organization of their care moving forward 1,2 .To address these concerns, oncology providers in the United States are mandated at the end of curative-intent treatment to clarify for patients what comes next through the provision of survivor care plans (scps) 3 .The Institute of Medicine (iom) report on cancer survivorship 2 strongly encouraged post-treatment individualized scps to enhance coordinated and quality care while addressing survivors' and primary care providers' (pcps') transition concerns.An scp contains individualized diagnostic and treatment details, follow-up and surveillance guidelines, symptoms of recurrence to monitor, information on health behaviour, coping, and resources 1,2,4 , and some include specific tools supporting health-behaviour-change planning or self-management 1 .Although no jurisdiction or accreditor mandates delivery, Canadian provincial oncology centres are beginning to offer and evaluate scps 1, 5,6 .While scp research has grown, best practices for patient and provider assessment and scp delivery post-treatment remain unclear or not broadly instituted 2,7,8 .A developing literature supports the use of scps [9][10][11][12][13][14] , and describes optimal content, formats 4,[15][16][17][18][19][20][21][22] , and key-stakeholders' views 1, 23,24 , but methods of evaluating scps and their impact on stakeholders are just emerging 4,[25][26][27][28][29][30] .

(H1)
Alberta's scps will have high concordance with iom recommendations.(H2.a) cpe-s and cpe-p will have acceptable internal consistency, skewness, and kurtosis.(H2.b)Subscales will correlate moderately.(H3) cpe-s ratings will be relatively stable (reliable) over six months.(H4.a) cpe-s will have moderate to high correlations with theoretically related, and low to small correlations with theoretically unrelated measures, demonstrating initial convergent and discriminant validity.(H4.b)Higher baseline cpe-s will correlate moderately with more positive ratings of pcp consultations one month later, and with improvement in distress over six months, providing initial predictive validity.Ratings of the same consultation on the cpe-s and cpe-p will correlate highly.(H5.a) cpe-s will generalize to another province, patient population, and scp delivery method, showing similar item internal consistency, subscale correlations, and (H5.b) discriminant and convergent validity.

Care Plan Evaluation (CPE) Development
Because no published scale evaluated survivors' and providers' scp experiences, the CancerBridges multi-disciplinary team created the cpe, comprising four face-valid, 8-item (Likert-type scales [1-do not agree to 5-completely agree]) subscales (Satisfaction; Usefulness; Emotional Reaction; Communication Value).We selected domains based on scp delivery goals for survivors and pcps outlined in the iom report 2 : that they feel satisfied that their questions are addressed; that the scp was useful in the transition following treatment; that receiving the scp would reduce negative emotional reactions, including distress and abandonment fears, and instead provide relief; and that scps would improve communication among health professionals and survivors helping them to navigate health issues.To reduce response bias, we reverse-coded 14 (43.7%)items.We evaluated two scp demonstration projects, funded by the Canadian Partnership Against Cancer (cpac), differing by scp delivery methods, tumour groups, and provinces.Goals for the cpe-s were to examine initial psychometric qualities and generalizability, and, for the cpe-p, to examine initial psychometric qualities in Alberta.

Participants
In

Breast Cancer and Head and Neck Cancer Survivors, Nurses, Primary Care Providers
In Alberta, nurses identified consecutive survivors in their cancer clinic/community/and navigator roles (see Collie et al. 1 for details).Survivors' eligibility criteria included over age 18, English literate, stages I-III, and within 2 weeks (+/-) of end of active treatment regardless of formal oncology discharge status.For bca (n = 36), the study was provincial, including two urban (Calgary and Edmonton) and two rural (Drumheller and Lloydminster) Alberta Health Services (ahs) cancer-centre sites, and one community-based organization (cbo).For hn survivors (n = 21) we included one urban location (Calgary).The CancerBridges multidisciplinary team developed scps for bca and hn survivors differing only in tumour-specific details.Trained nurses delivered scps during consultations with survivors who then completed the cpe-s 1 .Nurses who delivered scps (N = 8 of 8) and pcps who received them by fax after consultations (N = 22 of 57), completed the cpe-p after consultations.

Colorectal Cancer Survivors
Manitoba's nurse-oncologist teams delivered scps during standard clinical consultations to consecutive colorectal cancer (crc) survivors at the time of medical-responsibilityfor-care transfer to community family physicians or nursepractitioners (pcps).We approached survivors in Cancer-Care, Manitoba cancer clinics (Winnipeg) or Winnipeg Regional Health Authority, with those consenting receiving mailed surveys that included the cpe-s and other measures.Eligibility criteria included patients discharged to pcp care, over age 18, English literate, Stage I-III crc, minimum 12-months post-diagnosis, with ct scan demonstrating no evidence of recurrence.

Survivor Care Plan Development
Both provinces' care plans included cpac-and iomrecommended features 2 : diag nosis a nd t reat ment summaries (ts); follow-up and surveillance plans; coping, adjustment, and healthy-living recommendations/survivor priorities and goals; resources and activities for survivors; care-team contact information, and distress screening (scps).In Alberta, nurses hand-entered ts information prior to consultations.To empower survivors to set their own priorities and plans, survivors and nurses interactively completed remaining scp items during consultations.In Alberta, in addition to post-consultation surveys that included the cpe-s, we retained scp elements as data.

Breast and Head and Neck
After a full-day group nurse training to assess distress and deliver scps, nurses personalized survivors' TSs using medical records.Nurses scheduled face-to-face or phone consultations to deliver scps, answer questions, and clarify needs and goals 1 .One breast nurse (of 7) delivered individualized scps during group classes.Following delivery, we scanned scps into medical records, provided survivors copies, and faxed them to pcps.We encouraged survivors to schedule scp consultations with pcps within the month.Following consultations, nurses, patients, and participating pcps completed respective cpe measures.Survivors completed the minimum dataset for distress screening across Canada 37 during scp delivery (T1), as part of surveys following scp delivery (T2), and at 6-months (T3).

Colorectal
CancerCare Manitoba's electronic record populated scps with printed copies including three sections: a patientspecific TS, a crc-specific guide to survivorship, and a general survivorship information booklet for all cancers.After staff education, the nurse-physician team that had cared for patients during their chemotherapy prepared their scps with project-manager support.Physicians were medical oncologists or family physicians in oncology, specially trained family physicians overseeing chemotherapy in smaller hospitals in a shared-care arrangement with medical oncologists.Nurse-physician teams presented and discussed patient scps at their final, 20-minute transitional appointment, where patients completed and staff reviewed with them the minimum data set for distress screening across Canada 37 .Nurse-physician teams faxed completed scps to participants' pcps, with whom they were encouraged to meet within the month.Participants completed surveys once after pcp consultation, usually one to three months later.

Demographics and Cancer History
Alberta and Manitoba survivors completed separate demographic and medical questionnaires (Table I).

H1: SCP Concordance with iom Recommendations
In Alberta, the cpe-s and cpe-p development site, we ensured that our scps matched closely with iom recommendations 4 so these evaluations were based on fulsome scp experiences.To ensure comparability with standardized scp templates, we used a coding scheme that matches aspects of actual-use tss and scps with the guidelines, yielding a concordance score 4 .Coding-scheme authors developed these for bca tss and scps in the United States, so several items were irrelevant to Alberta's public healthcare system.Coding-scheme authors developed no coding guidelines for hn scps, so we copied bca coding-scheme guidelines applicable to both and added criteria through consultation with a hn nurse specialist.Two independent coders rated our bca and hn scps separately against these criteria with kappa (κ) statistics of agreement calculated.They then consensus-coded their work to resolve conflicts (bca: ts κ = 0.733, scp κ = 0.500; hn: ts κ = 0.808, scp κ = 0.455).

Edmonton Symptom Assessment System (esas)
In Alberta, to reduce multiple testing when examining discriminant and predictive validity, we used the mean (of 9 items) esas [37][38][39] .The esas is a valid and reliable part of the Canadian minimum dataset for distress screening.Patients rate current experience of nine symptoms (pain, tiredness, nausea, depression, anxiety, drowsiness, appetite, well-being, and shortness of breath) on 10-point Likert-type scales (0-symptom is absent to 10-most severe).

Contact Sheets
In Alberta, we used survivors' ratings of topic-discussion frequencies (5-point Likert-type scales: 1-not at all to 5-a lot) following nurse scp consultations to assess cpe-s convergent validity 40 because each topic theoretically relates to pieces of an ideal scp consultation.Interactive consultations led by survivors' priorities determined whether survivors discussed these nine topics: Information about cancer, Resources for cancer patients, Coping with cancer, Feelings, Work issues, Sexuality, Life-style changes, Taking control of life, and Follow-up tests and appointments.For predictive validity, we used survivor ratings of five positive and four negative affect words (Likert-type scale: 0-not applicable to 5-a lot) describing their pcp consultations, predicting that better scp nurse consultations (higher cpe-s) at baseline would prepare them to feel more positively about pcp consultations one month later.

Patient Continuity of Care Questionnaire (PCCQ)
In Manitoba, we examined convergent validity with the pccq 41,42 consisting of 41 Likert-type items identifying continuity-of-care issues post-discharge from a hospital setting.This is a reliable and valid short questionnaire developed in Canada that Manitoba investigators modified for use at the time of discharge to pcp follow-up 43 .Delivering more highly evaluated scps should improve survivors' continuity of care [44][45][46] .

Data Analysis
We conducted the following tests of our hypotheses:   ; and examined convergent validity using an intra-class correlation between survivors' cpe-s and nurses' cpe-p of the same consultation.(H4.b) examined predictive validity expecting higher baseline cpe-s ratings to predict 1) moderate correlations with higher positive and lower negative affect during pcp consultations a month later, and 2) moderate correlations with Total esas Distress improvement using slopes across three time-points.(H5.a) repeated item analysis for internal consistency and subscale correlations to assess generalizability to another province (Manitoba), tumour group (colorectal), and type of scp.(H5.b) expected small correlations between cpe-s and theoretically unrelated demographic and medical variables for discriminant and moderate to large correlations for convergent validity with continuity-of-care (pccq), a measure theoretically related to the cpe-s.
We present descriptive cpe-s statistics and box plots of tumour groups' total scores to explore distributions.For all analyses, due to the large number of tests, we used p ≤ 0.01.

Participants
Alberta's bca and hn survivors were similar in age, distance to cancer centre, race, marital status, education, employment status, majority English first language, but differed in expected ways by sex, time to scp delivery (differing treatment lengths), medical details, and household incomes

(H2.b) Subscale Inter-Relationships
Table III shows moderate to large Spearman correlations among subscales in each tumour group.To eliminate redundancy, analyses below use Total scores as providing the best cpe measure.Tumour-group correlation patterns were similar, showing a high degree of generalizability.

(H3) Stability
Total score intra-class correlation demonstrated stability between baseline and six months (0.651).We examined change over time and could demonstrate no significant Time (F = 3.8 (36), p = 0.06) or Group × Time (F = 0.83 (36), p = 0.78) changes.Out of a possible total score of 140, bca

(H4) Discriminant/Convergent Validity (H4.a) Discriminant
As hypothesized, the cpe-s Total score correlations with demographic and medical variables were small, demonstrating that it is not measuring age, time, or treatmentrelated issues.As predicted, correlations between cpe-s Total score and esas total were small and non-significant for bca (r = -0.05,p = 0.77); however, hn was moderate though non-significant (r = -0.41,p = 0.10).Care plan evaluationsurvivors is related to distress at a low level, but does not measure symptom distress (Table III).

(H4.a) Convergent
As expected, cpe-s correlations with relevant scp-consultation topic discussions were moderate to large (bca: median = 0.31; SD = 0.09; range 0.28-0.54;hn: median = 0.53; sd = 0.17; Range 0.29-0.76)(6 correlations significant at p ≤ 0.01).Breast cancer and hn differed on significant correlations: bca -Coping with cancer, Feelings, and Work issues; hn -Information about cancer, Feelings, and Follow-up tests and appointments (Table III).Intra-class correlations between nurses' cpe-p and survivors' cpe-s indicated similar scpconsultation ratings (0.75), providing evidence of convergent validity from multiple perspectives.Survivors did not complete a cpe-s rating for their scp consultation with pcps.

(H5.a) Generalizability
Manitoba's cpe-s for crc survivors (Table II) had similar internal consistency (Total Score α = 0.94; Satisfaction α = 0.78; Usefulness α = 0.75; Emotional reaction α = 0.80; Communication value α = 0.83).Subscale scores also correlated with similarly large effect sizes (0.77 to 0.93).Figure 1 shows that the 3 tumour groups reported similar cpe-s levels despite scp-delivery differences.(H5.b)Likewise as predicted, the cpe-s correlated at a low level with demographic and medical variables, but significantly with pccq scores, indicating a strong relationship with continuity of care (Table III).
We randomly selected one cpe-p per nurse for alpha calculations so nurses who filled these out for multiple patients were not represented more than once.Based on item-to-total correlations, missing items, and comments, we eliminated two items per subscale.The removal of these eight items (6 reverse-scored) improved some alphas, eliminated confusing items, and provided a shortened measure (Satisfaction, 0.73, to 0.70; Usefulness, 0.62, to 0.69; Emotional reaction, 0.57, to 0.74; Communication value, 0.51, to 0.71; Total score, 0.89, to 0.90).The shortened cpe-p demonstrated acceptable internal consistency, particularly the Total Score, and has four 6-item subscales (Table IV).Most items, subscales, and total scores had acceptable skewness (Items: -2.15 to -0.05, median = -0.92; 2 greater than ±2; Subscales and total scores: -0.09 to 0.44, median = -0.22)and kurtosis (Items: -1.34 to 7.64, median = 0.44; 7 greater than ±2; Subscales and Total scores: -0.15 to 0.30, median = 0.05).

(H2.b) Subscale Inter-Relationships
For nurses and pcps, most subscales were highly correlated, indicating that Total Score best represents the cpe-p (Table IV).Nurses reported somewhat lower total scores than pcps (Table IV).

DISCUSSION
We report first steps in scp-evaluation-measure creation for survivors (cpe-s) and healthcare providers (cpe-p).Each measures satisfaction, usefulness, emotional reaction, and communication value for the specific scp delivered.For initial psychometrics, we used scp-delivery data from bca and hn survivors in Alberta.To ensure the scps we evaluated matched closely with iom recommendations 47 , we applied a published coding system 4 , demonstrating close concordance with guidelines at or above levels reported by LIVESTRONG™ Survivorship Centers in the United States.We demonstrated acceptable internal consistency for cpe-s and cpe-p after eliminating one and two items per subscale, respectively.We recommend revised scales with four 7-item, and 6-item subscales respectively (Appendix 1, 2).Subscale constructs overlap with moderate to large effect sizes.Therefore, total scores best represent the two measures.We found survivors' cpe-s ratings reliable over six months, and that higher cpe-s at baseline predicted

A B
distress-symptom improvement over six months.Better survivor ratings at baseline also predicted a more positive pcp scp-discussion experience a month later.We found that cpe-s generalized to another province, tumour group, and an independently developed scp.We found evidence for discriminant/convergent validity in both provinces.For convergent validity, cpe-s correlated moderately with their ratings of scp-delivery topic discussions in Alberta, and better continuity-of-care in Manitoba.Discriminant correlations were low for cpe-s with demographic and medical variables, and with distress symptoms.
These indicators further scp-evaluation development from multiple perspectives.We encourage clinicians to use these scp-implementation tools for patients and healthcare providers.Measuring patient perceptions of scps and healthcare consultations as they leave oncologycentre treatment can facilitate clinical understanding of unmet needs and clarify empowerment strategies for selfmanaging survivorship tasks.Through cpe-s correlations, we found that tumour groups valued aspects of scp conversations differently: bca ratings correlated highly with coping, feelings, and work-issue discussions, whereas hn correlations highlighted cancer information, tests and appointments, and feelings.These correlations offer evidence that patients valued discussing feelings and coping as much as receiving cancer information.For crc patients, the cpe-s strongly correlated with management of continuity of care.The cpe-s thus captures an aspect of continuity of care as patients enter survivorship, an important patient-related outcome at discharge.
Interestingly, higher end-of-treatment cpe-s scores also correlated with future improvements in symptoms over six months.This indicates the potential for comprehensive care planning consultations to facilitate better prospective symptom management during survivorship transition, a time-period associated with many difficulties for survivors.
These measures also facilitate clearer examination of where and how patients benefit from transition consultations at discharge, and might improve the types of conversations we have and processes we engage as healthcare providers.Primary care providers could utilize their cpe-p for guidance to improve communication with survivors in their practices.If they or their survivors gave low ratings on communication value following an scp discussion, for instance, they might seek to provide better guidance, information, or referrals to the survivor that would improve communication with other providers.
Patient and provider scp-experience evaluations could also facilitate research, for instance, moderator or mediator analyses when examining medical, psychosocial, healthy lifestyle, and survival outcomes.If survivors are given one scp at discharge, then their cpe-s could correlate with or moderate their longitudinal outcomes.If survivors' scp conversations with pcps evolve over time, with multiple opportunities to update and enhance their scp and provide multiple cpe-s ratings over time, researchers could examine whether those changes mediated outcomes.Sharpening our understanding of the links between patients' evaluations of the scps we deliver and their outcomes might allow researchers to understand whether and how scps improve patient experience, allowing deeper insight if randomized trials have null outcomes 35,48,49 .

Limitations
This study used convenience data from two Canadian provinces during funded demonstration projects.These two cpe measures would benefit from further development and psychometric evaluations using larger survivor samples.Questions that ask providers about integrating scps into their practices may be welcome additions and improve the cpe-p 26,35 .Small sample sizes, predominantly Caucasian and urban participants, and many exploratory tests (despite p ≤ 0.01 criterion) may limit generalizability.However, studies documenting the creation and implementation of scp evaluations are few, and this work may be a valuable starting point to facilitate further development of patient-experience measures to improve clinical practice and the specificity of scp research questions.

CONCLUSION
Cancer survivors, nurses, and pcps valued our scps, and initial psychometrics of our evaluation measures are promising even though groups' post-treatment symptoms, prognoses, and trajectories widely diverged.We recommend THANK YOU

(
H1) reported percentage concordant reports when evaluating Alberta's match with iom scp templates using Stricker et al.'s 2012 coding rules 4 .(H2.a) counted missing answers, and analyzed skewness and kurtosis of items and subscales to evaluate cpes and cpe-p usability and psychometrics, and used Cronbach's alphas and item-to-total correlations to evaluate cpe-s and cpe-p internal consistency.e62 Current Oncology, Vol. 25, No. 1, February 2018 © 2018 Multimed Inc.

01 .
, and subscale and total score correlations for the Care Plan Evaluation-Survivors (cpe-S) Cronbach's alpha for each subscale and total score are bolded and in the diagonals.Spearman's rho correlations.e66 Current Oncology, Vol. 25, No. 1, February 2018 © 2018 Multimed Inc.

FIGURE 1
FIGURE 1 cpe-S and cpe-P Distributions across Survivor and Provider Samples.Graphed are box-and-whisker plots.A) Care Plan Evaluation-Survivor (cpe-S) distributions for three tumour groups, two in Alberta and one in Manitoba.B) Care Plan Evaluation-Provider (cpe-P) distributions for two providers, nurses who delivered care plans and primary care physicians (PCPs) who received them.Bottom line on whisker = the smallest observation; bottom line on box = lower quartile; middle line on box = median; top line on box = upper quartile; top line on whisker = largest observation; circles = mild outlier.
accordance with the Declaration of Helsinki, this study received ethics and data sharing approval through respective research ethics boards (The Conjoint Health Research Ethics Board at the University of Calgary; Research Ethics Board Faculty of Medicine, University of Manitoba, and Research Resource Impact Committee of CancerCare Manitoba).All participants signed informed consent.

TABLE I Continued
(H2.b) examined cpe-s and cpe-p subscale-score interrelationships, predicting moderate correlations, using Spearman's rho (as effect size estimates, 0.20 as small, 0.50 moderate, and 0.70 large) by tumour group for cpe-s (bca, hn), and by provider for cpe-p (Nurses, pcps).

TABLE III
Discriminant and convergent validity with cpe-S total scores for Alberta and Manitoba

TABLE IV
Means, internal consistency, and subscale and total score correlations for the Care Plan Evaluation-Healthcare providers (cpe-P) our revised cpe-s and cpe-p with four 7-item, and 6-item subscales, respectively, comprising each total score.We hope this initial work allows researchers and clinicians to improve the specificity of their research questions and individualized scp consultations.We also hope others will continue to improve these scp measures.For each of the statements below, circle the number that shows how much you agree, using the scale listed below.For each of the statements below, circle the number that shows how much you agree, using the scale listed below.It is clear from their plan who to go to for each cancer need in the future.I felt encouraged to suggest some of the options listed in the care plan for making changes to the survivor's lifestyle.Having this care plan, I have hope that I can help the survivor navigate this cancer experience.Having this medical information makes it easier for me to co-ordinate and to communicate with other members of this survivor's medical team.