Lessons learned from a cancer knowledge translation grants program: results of an evaluation

Special Article


Lessons learned from a cancer knowledge translation grants program: results of an evaluation


M.A. O’Brien, PhD*, T. Makuwaza, MA*, I.D. Graham, MA PhD§, L. Barbera, MD||,#**, C.C. Earle, MD MSc††, M.C. Brouwers, PhD‡‡, E. Grunfeld, MD DPhil*††



doi: http://dx.doi.org/10.3747/co.26.5531


ABSTRACT

Background

A novel way to build capacity in knowledge translation (kt) is through kt-focused grant competitions. Since 2009, the Knowledge Translation Research Network (KT-Net) has had a cancer-related kt grants program. We undertook an evaluation of the program to determine if KT-Net was achieving its aims of building capacity in cancer kt, advancing the science of kt, building partnerships, and leveraging funding.

Methods

An adapted framework guided the evaluation. Nine funded studies from 4 competitions were included. Semi-structured telephone interviews were held with researchers, stakeholders (including knowledge users), members of grant review panels, and experts in kt. Interview transcripts were audio-recorded, transcribed, and analyzed thematically. A review of proposal and report documents was also conducted.

Results

Funded researchers indicated that the grant competition was an essential funding program for cancer kt research. Competitions were perceived to build capacity in cancer kt among early-career researchers and to encourage innovative cancer kt research for which alternative funding sources are limited. The grants program resulted in incremental gains in advancing the science of kt. Suggestions to improve the program included stronger partnerships between the funder and the provincial cancer-system organization to optimize the application of research that is relevant to the organization’s strategic objectives.

Conclusions

The grants program met many of its aims by providing cancer researchers with an opportunity to gain capacity in cancer kt and by making incremental advances in kt science. Suggestions to improve the program included closer partnerships between the funder and the cancer-system organization.

KEYWORDS: Knowledge translation, capacity building, grant competitions, integrated knowledge translation

BACKGROUND

Investment in health research is often undermined by a lack of appropriate and effective delivery mechanisms for the knowledge generated14. When significant delays occur between the production of evidence-based information and its incorporation into clinical practice, patients are unlikely to realize the benefits of investment in health research. For example, a landmark study used cumulative meta-analysis and bibliometric analysis to demonstrate a 13-year gap between the accumulation of sufficiently strong evidence of the benefit of thrombolytic treatment for acute myocardial infarction and the recommendations for that approach as a standard treatment5. Examples relevant to cancer control are the insufficient uptake of fecal occult blood screening to prevent colorectal cancer6 and variability in the management of persistent pain in cancer patients7. Accordingly, there is a need for a better understanding of how research is translated into practice, including barriers and enablers, and of how effective specific strategies are for accelerating the uptake of evidence-based practices8.

In Canada, several funding programs that support inquiry within the growing field of knowledge translation (kt) are available to researchers. At the national level, those programs have included, but are not limited to, kt and commercialization funding from the Canadian Institutes of Health Research (cihr) and knowledge-to-action grants provided by the Canadian Cancer Society Research Institute in partnership with the cihr’s Institute of Cancer Research. Canadian researchers can also submit proposals to the U.S. National Cancer Institute’s dissemination and implementation grants program. In Ontario, provincial funding opportunities include the Adopting Research to Improve Care program of the Council of Academic Hospitals in Ontario, in partnership with Health Quality Ontario. One cancer-specific funding program is managed by the Knowledge Translation Research Network (KT-Net), which is part of the Health Services Research Network—a joint initiative of the Ontario Institute of Cancer Research (oicr) and Cancer Care Ontario, the provincial cancer-system organization.

The overarching aims of the KT-Net grants program are to increase capacity in cancer kt research, to advance the science of kt, to develop strategic partnerships between researchers and knowledge users at the cancer-system organization, and to leverage funding for subsequent applications to other granting organizations. We hypothesized that the cancer kt grants program would provide the opportunity for researchers and knowledge users to collaborate on kt-related grant applications that would address the strategic objectives of oicr or the cancer-system organization, or both.

The purpose of the present paper is to describe the results of an evaluation of the KT-Net grants program to determine the extent to which the program achieved its aims.

METHODS

Study Context

Between 2009 and 2016, KT-Net held 7 competitions. Researchers were invited to submit proposals focused on cancer kt that aligned directly with the priorities of the oicr or the provincial cancer-system organization, or both. The present evaluation includes 4 competitions held between 2009 and 2012. Studies from 3 competitions were excluded: 2 competitions were ongoing at the time of the evaluation, and relevant outcomes data were incomplete; the 7th competition was launched during the evaluation. Table I provides descriptive information about each funded study. The total funding for the 4 competitions was just over CA$1 million, with an additional CA$75,000 provided by the Canadian Partnership Against Cancer for the 1st competition. The average amount of available funding per competition was approximately CA$125,000 per study and the duration of each study was 1–2 years.

TABLE I Description of cancer knowledge translation studies funded by the Knowledge Translation Research Network (KT-Net), 2009–2012

 

Study Design

We used a descriptive qualitative design21 based on an evaluation framework adapted from a framework used by the cihr to evaluate its kt funding programs22. The adapted evaluation framework included 7 key questions, with indicators for each question (Table II).

TABLE II Evaluation frameworka


 

Sampling and Recruitment

We approached a broad range of stakeholders of the KT-Net grants competition: applicants to the grants competition between 2009 and 2012, knowledge users on study teams, members of the KT-Net Advisory Committee, grant panel reviewers, and stakeholders who were affiliated with the provincial cancer system. We asked researchers to nominate knowledge users on their study team. We sent study invitations to all potential participants by e-mail and a research assistant followed up. We used the modified Dillman approach for follow-up, whereby potential participants were sent an e-mail message every 2 weeks for 6 weeks in total23. After 6 weeks, potential participants were not contacted again.

Data Collection

Interviews

An experienced qualitative researcher (TM) using an interview guide conducted semi-structured telephone interviews. We developed the interview guide based on questions in the evaluation framework. The interviewer solicited participant feedback about their experiences with the funding program and their views about their respective research studies (if applicable). Interviews were audio-recorded, transcribed verbatim, and anonymized.

Review of Documents

Relevant documents, including calls for proposals, proposals, and reports of funded studies, were examined. One team member (TM) extracted data pertaining to the number of presentations, reports, and publications emerging from the funded studies; a second team member (MAO) checked the extracted data for accuracy.

Data Analysis

Using the principles of the constant comparative method24, 2 team members performed line-by-line open coding of the interview transcripts25,26. The presence of converging and discordant responses pertaining to the same study and across studies was then examined. Emerging themes were identified from the coded data. During coding and identification of themes, disagreements were resolved through discussion. To ensure transparency in the interpretation of the results, an audit trail was maintained to document all major analytic decisions27. The data management software NVivo (version 9: QSR International, Melbourne, Australia) was used to facilitate data management.

Research Ethics Review

The University of Toronto Research Ethics Board approved the study (protocol 30867).

RESULTS

Demographics (Interviews)

Of 37 individuals (21 researchers, 8 stakeholders, 6 study-team knowledge users, and 2 reviewers) invited to participate, 20 were interviewed. Another 2 participants (an unfunded researcher and a knowledge user) elected to provide written comments by e-mail. The evaluation included all 22 participants (59% of the invitees; 6 of 7 funded researchers; 2 of 14 researchers who applied, but who were not funded; 8 of 8 stakeholders; 4 of 6 knowledge users on research teams; and 2 of 2 reviewers). The knowledge users included cancer centre leaders and administrators, and directors and managers within the provincial cancer-system organization. The stakeholders included experts in cancer-related kt at the provincial and national levels. Participants were a median of 48 years of age (range: 31–65 years), with 3 participants not reporting their age. Of the participants, 16 were women, and 6 were men. The subsections that follow report the results by relevant key evaluation question. The grants program as a whole is being considered, unless otherwise specified.

Question 1: To What Extent Are the Grant Competitions Achieving Their Outcomes?

The expected outcomes of the grants program as a whole were building capacity in cancer kt, advancing kt science, establishing strategic partnerships, and leveraging funding. There is evidence that the grant competitions are achieving their aims with respect to these indicators:

  • ■ Number of competitions held and number of proposals funded

  • ■ Number and type of knowledge products

  • ■ Unique kt strategies developed or tested

  • ■ Number and type of highly qualified personnel supported

  • ■ Number and type of partnerships created

  • ■ Amount and source of leveraged funding

Data sources for the foregoing information were the document review and interviews.

Number of Competitions Held and Number of Proposals Funded

During the 4 competitions included in the evaluation, 28 applications were submitted, and 9 were funded. The number of applications increased to 13 in 2011 and 10 in 2012 from 4 in 2009. All funded proposals were aligned with the priorities of the oicr or the cancer-system organization. In 2012, a special call titled “Tightening Loose Connections—Integration of Cancer Research and Cancer Policy Through Cancer Care Ontario” was also issued. For that special call, 1 proposal was received and subsequently funded.

Number and Type of Knowledge Products

Aggregated across the 9 funded studies, resultant knowledge products included 61 presentations, 12 journal articles, and 10 reports. Involvement of highly qualified personnel across all studies included 54 researchers, 24 knowledge users, and 15 research personnel. With respect to building capacity in cancer kt research, funded studies supported 7 trainees, including 2 clinical research fellows, 1 postdoctoral fellow, and 4 masters students. In total, 100 individuals were involved in the 9 studies (approximately 9 per CA$100,000 of funding).

KT Strategies Developed or Tested

The document review highlighted the ways in which the grant competitions addressed the practice of kt. For example, Sussman et al.a used an existing telemedicine platform, the Ontario Telemedicine Network, in an effort to integrate primary care physicians into multidisciplinary case conferences at a regional cancer centre, with the intent of improving communication between specialists and primary care physicians. Lim et al.10 deployed a strategy of educating specialists involved in making a lung cancer diagnosis, thus improving the current process of molecular testing in lung cancer. Wei et al.16,17 developed an evidence-based clinical pathway for pancreatic cancer surgery by conducting needs assessment workshops and subsequently incorporating end-user recommendations into the instrument. Brouwers et al.20 conducted a review of systematic reviews of kt interventions relevant to cancer control. They found thirty-four reviews of 19 interventions and concluded that, although some interventions were promising, the overall approach to kt in cancer control was “patchy and unsystematic.” In a separate study, the same authors examined resources to guide the development, implementation, or evaluation of specific cancer screening uptake interventions (small media, client reminders, and provider audit and feedback)b. They found that there was a need to improve the operationalization of those tools in practice.

Building Strategic Partnerships

Interview data provided information about strategic partnerships that were achieved as a result of the funding program. One researcher described the creation of new collaborations between clinical units as a result of their research, another reported on interprovincial collaborations, and a third researcher described sustained partnerships with research partners that continued after project completion. On the other hand, stakeholders were largely unaware of associations occurring at a study level, but speculated about possible connections developing between researchers and the cancer-system organization.

Leveraging Funding

Three researchers from different studies were successful in leveraging CA$173,625 in additional funding from 3 different granting agencies. Another researcher was successful in receiving an additional CA$100,000 in funding from the Canadian Cancer Society Research Institute based on pilot data from their funded study.

Pertinent Themes

We identified 3 themes pertaining to this evaluation question that arose from the interviews:

  • ■ Essential opportunity to access specific cancer-relevant kt funding

  • ■ Opportunity to gain capacity in cancer kt research

  • ■ Incremental contributions to the science of kt

Essential Opportunity to Access Specific Cancer-Relevant

KT Funding

Researchers from funded studies and stakeholders both perceived that the grant competitions provided an essential opportunity to access cancer-relevant kt funding, with researchers viewing the competition as a platform for supporting methodical and rigorous kt research in Canada. Researchers reported that funding for kt research in general is limited and that obtaining funding is often difficult. Limited funding opportunities were partially attributed by several participants to the lesser valuing or lower prioritization of kt as a field of research, especially when compared with the field of clinical science:

And so I think [that KT-Net] gives an opportunity to do some of these projects which actually require a framework to do it in and ... funding to do it well in a rigorous methodological way.

— Researcher 4

Opportunity to Gain Capacity in Cancer KT Research

Funded researchers described themselves as having a range of kt research experience, with several having little or no experience and others having either some experience or considering themselves to be expert kt researchers. We were interested to discover whether kt capacity or skills had been attained as a result of being a grantee and engaging in kt research. Two researchers who perceived that they were new to the field of kt reported an increased understanding of this field of investigation, with one participant acknowledging how lessons learned during their grant informed the application for a subsequent successful kt grant:

And part of the way that that [new] grant was written was actually informed by what I learned on the KT-Net grant. One component of it really informed how I was able to write the [name of grant competition] grant, which was successful.

— Researcher 7

The other novice kt researcher perceived that their research study had actually provided the framework upon which to establish an entire research program at their institution. That participant was emphatic about how the grant competition had been important for their work. However, an experienced kt researcher reported not gaining any new skills from the competition, which was not surprising, given that this individual had already been an investigator on numerous kt studies. A mid-career kt researcher also reported no capacity building for their team, stating that it is not feasible to truly build capacity with the funds provided by KT-Net grants, because the actual funds per grant are relatively small. However, that researcher viewed capacity-building in terms of building a sustainable kt skillset within their research team rather as part of their own individual development and strengthening of skills as a researcher engaging in kt studies.

Aside from increased capacity in kt research by new investigators, the early and mid-career kt researchers reported leveraging opportunities and obtaining more funds for additional dissemination activities. They also reported leveraging their increased kt skills to successfully apply for additional funding aimed at new research.

Incremental Contributions to the Science of KT

Interviewed stakeholders acknowledged that research in general is frequently incremental, with small advances contributing to the overall body of knowledge in a given field. The grants program was also perceived to have contributed incremental advances to the science of kt. One researcher observed that research is sometimes about generating empirical data that helps to decide whether to take the next step in the research process. Stakeholders perceived that the results of funded studies were “interesting” and “incremental contributions” to the field of kt, generating primary data that could be leveraged for subsequent work, including increased funding for larger projects. The idea of incremental contributions from research was articulated thus by one stakeholder:

But oftentimes, it is often by looking back ten years, fifteen years, twenty years, that you sometimes see these new discoveries. With knowledge translation I suspect that the horizon is not as long. But, you know, I would like to think that research that we are doing today, it may take five years for people to look backwards and say, “Oh boy, look at that—an interesting idea—that is where it started.”

— Stakeholder 2

Stakeholders also commented that the effect of the grants program had been to provide a mechanism for the cancer research community to become aware of the priorities of the oicr and the cancer-system organization and therefore to encourage research applications that addressed those priorities. It also provided funding for research that was ready for incorporation into practice and for pilot work in a milieu in which kt research had limited funding.

Question 2(a): What Factors Facilitate or Inhibit the Achievement of Outcomes of the Grant Competitions?

The results for this question pertain to the grants program as a whole and were derived through the semi-structured interviews with participants.

Facilitators

Researchers did not describe specific facilitators of outcome achievement by the grants program. Instead they described facilitators of outcome achievement during their specific study. Those facilitators are discussed in the Question 2(b) subsection.

Barriers

Participants identified 2 key barriers to outcome achievement of the grants program. One barrier was the perception that kt is an emerging field and that researchers lack clarity about what constitutes kt research. Stakeholders and researchers both viewed kt as a growing field, with the term “kt” and the development of kt science being regarded as fairly new phenomena. An early-career kt researcher believed that kt language and kt frameworks might be unfamiliar to some researchers and therefore a barrier to applying for funding. A knowledge user repeated the idea of kt being an area difficult to define:

I find knowledge translation in general a little bit of a vague process. Like, I have participated in it, I review for cihr, and I find that measurables are hard to come by sometimes.

—Knowledge user 3

An expert in kt reported that a lack of clarity concerning kt is a wider problem in both Canada and the United States. This theme of uncertainty about what constitutes kt research was illustrated by one stakeholder who commented on the types of proposals submitted to the grant competitions. That stakeholder thought that the competitions did not always fund kt research, but rather what that individual regarded as an “education intervention,” “communication research,” or “efficacy research.” That stakeholder recommended that the KT-Net call for proposals be more explicit about the types of kt proposals that are desired. A reviewer noted that, within research proposals, some applicants better articulated the clinical priorities than the kt components, thus emphasizing the importance of clear kt requirements in the call for proposals. Inclusion of a letter of intent in the application process of the grant competitions was suggested as a mechanism whereby applicants could receive feedback that might promote kt research proposal submissions of higher quality.

A second barrier to outcome achievement of the grants program was the perception held by stakeholders (but not researchers) that the actual dollar value of the grants awarded to applicants was relatively small. However, administrators and funded researchers were both doing a lot with relatively little funding. Some stakeholders perceived a direct correlation between limited funding and what they believed to be only a modest impact of funded research studies on patient care and only small contributions to the field of kt research. Participants noted that it might be unrealistic to expect that an individual study should change policy:

I think it has made small, nice, incremental contributions, but again, fairly small and not because it is not important, but because we’re talking some quite small grant money. So very small, tiny interesting contributions, and I think the money—in terms of, like, the amount, the number of projects that have been funded—and given the amount ... I think the money has been really well used. You know, like, people really take what they can and really, really use it well.

— Stakeholder 1

One stakeholder noted that it is important to manage expectations about funded studies. They perceived that reasonable goals were acceptable, but that “game changers” or “policy changers” should not necessarily be the anticipated outcomes of funded studies. Many stakeholders advocated for more funding for the competition, as well an increase in individual grant allocations. They perceived that larger grants might sustain projects longer, allowing them to make a more measurable difference to a given problem. Researchers did not provide any comments regarding the issue of grant size in relation to the impact of the grants program.

Question 2(b): What Factors Facilitate or Inhibit the Achievement of Outcomes of Individual Studies?

Facilitators

According to several researchers, factors that facilitated the achievement of outcomes in their respective studies were willing research participants and maintenance of flexibility during local implementation of the study. The involvement of knowledge users was also identified as an important facilitator by researchers. Researchers reported a significant level of engagement with knowledge users throughout the lifecycle of the funded studies. Knowledge users had various points of contact with their respective research studies that included, but were not limited to, participating in proposal development, linking researchers with study participants, acting as key informants about clinical context, helping to interpret findings, being the recipient of findings, and in general, facilitating the achievement of study outcomes. A key facilitator to knowledge user participation was identified by researchers as the involvement of individuals in the context of an established partnership or a prior affiliation. Aside from that leveraging of prior collegial relationships, researchers also described how meaningful knowledge user involvement in their research studies was enabled by having knowledge users who were interested in the research topic, by valuing the opinions of their knowledge users, and by ensuring that participation by the knowledge users was not overly burdensome.

A key facilitator to effective dissemination and uptake of study results evident in two funded studies was the influential role the researcher held as an embedded scientist within the cancer-system organization. Those researchers mentioned that the role of embedded scientist was critical in bridging two worlds, namely research and policy, and was perceived to be significant in facilitating use of the study results.

Barriers

Participants described three barriers to uptake of findings generated by the funded kt studies:

  • ■ Knowledge users not being cognizant of the expectations associated with their role

  • ■ The intensive interpersonal networking required to generate interest and uptake of research findings

  • ■ Lack of researcher persistence in promoting uptake of study findings

Knowledge User Role

Although most researchers perceived that there had been sufficient engagement of the knowledge user group during the research study, 2 of the 3 knowledge users interviewed felt that their role had been more peripheral than described by the researcher. Neither the researchers nor the knowledge users held any expectations about their role in championing the use of new knowledge emerging from the study. When asked whether the knowledge users could have played a greater role in advocating for use of the research findings, one researcher felt that advocating was a rare thing for knowledge users to do:

I would say that is rare, not because the best intentions are lacking, but it comes down to the fact that these people wear multiple hats, they have multiple responsibilities, they are pressed for time, and they may not. And it is not even necessarily that they view it as their role, but it is simply that is well beyond what is possible.

— Researcher 1

One knowledge user suggested that they did not advocate for uptake of the study findings because the timing was not right for them to feasibly act on the findings. The views of several researchers and knowledge users suggested that, although knowledge users bring their perspectives to the research team, there may be ambiguity concerning their role with respect to disseminating or promoting uptake of study findings in their organization.

In addition to the problem of ambiguity concerning the role of knowledge users, opportunities for integrated kt within the service organization in which those individuals are situated might be insufficient. One of the funded studies explicitly examined opportunities and challenges to integrated kt in 3 cancer system programs: colorectal cancer screening, pancreatic cancer, and prostate cancer9. Several of the challenges to integrated kt identified by Gagliardi et al.9 were also identified by the knowledge users who were interviewed. Gagliardi et al. found that barriers to integrated kt at the cancer-system organization included “recognition of, and support for i [integrated] kt through leadership, mentors, champions and brokers; and a mismatch in timing and goals between researchers and decision-makers.” Those challenges, together with ambiguity concerning the role of knowledge users on individual studies might contribute to barriers to uptake of study findings.

Intensive Interpersonal Networking

Researchers perceived that intensive interpersonal networking was required to generate interest in the findings of their study and subsequent uptake. Personal networking was described as being time-consuming and potentially non-productive if knowledge users were not aware of their role.

Researcher Persistence in Promoting Uptake

Lack of researcher persistence in promoting the results of their study was mentioned by one participant as an important barrier to uptake: “The fault is mine that I haven’t undertaken additional activities to promote awareness of and understanding of the implications of the findings” (Researcher 1). Other researchers did not raise this issue. The view expressed by Researcher 1 was upheld by a stakeholder who felt that researcher persistence was even more relevant when considering that institutional memory is short because of the varying priorities of stakeholders and the personnel turnover within agencies, particularly in governmental institutions, with research “receptors” changing in the absence of established ways to translate knowledge from a previous receptor to a new one. The stakeholder suggested that it was incumbent upon the researcher to keep checking in and pushing results that could have relevance to a particular institution—although it is unclear whether a single researcher would achieve success in a large institution.

Question 3: To What Extent Have KT-Net’s Grant Competitions Reached a Broad and Diverse Range of Researchers and Knowledge Users?

The primary source of data for this question was the review of documents. There is evidence that the funding program reached a broad and diverse range of researchers and knowledge users. Applicants were affiliated with 5 universities, including the Northern Ontario School of Medicine. Researchers named on each study team included academic researchers (such as medical oncologists, surgeons, and clinical epidemiologists) and clinical staff (such as nurses, social workers, clinical psychologists, and kinesiologists), and decision-makers at the cancer-system organization. Researchers also drew a diverse range of knowledge users into their research teams, including funders, clinicians, allied health team members, hospital administrators, provincial policymakers, managers, and executives at the cancer-system organization, and also key opinion leaders and decision-makers from various areas of study.

Question 4: What Would Be the Effect on Funded Researchers and Knowledge Users If the Grant Competition No Longer Existed? What Would Be the Effect on the Uptake of More Effective Cancer Health Services and Products?

More than half the interviewed researchers indicated that losing the competition would negatively affect them as a community of kt researchers because not many funders understand kt or are specifically focused on cancer-relevant kt. Another researcher said that, if the grant competition no longer existed, it would “be missed and leave a hole,” because the mere existence of the competition has raised the profile of kt.

The loss of the funding program in relation to its effect on cancer health services and products was something that neither researchers nor stakeholders could clearly determine. The question elicited a range of responses, with several participants being unsure of an effect or perceiving that there would be no effect. Others believed that there would be some loss. But the participants who anticipated some loss also acknowledged that the cancer care system would not “collapse” should the grant competition cease to exist.

Reviewers who were asked to weigh in on these issues voiced the same concern as researchers about the loss of funds for an already under-resourced area of research. One reviewer also perceived that losing the competitions would potentially have negative effects on patient outcomes:

If [cancer-system organization] decides that the purpose is to actually support bringing research into practice and studying that process, then I think that not having the competition would ultimately be detrimental to patient outcomes.

— Reviewer 2

The hope for most researchers, stakeholders, and reviewers alike was that the grant competition would continue to provide a niche opportunity for cancer-relevant kt researchers.

Question 5: What Are the Unanticipated Outcomes, Positive or Negative, Resulting from KT-Net’s Grant Competitions?

We had hoped for positive unanticipated outcomes from the grants program, such as legitimization of kt research in cancer care and increased awareness of kt science and research by cancer system leaders. We did find some evidence that several cancer system leaders were aware of the funding program; however, we did not find any evidence of other unanticipated outcomes. We did note that the cancer-system organization launched its own funding program in 2018. Although kt priorities were included in that funding call, we cannot attribute the organization’s inclusion of kt research to the KT-Net grants program.

Participants suggested that stronger links between KT-Net as the funder and the cancer-system organization were needed. Stakeholders affiliated with the provincial cancer-system organization provided the insight that the funded projects might have been mission-aligned with oicr priorities, but not necessarily with the priorities of the cancer-system organization, which meant that any results might not achieve the hoped-for uptake:

You know, so if I look at what is—what was—a huge thing that we [the cancer-system organization] were rolling out, in cancer surgery over the last five years, was a new way of centralizing high-complexity thoracic surgery. So there wasn’t a [KT-Net] project associated with that. So if I think that was one of our great big priorities, and there wasn’t a project that supported that, instead there was something on a pathway for pancreatic [cancer surgery].

— Knowledge user 3

To forge stronger links between the cancer-system organization and the funder, stakeholders suggested that representatives from the organization’s screening and clinical programs be included on KT-Net’s Advisory Committee. A second suggestion was to involve representatives during development of the call for proposals and to be explicit with respect to the role of those representatives as “institutional champions” of new knowledge from funded studies, which could encourage uptake of findings at the organization. With optimism, stakeholders reported that a newly hired chief research officer at the organization would hopefully establish effective strategies for better use of research evidence by the organization—and, by extension, better use of relevant research evidence generated by KT-Net–funded studies.

Researchers indicated that funders could play a greater role in promoting and using findings from studies. That sentiment was indirectly corroborated by stakeholders, who commented how discussion of research findings was not consistently on the agenda at higher-level cancer-system organization meetings; consequently, awareness of study findings was perceived to be low. Stakeholders indicated that several attempts had been made to have researchers attend relevant meetings to speak about their work and its implications for the organization, but that effort had not been sustained:

Now that I am looking back, I am thinking why did [name of individual] never talk about this at the [name of cancer health system decision-maker meeting] and the attempt we made to have the researchers come to that [meeting]. It felt like a one-off effort instead of being embedded in the work.

— Stakeholder 1

Stakeholders recommended that findings from kt studies be shared through a formalized process—for example, bringing together stakeholders from the cancer-system organization in meetings in which funded researchers could share new knowledge from their studies. Such meetings could be understood as a first step to uptake of the findings.

Another area in which researchers felt that funders could take a greater role was the brokering of partnerships between researchers and knowledge users where existing relationships do not exist. Such brokering could expedite the process of assembling of an interdisciplinary research team at the proposal stage and could also aid execution of the study, if funded. The efforts involved in assembling an interdisciplinary research team with appropriate knowledge users was described as a huge impediment to the submission of kt proposals and the execution of studies, particularly in KT-Net grants, whose funding timeframe is relatively short (1–2 years):

It would be great if there was some vehicle for your team—[the] KT-Net team—to sort of broker, I could say, some of those relationships. That would be really fantastic. Because it takes time to develop those relationships with the right people, and in the course of a short grant, you know that is sometimes a little bit challenging.

— Researcher 3

Question 6: What Are Stakeholders’ Views of the Grant Competition Procedures, Including Proposal Submission, Review Process, and Research Agreement?

The grant competition processes, including those related to application, review, and grant administration were viewed as streamlined and straightforward. Stakeholders and researchers alike saw the application process as “easy,” with researchers appreciating the clear instructions provided, together with requirements for a clear and concise application. Participants felt that those aspects made applying for funding less intensive than was the case with some other national-level funding opportunities that were termed “administratively heavy”:

It was fantastic to apply for, because you actually just had to be clear and concise, and you didn’t have to be long winded, so that was very very nice.

— Researcher 5

One applicant who self-identified as a new kt researcher said that, initially, kt frameworks (a requirement in the call for proposals) had been unfamiliar to them. That applicant therefore reached out to a more experienced kt researcher for guidance and advice, which allowed for the submission of a higher-quality proposal that was awarded funding.

Members of the grant Review Panel observed that the review process had progressively evolved with each funding cycle and that the grant competitions now have a standard and thorough review process. They indicated that communication with administrators is good and that reviewers are provided with sufficient turnaround time to assess proposals. Working particularly well is the tiered system of proposals (first being reviewed individually, and then collectively with the entire Review Panel), a process that one participant noted has certain collaborative merits when compared with the purely online review mechanisms of other competitions. Like the applicants, the reviewers appreciated the more streamlined application process and felt that enough information was provided by researchers.

Applicants appreciated the quick response from administrators with respect to the funding decision and timely communication throughout the funding period. Stakeholders and applicants both perceived that providing reviewer comments (which had not consistently been included as feedback to applicants) should be the norm, because feedback could provide constructive criticism for improvement and development of higher-quality proposals. Other general recommendations by reviewers included increasing the number of people reviewing each proposal (currently 3 reviewers per proposal), which would result in greater reliability, although it was noted that such a change might not always be feasible. Lastly, a reviewer noted the importance of methodology in research, emphasizing that the quality of the methods used should be an important criterion when assessing proposals.

A reviewer with expertise in kt noted that, when reviewing proposals, it was not always clear whether the priority for funding should be kt science, kt practice, or end-of-grant kt. That priority was similarly unclear to applicants and reviewers. Reviewers had different ideas about priorities for funding, and they indicated that the priorities of the cancer-system organization for funding (if any) were unclear.

Question 7: What Are Your Suggestions to Improve the KT-Net Grant Competitions?

Suggestions to improve the grant competitions included providing additional clarity in the call for proposals about the types of kt studies to be funded, adding a letter-of-intent stage to the competition, and providing feedback to applicants about their proposals. Other key suggestions included facilitating closer collaboration between the funder and the cancer-system organization during the call for proposals stage, providing additional clarity about the role of knowledge users, and helping applicants to broker relationships with knowledge users.

DISCUSSION

The key findings of this evaluation highlight the achievement of several aims of the cancer kt grant competitions: building capacity in cancer kt research, especially among researchers new to kt; contributing to incremental advances in the science of kt; building strategic partnerships; and leveraging funding. The evaluation also highlighted perceived facilitators and barriers to achieving the aims of the funding program. One of the barriers was the extent of the partnership between the funder (KT-Net) and the cancer-system organization. Facilitators and barriers to uptake of the research results from individual funded studies were identified. Knowledge users were identified as key facilitators in uptake, although ambiguity about the expectations for their role remained.

With respect to knowledge users on individual studies, we observed a range of involvement and noted elements that made for successful collaboration: established partnerships, knowledge user interest in the research topic and perceived benefit from participation, relatively little burden on knowledge users, and a valuing of knowledge user opinion by researchers. Although knowledge user participation could be viewed as critical for kt projects, there is ambiguity with respect to the extent of their involvement as perceived by researchers and knowledge users themselves. That situation then raises the question of what a funder can do to resolve the lack of clarity surrounding the knowledge user role. It might be incumbent on funders to clarify the expectations for knowledge users within the funding requirements, such as the call for proposals. Memoranda of understanding or position descriptions might also help. At the outset, it is important to anticipate how study results might be used, and researchers have to be cognizant of the importance of including decision-makers with the capacity for effective uptake as knowledge users on the study team. It might be difficult for researchers to identify potential research findings receptors within a large organization, thus making the recruitment of knowledge users challenging. A suggestion emerging from our study was therefore to have funders broker relationships between researchers and knowledge users.

Several stakeholders (but not researchers) perceived that the amount and duration of funding limited the ability of the grant competitions to contribute to advancing the science of kt. However, the competitions were intended to provide opportunities for researchers to obtain pilot data so that they could subsequently apply to other agencies for additional and potentially larger funding amounts. The average amount of funding was approximately CA$125,000 for a 1- to 2-year study. In comparison, the average Knowledge to Action grant awarded by cihr is CA$151,11528. In the United States, the average per-year allocation is US$556,597 for an R01 grant, US$164,923 for an R21 grant, and US$75,083 for an R03 grant29. The KT-Net grant competitions appear to be most closely aligned with the goals of the cihr’s Knowledge to Action grants and the U.S. R21 grants, which focus on exploratory or developmental research, with a limit of US$275,000 for a 2-year project and a limit of US$200,000 in any single year (https://grants.nih.gov/grants/funding/r21.htm). It should be noted that, in the United States, applicants can include researcher salary support, making direct comparisons with the Canadian setting difficult. Ontario’s Adopting Research to Improve Care program provided total funding for direct costs of CA$7.6 million (an average of CA$1.27 million per project over 2–3 years)30. However, that program is more extensive and geared toward scaling up interventions province-wide, which likely accounts for the higher dollar amount per study. We do not know if a larger amount would have produced a greater advancement in the science of cancer kt.

One of our key lessons was that the funding program has to improve its strategic partnership with the cancer-system organization. In a systematic review of community–academic partnerships, Drahota and colleagues31 found that key facilitators included trust and respect between the partners, shared vision and goals, and effective communication. Barriers were lack of trust and respect, lack of shared goals, and ineffective communication31. An important consideration is that a partnership cannot be based on any one individual, because the partnership will be at risk should the individual leave the organization or change their role. Other aspects of the partnership can include monetary or in-kind contributions of each partner, as in the cihr’s Partnership for Health System Improvement grants. Specific partnership agreements that specify the extent of the partnership between organizations could be helpful. However, an improved partnership at the organizational level might not mitigate against challenges to integrated kt at the program level9.

We also noted a perception on the part of stakeholders that funders should have a greater role in brokering the relationship between individual researchers and decision-makers at the cancer system organization. Funders can shape the direction of competitions with partners insofar as they control the guidelines for funding applications. Funders can insist that knowledge users be part of the funding team from the outset of the application, as is the case with the cihr Partnerships for Health Systems Improvement grants28 and could require evidence that a partnership goes further than the contribution of a letter or signature. McLean et al.28, in their evaluation of the cihr kt funding program, commented that the role of the cihr has been strategic with respect to the uptake of knowledge. The cihr kt program has deliberately included support for partnered integrated kt grants that integrate knowledge users and consider knowledge user needs. Sibbald et al.32 studied the cihr’s integrated kt funding program and, like the participants in our study, suggested that funders could take a greater role in facilitating and sustaining partnerships in awarded grants.

Several funding agencies believe that funders could play a greater role in implementation of findings. Brantnell and colleagues33 collected information from 10 Swedish funding agencies about their roles with respect to implementation. They found that half the funders believed that it was the funder’s role to be responsible for clinical implementation. Other identified roles were more traditional for funders and related to advocacy: raising the profile of implementation as something important among decision-makers and the public, monitoring implementation outcomes, and disseminating knowledge33.

Several suggestions by the evaluation participants were incorporated into subsequent grant competitions to strengthen the partnership between the funder and the cancer-system organization. The funding competition instituted a letter-of-intent phase, invited the new chief research officer at the cancer-system organization to join the competition’s Advisory Committee, and required a detailed letter of support from the study knowledge user or users. New guidance to applicants in the call for proposals indicated that a description of the knowledge users and their position in the organization would be required in addition to the letter of support. In addition, guidance for the content of the letter was provided. Letter-writers were instructed to include a description of how the project is relevant to a priority area or areas; why the specified problem is relevant to them and their organization; their role in the development of the proposal and in the project if funded; and how their organization would use the findings and facilitate dissemination. Additional steps would likely be needed to build an effective partnership that includes trust and respect between partners, shared vision and goals, and effective communication.

Researchers and stakeholders both suggested greater clarity about either kt frameworks (researchers) or kt research studies (stakeholders). To address the lack of knowledge about kt frameworks, KT-Net provided applicants to subsequent competitions with a link to relevant implementation frameworks found in the literature34. However, it is recognized that provision of that information might be insufficient to raise awareness of appropriate frameworks by researchers, especially those who are new to the field of kt. To address disagreements between reviewers about what constitutes kt research, KT-Net provided, in later competitions, additional instructions for reviewers about how to grade applications. Each proposal was read by 3 reviewers, and any disagreements were addressed during meetings. We acknowledge that instructions or even discussions might not mitigate fundamental disagreements on the part of reviewers about what constitutes kt research.

Study Limitations and Strengths

We attempted to recruit a diverse range of stakeholders from funded and unfunded studies; however, only 2 individuals from unfunded studies consented to participate in either an interview or the provision of written comments by e-mail. All e-mail feedback was incorporated into the presentation of the results. We cannot know whether, from the perspective of those individuals, there were additional barriers to applying for funding.

Strengths of the evaluation were that data from four competition cohorts and information collected from both a document review and from interviews with a range of participants (researchers, knowledge users, and stakeholders) were included. Interview data provided more nuanced information about each evaluation question.

CONCLUSIONS

Our evaluation revealed that this novel cancer kt grants program has generally met its aims. The program has built capacity in cancer kt research among early-career kt researchers, made contributions to incrementally advancing the science of kt, and through the cancer kt–funded projects, allowed for the leveraging of subsequent funding opportunities. Although the kt grants program did not require the engagement of knowledge users on the research team, many were so engaged, revealing that the involvement of knowledge users can be an important facilitator to achieving outcomes during kt studies. Suggestions for improving the program included promoting the partnership between the funder and the cancer-system organization and facilitating more cancer-system engagement with the program. A relatively modest financial investment in the cancer kt grants program has paid off, although there is room for improvement.

ACKNOWLEDGMENTS

We extend our thanks to the study participants, who generously gave their time to participate in the interviews. IDG is a cihr Foundation Grant recipient (fdn no. 143237). A brief report of the findings of this study was submitted to Cancer Care Ontario in November 2015.

This study was conducted with the support of the Ontario Institute for Cancer Research and Cancer Care Ontario through funding provided by the Government of Ontario.

CONFLICT OF INTEREST DISCLOSURES

We have read and understood Current Oncology’s policy on disclosing conflicts of interest, and we declare the following interests: MAO is the Scientific Associate for KT-Net and led the evaluation. MAO had no involvement in collecting data for the evaluation. EG is the Director for KT-Net and was a co-investigator for one of the funded studies. EG had no involvement in collecting or analyzing data for the evaluation. IDG was a co-investigator for one of the KT-Net–funded studies. IDG had no involvement in collecting or analyzing data for the evaluation. MCB was principal investigator for two of the funded studies and co-investigator for another two studies. MCB had no involvement in collecting or analyzing data for the evaluation. The remaining authors have no conflicts of interest to disclose.

AUTHOR AFFILIATIONS

*Department of Family and Community Medicine, University of Toronto, Toronto, ON,
Centre for Urban Health Solutions, St. Michael’s Hospital, Toronto, ON,
School of Epidemiology and Public Health, Faculty of Medicine, University of Ottawa, Ottawa, ON,
§The Ottawa Hospital Research Institute, Ottawa, ON,
||Tom Baker Cancer Centre, Calgary, AB,
#University of Calgary, Calgary, AB,
**ices, Toronto, ON,
††Ontario Institute for Cancer Research, Toronto, ON,
‡‡Department of Oncology, McMaster University, Hamilton, ON.

REFERENCES

1 Ioannidis JP, Greenland S, Hlatky MA, et al. Increasing value and reducing waste in research design, conduct, and analysis. Lancet 2014;383:166–75.

2 Macleod MR, Michie S, Roberts I, et al. Biomedical research: increasing value, reducing waste. Lancet 2014;383:101–4.

3 Woolf SH, Johnson RE. The break-even point: when medical advances are less important than improving the fidelity with which they are delivered. Ann Fam Med 2005;3:545–52.

4 Grunfeld E, Zitzelsberger L, Evans WK, et al. Better knowledge translation for effective cancer control: a priority for action. Cancer Causes Control 2004;15:503–10.

5 Antman EM, Lau J, Kupelnick B, Mosteller F, Chalmers TC. A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts: treatments for myocardial infarction. JAMA 1992;268:240–8.

6 Rabeneck L, Paszat LF. A population-based estimate of the extent of colorectal cancer screening in Ontario. Am J Gastroenterol 2004;99:1141–4.

7 Barbera L, Seow H, Husain A, et al. Opioid prescription after pain assessment: a population-based cohort of elderly patients with cancer. J Clin Oncol 2012;30:1095–9.

8 Graham ID, Logan J, Harrison MB, et al. Lost in knowledge translation: time for a map? J Contin Educ Health Prof 2006;26:13–24.

9 Gagliardi AR, Webster F, Brouwers MC, Baxter NN, Finelli A, Gallinger S. How does context influence collaborative decision-making for health services planning, delivery and evaluation? BMC Health Serv Res 2014;14:545.

10 Lim C, Sekhon HS, Cutz JC, et al. Improving molecular testing and personalized medicine in non-small-cell lung cancer in Ontario. Curr Oncol 2017;24:103–10.

11 Bombard Y, Rozmovits L, Trudeau M, Leighl NB, Deal K, Marshall DA. Access to personalized medicine: factors influencing the use and value of gene expression profiling in breast cancer treatment. Curr Oncol 2014;21:e426–33.

12 Bombard Y, Rozmovits L, Trudeau ME, Leighl NB, Deal K, Marshall DA. Patients’ perceptions of gene expression profiling in breast cancer treatment decisions. Curr Oncol 2014;21:e203–11.

13 Bombard Y, Rozmovits L, Trudeau M, Leighl NB, Deal K, Marshall DA. The value of personalizing medicine: medical oncologists’ views on gene expression profiling in breast cancer treatment. Oncologist 2015;20:351–6.

14 MacDonald KV, Bombard Y, Deal K, Trudeau M, Leighl N, Marshall DA. The influence of gene expression profiling on decisional conflict in chemotherapy treatment decision-making for early-stage breast cancer. Eur J Cancer 2016;61:85–93.

15 Marshall DA, Deal K, Bombard Y, Leighl N, MacDonald KV, Trudeau M. How do women trade-off benefits and risks in chemotherapy treatment decisions based on gene expression profiling for early-stage breast cancer? A discrete choice experiment. BMJ Open 2016;6:e010981. [Erratum in: BMJ Open 2016;6:e010981corr1]

16 Wei AC, Devitt KS, Ramjaun A, Gallinger S. Developing an evidence based clinical pathway for patients undergoing pancreaticoduodenectomy [abstract 129]. CMAJ 2012;55 (suppl 2):S109.

17 Wei AC, Devitt KS, Ahmed M, et al. Implementing an evidence-based clinical pathway for patients undergoing pancreaticoduodenectomy: outcomes of a pilot project [abstract 119]. CMAJ 2013;56(suppl 3):S122.

18 Feldman-Stewart D, Madarnas Y, Mates M, et al. Information needs of post-menopausal women with hormone receptor positive early-stage breast cancer considering adjuvant endocrine therapy. Patient Educ Couns 2013;93:114–21.

19 Feldman-Stewart D, Madarnas Y, Mates M, et al. Information for decision making by post-menopausal women with hormone receptor positive early-stage breast cancer considering adjuvant endocrine therapy. Breast 2013;22:919–25.

20 Brouwers MC, Garcia K, Makarski J, Daraz L on behalf of the Evidence Expert Panel, KT for Cancer Control in Canada Project Research Team. The landscape of knowledge translation interventions in cancer control: what do we know and where to next? A review of systematic reviews. Implement Sci 2011;6:130.

21 Sandelowski M. Whatever happened to qualitative description? Res Nurs Health 2000;23:334–40.

22 McLean RK, Graham ID, Bosompra K, et al. Understanding the performance and impact of public knowledge translation funding interventions: protocol for an evaluation of Canadian Institutes of Health Research knowledge translation funding programs. Implement Sci 2012;7:57.

23 Dillman DA. Mail and Internet Surveys: The Tailored Design Method. 2nd ed. New York, NY: John Wiley and Sons; 2000.

24 Charmaz K. Constructing Grounded Theory. 2nd ed. Thousand Oaks, CA: Sage Publications; 2014.

25 Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Heal Res 2005;15:1277–88.

26 Crabtree BF, Miller WL. Doing Qualitative Research. 2nd ed. Thousand Oaks, CA: Sage Publications; 1999.

27 Guba EG, Lincoln YS. Fourth Generation Evaluation. Thousand Oaks, CA: Sage Publications; 1989.

28 McLean R, Tucker J. Evaluation of CIHR’s Knowledge Translation Funding Program [Web page]. Ottawa, ON: Canadian Institutes of Health Research; 2013. [Available at: http://www.cihr-irsc.gc.ca/e/47332.html; cited 15 March 2019]

29 Neta G, Sanchez MA, Chambers DA, et al. Implementation science in cancer prevention and control: a decade of grant funding by the National Cancer Institute and future directions. Implement Sci 2015;10:4.

30 Moore JE, Grouchy M, Graham ID, Shandling M, Doyle W, Straus SE. The Council of Academic Hospitals of Ontario (caho) Adopting Research to Improve Care (artic) program: reach, sustainability, spread and lessons learned from an implementation funding model. Healthc Policy 2016;11:27–39.

31 Drahota A, Meza RD, Brikho B, et al. Community–academic partnerships: a systematic review of the state of the literature and recommendations for future research. Milbank Q 2016;94:163–214.

32 Sibbald SL, Tetroe J, Graham ID. Research funder required research partnerships: a qualitative inquiry. Implement Sci 2014;9:176.

33 Brantnell A, Baraldi E, van Achterberg T, Winblad U. Research funders’ roles and perceived responsibilities in relation to the implementation of clinical research results: a multiple case study of Swedish research funders. Implement Sci 2015;10:100.

34 Tabak RG, Khoong EC, Chambers DA, Brownson RC. Bridging research and practice: models for dissemination and implementation research. Am J Prev Med 2012;43:337–50.


Correspondence to: Mary Ann O’Brien, Department of Family and Community Medicine, University of Toronto, 500 University Avenue, Fifth Floor, Toronto, Ontario M5G 1V7. E-mail: maryann.obrien@utoronto.ca

(Return to Top)


Current Oncology, VOLUME 26, NUMBER 4, August 2019








Copyright © 2019 Multimed Inc.
ISSN: 1198-0052 (Print) ISSN: 1718-7729 (Online)