Inter-laboratory proficiency testing scheme for tumour next-generation sequencing in Ontario: a pilot study

Original Article

Inter-laboratory proficiency testing scheme for tumour next-generation sequencing in Ontario: a pilot study

T. Spence, PhD*, N. Stickle*, C. Yu, CCRP*, H. Chow*, H. Feilotter, PhD, B. Lo, MD, PhD, E. McCready, PhD§, B. Sadikovic, PhD||, L.L. Siu, MD*, P.L. Bedard, MD*, T.L. Stockley, PhD*




A pilot inter-laboratory proficiency scheme for 5 Ontario clinical laboratories testing tumour samples for the Ontario-wide Cancer Targeted Nucleic Acid Evaluation (octane) study was undertaken to assess proficiency in the identification and reporting of next-generation sequencing (ngs) test results in solid tumour testing from archival formalin-fixed, paraffin-embedded (ffpe) tissue.


One laboratory served as the reference centre and provided samples to 4 participating laboratories. An analyte-based approach was applied: each participating laboratory received 10 ffpe tissue specimens profiled at the reference centre, with tumour site and histology provided. Laboratories performed testing per their standard ngs tumour test protocols. Items returned for assessment included genes and variants that would be typically reported in routine clinical testing and variant call format (vcf) files to allow for assessment of ngs technical quality.


Two main aspects were assessed:

  • ■ Technical quality and accuracy of identification of exonic variants

  • ■ Site-specific reporting practices

Technical assessment included evaluation of exonic variant identification, quality assessment of the vcf files to evaluate base calling, variant allele frequency, and depth of coverage for all exonic variants. Concordance at 100% was observed from all sites in the technical identification of 98 exonic variants across the 10 cases. Variability between laboratories in the choice of variants considered clinically reportable was significant. Of the 38 variants reported as clinically relevant by at least 1 site, only 3 variants were concordantly reported by all participating centres as clinically relevant.


Although excellent technical concordance for ngs tumour profiling was observed across participating institutions, differences in the reporting of clinically relevant variants were observed, highlighting reporting as a gap where consensus on the part of Ontario laboratories is needed.

KEYWORDS: External quality assessment, inter-laboratory comparison, next-generation sequencing, tumour molecular profiling


Next-generation sequencing (ngs) for molecular profiling of solid tumours is rapidly becoming standard-of-care testing in molecular laboratories in Canada because of the simultaneous yield of clinically useful genetic information, benefit of tissue preservation by avoiding sequential testing, and declining cost of ngs equipment and operations. Proficiency testing (pt) schemes (also known as external quality assessment if offered by an external body), in which laboratories are assessed on blinded analysis of samples with known results, is an integral part of ensuring quality of ngs and other molecular tests1,2. However, a national pt scheme for molecular pathology laboratory testing (using ngs or any method) does not exist in Canada, likely because of the effort of establishing a scheme potentially subscribed to by only a small number of Canadian laboratories, and because of the effort of sample acquisition and pt assessment. In Ontario, all licensed and accredited clinical laboratories offering testing for patient care are required to participate in pt as mandated by the provincial accreditation body (Institute for Quality Management in Healthcare), based on the International Organization for Standardization 15189 standard. The Institute for Quality Management in Healthcare requires that, to maintain accreditation, clinical laboratories complete pt for 4 samples within a 12-month period for each clinical test. In the absence of a pt program offered by a Canadian organization for tumour molecular profiling, accredited laboratories must identify suitable alternatives, such as international pt programs or informal sample exchange, to meet the Ontario requirements.

A key consideration in the design of a pt scheme for tumour molecular profiling is the selection of sample source material. To perform both pre-analytic and analytic comparisons of laboratory proficiency, the optimal material is formalin-fixed, paraffin-embedded (ffpe) tumour tissue, because that sample source allows for an assessment of pre-analytic variables. However, obtaining ffpe tumour tissue for pt is often hampered by a small tumour amount and suboptimal quality of the available clinically relevant material. Tumour heterogeneity can also lead to potential differences in results. Although tracking of ffpe sections sent to participating laboratories is possible, that approach does not ameliorate the risk of error. An alternative approach is to use dna extracted from ffpe tissue3, with the inherent limitation that use of dna prevents identification of any potential issues related to the pre-analytic phase. Other source material could also be used, such as cell lines embedded in paraffin or synthetic dna controls, each with its own limitations. Any of those sample issues might lead to inappropriate discrepancies in pt testing results originating solely in the material sent within the pt scheme.

The other significant aspect in pt schemes is whether the scheme assesses only the technical aspects (for example, by requesting return of variants only) or also assesses the post-analytic clinical interpretation and reporting aspects (for example, by requesting that variant interpretations or mock clinical reports be returned). For testing of solid tumours, general guidelines about reporting aspects for laboratory tests are available4; however, those guidelines might not be sufficiently detailed for pt schemes, which are often specific to a gene or a disease indication. Proficiency testing schemes might also request return of various types of data—for example, only the Human Genome Variation Society nomenclature for identified variants, or variants plus data files for data quality analysis, which typically compares data across laboratories rather than scoring based on an evaluative scheme.

In the present study, a pilot pt scheme (Figure 1) was implemented for Ontario laboratories participating in the Ontario-Wide Cancer Targeted Nucleic Acid Evaluation (octane) study, which is an ongoing prospective trial open at 5 academic cancer centres. The trial aims to enable genotype–drug matching through somatic ngs testing of ffpe solid tumour tissue from patients with advanced cancer and to facilitate clinical and genomic data-sharing. The pt scheme was designed as a pre- to post-analytic scheme, with dissemination of ffpe tumour material and return of variants, variant call format (vcf) files, and information about clinically reportable genes and variants. We highlight the successes and challenges of that approach to pt schemes for solid tumour molecular profiling in the Ontario context.



FIGURE 1 Flow-chart depicting the pilot proficiency-testing workflow. Ten formalin-fixed, paraffin-embedded (FFPE) tumour proficiency testing cases with known variants from prior testing at the reference centre were distributed to each of the 4 participating laboratories, with tumour site and histology indicated. Cases were processed at participating centres per routine clinical testing for the Ontario-wide Cancer Targeted Nucleic Acid Evaluation (OCTANE) study. Participating centres were requested to provide a list of the genes reported for each tumour site, clinically reportable variants, and variant call format (VCF) files. Comparative assessment was performed to evaluate concordance in the technical identification of variants and concordance in the clinically reported and annotated variants. H/E = hematoxylin and eosin stained; HGVS = Human Genome Variation Society.


Participating Laboratories

Participating laboratories included the Princess Margaret Cancer Centre (pmcc), University Health Network, Toronto (which acted as the reference site for source materials and result evaluations); the Juravinski Cancer Centre (jcc), Hamilton; the London Health Sciences Centre (lhsc), London; The Ottawa Hospital (toh), Ottawa; and the Kingston Health Sciences Centre (khsc), Kingston. A laboratory director from each site was involved in the study design.

Tumour Tissue Specimens

From a cohort of ffpe tissues banked at the reference laboratory, 10 ffpe specimens of tumour tissue were chosen from resections in patients enrolled in the octane trial (NCT02906943 at with approval from the Ontario Cancer Research Ethics Board (ID: 16-018). All 10 specimens had previously been tested by ngs at the reference laboratory. Specimens were chosen so as to provide to each of the 4 participating sites a variety of genes and variants, and to meet these additional criteria: tumour cellularity greater than or equal to 20% in the tumour area, and sufficient ffpe material to provide 2 unstained sections at 7 μm thickness and 1 slide stained with hematoxylin and eosin. Sections were numbered as cut, and the section numbers distributed to each laboratory were recorded to enable tracking of the location within the tumour block in case of tumour heterogeneity for variants. Unstained and uncircled sections were provided on air-dried, uncoated slides. Tumour site and histology as determined by a study pathologist at the reference site were also provided to the participating laboratories.

Molecular Profiling Assays

The participating laboratories used 2 ngs panels in testing dna samples. The reference laboratory (pmcc) used a custom hybridization capture ngs panel of 555 cancer-related genes [UHN Hi5 panel (SureSelect: Agilent, Santa Clara, CA, U.S.A.)] sequenced on the NextSeq platform (Illumina, San Diego, CA, U.S.A.). The other 4 participating laboratories (jcc, lhsc, toh, khsc) used a commercial amplicon-based hotspot panel that included regions of 50 genes (Ion AmpliSeq Cancer Hotspot Panel v2: Thermo Fisher, Waltham, MA, U.S.A.) and was sequenced on the Ion Torrent PGM platform (Thermo Fisher) at each site. Those 4 laboratories were accredited by the Ontario provincial laboratory accreditation body (the Institute for Quality Management in Healthcare), and the remaining laboratory (pmcc) was accredited by the College of American Pathologists (cap) and was certified as meeting the U.S. Clinical Laboratory Improvement Amendments. Laboratories were instructed to not communicate results with other laboratories and to treat the samples in a manner similar to other clinical samples as much as possible for the entire workflow.

Variant Assessment and Reporting

For each of the 10 specimens, participating laboratories were requested to return the following information to the reference laboratory within 4 weeks (supplemental Appendix 1):

  • ■ A list of genes for each tumour site the laboratory would consider to be clinically reportable from the panel in use

  • ■ The variants identified in each of the 10 specimens that the laboratory considered to be clinically reportable, with variant interpretations

  • ■ A list of variants that met laboratory-defined minimum technical quality metrics for high-quality variants (that is, all high-quality variants whether considered clinically reportable or not)

  • ■ The vcf files from the relevant ngs panel test at each site

The khsc provided vcf files that were pre-filtered according to their current laboratory practice; the jcc, toh, and lhsc provided unfiltered vcf files. Clinically reportable variants were defined as variants that would routinely be reported based on the clinical reporting practices for the octane study at that laboratory. Variant interpretations were requested to be performed and returned according to each laboratory’s typical process for the octane study. Laboratories were requested to include Human Genome Variation Society nomenclature and any other nomenclature system typically used for reports generated for the octane study.

Assessment of Technical and Reporting Performance

A comparative analysis was conducted to assess concordance in the variant information returned from each participating site with the variants identified by the reference site. Because the reference site used different ngs chemistry, analyses included assessment of concordance between the participating sites, but discordance with the reference laboratory results. That approach was instituted to rule out the bias of considering the reference laboratory results to be “true.”

Variants were assessed by the accuracy with which reported variants were identified, including correct Human Genome Variation Society nomenclature. Technical quality metrics were assessed using the vcf files and BEDTools (version 2.23.0)5, with an intersection browser extensible data file created to identify overlapping regions in the UHN Hi5 panel and the Ion AmpliSeq Cancer Hotspot Panel v2, and to identify common and unique variants in the reference laboratory dataset and in each of the datasets from the 4 participating laboratories.

To assess reporting, the genes and variants that each laboratory provided as being clinically reportable were manually compared.


Reportable Genes by Tumour Site

Participating laboratories provided the list of genes typically reported clinically at their institution for each tumour site in the pt specimens, per their usual practice within the octane study. The list of reportable genes for each tumour site differed significantly between the participating centres (Table I). Of the participating laboratories, 2 chose to report on all genes on their panel, and 1 elected to report only variants in genes that were routinely reported in clinical practice at their institution, regardless of tumour type.

TABLE I List of genes that each participating laboratory would consider to be reportable for each tumour type included within the proficiency-testing specimens


Comparison of Base Calling and Quality Assessment of Raw Data

An analysis of the agreement in technical detection of variants identified in the vcf files for each case demonstrated 100% concordance (98 of 98 variants) in the identification of exonic variants from all sites for the 10 cases (Table II). Of the 98 exonic variants detected, a subset was identified below the lower variant allele frequency (vaf) and quality thresholds defined for pt evaluation—that is, less than 5% vaf and less than 100× (pmcc) or 500× (jcc, lhsc, toh, khsc) coverage. Of those low-vaf or low-coverage variants, none was considered clinically reportable by more than 1 laboratory.

TABLE II Comparison of base calling and quality assessment of raw data for 98 exonic variants


Variants Considered Clinically Reportable

The pt results provided by each institution included a list of variants in each case that were considered clinically reportable per routine practice in the octane study (Table III). A high degree of variability in the variants considered clinically reportable was also observed, with only 3 variants from the 10 cases being concordantly reported by all 5 participating centres. Concordant reporting of 5 variants by 4 or more centres and of 10 variants by 3 or more centres was observed.

TABLE III Clinically reported variants, variant annotations, and number of concordantly reported variants by participant site


Interpretation of Clinically Reportable Variants

Variants considered clinically reportable were classified by 4 of the laboratories using a published somatic variant classification scheme. The joint guideline from the Association for Molecular Pathology (amp), the American Society of Clinical Oncology (asco), and cap (amp/asco/cap) published by Li et al.6 was applied by 3 laboratories, and the Sukhai et al.7 guideline was used by 1 laboratory. Significant variability was observed in the classifications provided for specific variants (Table III). For example, one laboratory indicated that the TP53 p.Leu252del variant identified in case 8 was a tier ii variant, while another indicated that the same variant was a tier iii variant. That same variant was classified as class 3A by a 3rd site and was not clinically reported by the remaining 2 laboratories. Similarly, the TP53 p.Ser127Phe variant identified in case 1 was reported as tier ii by 1 site and as tier iii by 1, with the other 3 sites not reporting it. Furthermore, 1 site chose to include variants classified as tier iii or iv according to the amp/asco/cap guideline as clinically reportable.


The present study set out to evaluate the performance of solid tumour molecular profiling at the 5 Ontario sites (jcc, lhsc, toh, khsc, and pmcc) that provide ngs molecular profiling for the octane study. An analyte-based pt approach was used, with ffpe tumour tissue being sent out, and information related to variants considered clinically relevant being returned, in an end-to-end evaluation of laboratory performance.

Although the use of ffpe tissue allows for an evaluation of the pre-analytic phase, it can also adversely affect other aspects of the pt scheme. At the reference laboratory, it was difficult to source sufficient ffpe tumour tissue material meeting all parameters specified in the Methods section for distribution to the 4 participant sites. Of the 10 samples, 4 were ovarian tumours, a site from which large surgical specimens with sufficient cellularity and lack of significant necrosis were more readily available. Only 2 lung cancer samples and 1 colorectal cancer sample were included because of the typically small tumour specimens obtained for those tumour types, although lung and colorectal cancers represent the tumours that most commonly undergo ngs testing as the standard of care in Ontario. That bias in sample selection is likely to have affected the interpretation or reporting aspects of this pt, because all participating laboratories routinely test lung and colorectal cancers, but only 3 of the 5 participating laboratories typically report on ovarian cancer.

Another issue related to the use of ffpe tumour tissue is potential tumour heterogeneity, which can contribute to variability in the detection of variants. A recent survey involving 111 laboratories assessed inter-laboratory technical performance for ngs-based solid tumour oncology assays and identified substantial agreement (>98%) in the accuracy of detection for single nucleotide variants occurring at a vaf more than 15%2. Indeed, although we observed no difference in the final exonic variants identified in the present study (98 of 98), variability in the vaf and depth of coverage was evident during quality assessment of the vcf file data. Although that variability might be attributable to tumoural heterogeneity, it might also result from differences in pre-analytic sample processing or ngs quality or in differences in the sequencing technology, and further delineating the causes of those differences in vaf and coverage depth is not possible. It is also noteworthy that, although complete concordance was observed in the technical identification of the 98 variants, 28 of 98 variants were identified by 1 or more sites below the lower vaf and quality threshold cut-offs defined in the study.

With respect to the interpretation and reporting of variants in tumour molecular profiling, discrepancies were observed: only 3 variants were selected as clinically reportable by all participating sites, and only 10 variants were concordantly reported by 3 or more sites. In part, those results reflected site-specific interpretation of the instructions for the pt scheme (supplemental Table 1), because some laboratories reported only variants in genes routinely reported in clinical practice at that institution (rather than those reported in the octane study), regardless of tumour type. This site-specific reporting practice highlights the significant gap in the classification of what is considered a “clinically reportable” variant in the Ontario context and likely reflects practice in other Canadian provinces, because national standards for somatic variant interpretation do not currently exist. With respect to using published classification schemes to classify variants as “actionable,” there was also no consensus concerning the classification scheme applied, with 3 sites applying the amp/asco/cap guideline6, 1 using the Sukhai et al. guideline7, and 1 not using a guideline. Of the sites that used the amp/asco/cap guideline, 2 included only tier i or ii variants as “clinically reportable”; another laboratory included tier iiv variants. That observation underscores issues related to the understanding of the pt instructions, because tier iii and iv variants are generally not considered clinically actionable or reportable.


Our pilot molecular profiling pt scheme for solid tumours at 5 clinical laboratories in Ontario demonstrates the value of an analyte-based end-to-end pt approach, and also highlights issues related to selection of sample source material, evaluation of ngs quality, and discrepancies in somatic variant interpretation and reporting. Although complete concordance in the technical identification of variants was observed across laboratories and sequencing platforms, significant variability was found in the definition of those variants considered to be “clinically reportable,” compounded by site-specific practices for reporting and variant classification practices. Our pilot study demonstrates a successful pt scheme within the Canadian clinical laboratory context and also demonstrates a need to define the clinically relevant genes and variants to be reported and an appropriate variant classification scheme in solid tumour molecular profiling to reduce cross-institutional inconsistencies.

Supplementary Information


We thank Daria Grafodatskaya (Hamilton Health Sciences and St. Joseph’s Healthcare, Hamilton, ON), Henry Wong (Kingston Health Sciences Centre, Kingston, ON), and Laura Semenuk (Kingston Health Sciences Centre, Kingston, ON) for technical contributions.

The octane study was conducted with the support of the Ontario Institute for Cancer Research through funding provided by the Government of Ontario (grant nos. P.OCT.051 and P.AO.075).


We have read and understood Current Oncology’s policy on disclosing conflicts of interest, and we declare the following interests: HC reports grants from the Ontario Institute for Cancer Research during the conduct of the study. HF reports nonfinancial support from Thermo Fisher outside the submitted work. BL reports personal fees from Novartis, Bayer, Roche, and AstraZeneca outside the submitted work. BS reports fees from the Ontario Institute for Cancer Research during the conduct of the study. LLS reports other consideration from Merck (compensated), Pfizer (compensated), Celgene (compensated), AstraZeneca/Medimmune (compensated), MorphoSys (compensated), Roche (compensated), Geneseeq Technology (compensated), Loxo Oncology (compensated), Oncorus (compensated), Symphogen (compensated), Seattle Genetics (compensated); grants from Novartis, Bristol–Myers Squibb, Pfizer, Boehringer Ingelheim, GlaxoSmithKline, Roche/Genentech, Karyopharm, AstraZeneca/Medimmune, Merck, Celgene, Astellas, Bayer, AbbVie, Amgen, Symphogen, Intensity Therapeutics, Mirati Therapeutics, and Shattuck Labs; and other consideration from Agios Pharmaceuticals (spouse) outside the submitted work. PLB is a member of the steering committee for the American Association for Cancer Research Project genie, past chair of the Canadian Cancer Trials Group Investigational New Drug Committee, a member of the executive board for the Breast International Group, a section editor for The Oncologist and for JNCI Cancer Spectrum outside the submitted work. The remaining authors have no conflicts of interest to disclose.


*Toronto, ON: Advanced Molecular Diagnostics Laboratory, Princess Margaret Cancer Centre, University Health Network (Spence, Stockley); Bioinformatics and HPC Core, Princess Margaret Cancer Centre, University Health Network (Stickle); Cancer Genomics Program, Princess Margaret Cancer Centre, University Health Network (Yu, Chow, Siu); Division of Medical Oncology and Hematology, Princess Margaret Cancer Centre, University Health Network (Siu, Bedard); Department of Medicine, University of Toronto (Siu, Bedard); Department of Clinical Laboratory Genetics, University Health Network (Stockley); Department of Laboratory Medicine and Pathobiology, University of Toronto (Stockley),
Kingston, ON: Molecular Diagnostics, Kingston Health Sciences Centre (Feilotter); Department of Pathology and Molecular Medicine, Queen’s University (Feilotter),
Ottawa, ON: Molecular Oncology Diagnostics Laboratory, The Ottawa Hospital (Lo); Department of Pathology and Laboratory Medicine, University of Ottawa (Lo),
§Hamilton, ON: Hamilton Health Sciences and St. Joseph’s Healthcare (McCready); Department of Pathology and Molecular Medicine, McMaster University (McCready),
||London, ON: Pathology and Laboratory Medicine Program, London Health Sciences Centre (Sadikovic); Department of Pathology and Laboratory Medicine, Western University (Sadikovic).


1 Nagarajan R, Bartley AN, Bridge JA, et al. A window into clinical next-generation sequencing–based oncology testing practices. Arch Pathol Lab Med 2017;141:1679–85.
cross-ref  pubmed  

2 Merker JD, Devereaux K, Iafrate AJ, et al. Proficiency testing of standardized samples shows very high interlaboratory agreement for clinical next-generation sequencing–based oncology assays. Arch Pathol Lab Med 2018;143:463–71.

3 Schrijver I, Aziz N, Jennings LR, Richards CS, Voelkerding KV, Weck KE. Methods-based proficiency testing in molecular genetic pathology. J Mol Diagn 2014;16:283–7.
cross-ref  pubmed  

4 Jennings LJ, Arcila ME, Corless C, et al. Guidelines for validation of next-generation sequencing–based oncology panels: a joint consensus recommendation of the Association for Molecular Pathology and College of American Pathologists. J Mol Diagn 2017;19:341–65.
cross-ref  pubmed  

5 Quinlan AR, Hall IM. BEDTools: a flexible suite of utilities for comparing genomic features. Bioinformatics 2010;26:841–2.
cross-ref  pubmed  pmc  

6 Li MM, Datto M, Duncavage EJ, et al. Standards and guidelines for the interpretation and reporting of sequence variants in cancer: a joint consensus recommendation of the Association for Molecular Pathology, American Society of Clinical Oncology, and College of American Pathologists. J Mol Diagn 2017;19:4–23.
cross-ref  pmc  

7 Sukhai MA, Craddock KJ, Thomas M, et al. A classification system for clinical relevance of somatic variants identified in molecular profiling of cancer. Genet Med 2016;18:128–36.

Correspondence to: Tracy L. Stockley, 11E-418 Eaton Wing, 200 Elizabeth Street, Toronto, Ontario M5G 2C4. E-mail:

(Return to Top)

Supplemental material available at ( Return to Text



TABLE AI Gene list for the amplicon-based Ion AmpliSeq Cancer Hotspot Panel v2a


TABLE AII Gene list for the SureSelecta custom hybridization capture panel of 555 cancer-related genes (“UHN Hi5 Panel”)


Current Oncology, VOLUME 26, NUMBER 6, December 2019

Copyright © 2019 Multimed Inc.
ISSN: 1198-0052 (Print) ISSN: 1718-7729 (Online)