User login
Comparison of Precision Oncology Annotation Services in the National Precision Oncology Program
BACKGROUND
The National Precision Oncology Program (NPOP) provides comprehensive genomic profiling (CGP) through external vendors to patients within the Veterans Affairs Healthcare System who meet testing guidelines. We sought to assess the concordance of cancer therapy recommendations between Foundation Medicine (FM), one of the NPOP vendors, and OncoKB, an FDA-recognized public precision oncology knowledge database, which annotates human genetic variants associated with therapies guidance at varying levels of evidence.
METHODS
We selected FM CGP test reports with at least one therapy recommendation regardless of FDA approval or level of evidence were selected to compare FM and OncoKB therapy annotations of different mutation types, including short variants (SVs), rearrangements, and copy number alterations (CNAs) between 02/01/2019-03/13/2023. Therapy recommendations of annotations for unique combinations of gene, variant, and cancer type from FM and OncoKB were compared. Comparisons were scored as an Exact Match (EM) if FM and OncoKB therapy annotation was the same or a Partial Match (PM) if the FM therapy annotation was a subset of OncoKB’s or vice versa.
RESULTS
For annotations involving FDA-approved therapies, a total of 10,435 cases were compared for SVs, 546 for rearrangements, and 732 for CNAs. Among SVs annotations, 7,029 (67.4%) were EM and 787 (7.5%) were PM. Of rearrangement annotations, 328 (60.1%) were EM and 95 (17.4%) were PM. Of CNA annotations, 469 (64.1%) were EM and 28 (3.8%) were PM. For off-label therapies, agreement between annotation sources was much lower in all above scenarios. Examples included 3022 (29%) cases were identified as EM plus PM for SVs, 324 (59.3%) for rearrangements, and 42 (5.7%) for CNAs.
CONCLUSIONS
Therapy recommendations were inconsistent between FM and OncoKB annotation services, with a substantial disagreement among both FDA-approved and off-label therapy annotations. The limitation of time difference of annotations performed between FM and OncoKB therapy annotations accounted for some disagreement. Establishing accuracy and improving concordance between different annotation services is needed to better match treatments to patients and improve provider trust and reliability of annotation service.
BACKGROUND
The National Precision Oncology Program (NPOP) provides comprehensive genomic profiling (CGP) through external vendors to patients within the Veterans Affairs Healthcare System who meet testing guidelines. We sought to assess the concordance of cancer therapy recommendations between Foundation Medicine (FM), one of the NPOP vendors, and OncoKB, an FDA-recognized public precision oncology knowledge database, which annotates human genetic variants associated with therapies guidance at varying levels of evidence.
METHODS
We selected FM CGP test reports with at least one therapy recommendation regardless of FDA approval or level of evidence were selected to compare FM and OncoKB therapy annotations of different mutation types, including short variants (SVs), rearrangements, and copy number alterations (CNAs) between 02/01/2019-03/13/2023. Therapy recommendations of annotations for unique combinations of gene, variant, and cancer type from FM and OncoKB were compared. Comparisons were scored as an Exact Match (EM) if FM and OncoKB therapy annotation was the same or a Partial Match (PM) if the FM therapy annotation was a subset of OncoKB’s or vice versa.
RESULTS
For annotations involving FDA-approved therapies, a total of 10,435 cases were compared for SVs, 546 for rearrangements, and 732 for CNAs. Among SVs annotations, 7,029 (67.4%) were EM and 787 (7.5%) were PM. Of rearrangement annotations, 328 (60.1%) were EM and 95 (17.4%) were PM. Of CNA annotations, 469 (64.1%) were EM and 28 (3.8%) were PM. For off-label therapies, agreement between annotation sources was much lower in all above scenarios. Examples included 3022 (29%) cases were identified as EM plus PM for SVs, 324 (59.3%) for rearrangements, and 42 (5.7%) for CNAs.
CONCLUSIONS
Therapy recommendations were inconsistent between FM and OncoKB annotation services, with a substantial disagreement among both FDA-approved and off-label therapy annotations. The limitation of time difference of annotations performed between FM and OncoKB therapy annotations accounted for some disagreement. Establishing accuracy and improving concordance between different annotation services is needed to better match treatments to patients and improve provider trust and reliability of annotation service.
BACKGROUND
The National Precision Oncology Program (NPOP) provides comprehensive genomic profiling (CGP) through external vendors to patients within the Veterans Affairs Healthcare System who meet testing guidelines. We sought to assess the concordance of cancer therapy recommendations between Foundation Medicine (FM), one of the NPOP vendors, and OncoKB, an FDA-recognized public precision oncology knowledge database, which annotates human genetic variants associated with therapies guidance at varying levels of evidence.
METHODS
We selected FM CGP test reports with at least one therapy recommendation regardless of FDA approval or level of evidence were selected to compare FM and OncoKB therapy annotations of different mutation types, including short variants (SVs), rearrangements, and copy number alterations (CNAs) between 02/01/2019-03/13/2023. Therapy recommendations of annotations for unique combinations of gene, variant, and cancer type from FM and OncoKB were compared. Comparisons were scored as an Exact Match (EM) if FM and OncoKB therapy annotation was the same or a Partial Match (PM) if the FM therapy annotation was a subset of OncoKB’s or vice versa.
RESULTS
For annotations involving FDA-approved therapies, a total of 10,435 cases were compared for SVs, 546 for rearrangements, and 732 for CNAs. Among SVs annotations, 7,029 (67.4%) were EM and 787 (7.5%) were PM. Of rearrangement annotations, 328 (60.1%) were EM and 95 (17.4%) were PM. Of CNA annotations, 469 (64.1%) were EM and 28 (3.8%) were PM. For off-label therapies, agreement between annotation sources was much lower in all above scenarios. Examples included 3022 (29%) cases were identified as EM plus PM for SVs, 324 (59.3%) for rearrangements, and 42 (5.7%) for CNAs.
CONCLUSIONS
Therapy recommendations were inconsistent between FM and OncoKB annotation services, with a substantial disagreement among both FDA-approved and off-label therapy annotations. The limitation of time difference of annotations performed between FM and OncoKB therapy annotations accounted for some disagreement. Establishing accuracy and improving concordance between different annotation services is needed to better match treatments to patients and improve provider trust and reliability of annotation service.
Development of a National Precision Oncology Program (NPOP) Dashboard Suite and Data Mart For Monitoring Somatic Molecular Testing Use
BACKGROUND
As of May 2023, the Veterans Affairs (VA) National Precision Oncology Program (NPOP) has provided somatic molecular testing for nearly 36,000 Veterans with cancer. Automated tools to monitor test use (locally and nationally) have only been available for NGS testing in advanced stage lung and prostate cancer. To better track utilization of NPOP supported testing across all cancer indications, and to support strategies to promote wider adoption, we developed an automated data architecture to monitor program operations. Here, we describe the development of the NPOP data mart and summarize the core components of the NPOP Somatic Molecular Testing dashboards.
METHODS
SQL Server Integration Services was used to build the backend data mart, which required the ingestion of vendor-specific XML data and subsequent harmonization with data found in the VA’s Corporate Data Warehouse (CDW). The NPOP somatic testing dashboards, developed using Power BI, are securely hosted in the cloud, and accessible through SharePoint.
DATA ANALYSIS
The NPOP dashboard suite displays key measures using descriptive statistics, including counts, proportions, means, medians, and standard deviations. To support the visualization of comparisons we leveraged stacked and clustered bar charts, and violin plots.
RESULTS
The NPOP data mart refreshes nightly providing users with near real-time data. The NPOP somatic testing dashboards include an all vendor combined report and sub-reports organized by vendors: Foundation Medicine, Personalis, and Personal Genome Diagnostics and Tempus. All reports contain four views with the ability to toggle between tests ordered or completed. For current NPOP vendors, patient level data and turnaround time views were developed. Data are stratified by test category (i.e., NGS Solid, NGS Liquid, Heme, IHC) and can be viewed longitudinally (i.e., across time) and filtered by test date, VA facility, patient demographics, and cancer characteristics (diagnosis, stage). To date, over 50,000 tests have been ordered (90% through Foundation Medicine).
IMPLICATIONS
The NPOP data mart and operational dashboards synthesizes vast amounts of data into a visually consumable format that supports monitoring the uptake and variation of somatic molecular testing services being offered across the VA.
BACKGROUND
As of May 2023, the Veterans Affairs (VA) National Precision Oncology Program (NPOP) has provided somatic molecular testing for nearly 36,000 Veterans with cancer. Automated tools to monitor test use (locally and nationally) have only been available for NGS testing in advanced stage lung and prostate cancer. To better track utilization of NPOP supported testing across all cancer indications, and to support strategies to promote wider adoption, we developed an automated data architecture to monitor program operations. Here, we describe the development of the NPOP data mart and summarize the core components of the NPOP Somatic Molecular Testing dashboards.
METHODS
SQL Server Integration Services was used to build the backend data mart, which required the ingestion of vendor-specific XML data and subsequent harmonization with data found in the VA’s Corporate Data Warehouse (CDW). The NPOP somatic testing dashboards, developed using Power BI, are securely hosted in the cloud, and accessible through SharePoint.
DATA ANALYSIS
The NPOP dashboard suite displays key measures using descriptive statistics, including counts, proportions, means, medians, and standard deviations. To support the visualization of comparisons we leveraged stacked and clustered bar charts, and violin plots.
RESULTS
The NPOP data mart refreshes nightly providing users with near real-time data. The NPOP somatic testing dashboards include an all vendor combined report and sub-reports organized by vendors: Foundation Medicine, Personalis, and Personal Genome Diagnostics and Tempus. All reports contain four views with the ability to toggle between tests ordered or completed. For current NPOP vendors, patient level data and turnaround time views were developed. Data are stratified by test category (i.e., NGS Solid, NGS Liquid, Heme, IHC) and can be viewed longitudinally (i.e., across time) and filtered by test date, VA facility, patient demographics, and cancer characteristics (diagnosis, stage). To date, over 50,000 tests have been ordered (90% through Foundation Medicine).
IMPLICATIONS
The NPOP data mart and operational dashboards synthesizes vast amounts of data into a visually consumable format that supports monitoring the uptake and variation of somatic molecular testing services being offered across the VA.
BACKGROUND
As of May 2023, the Veterans Affairs (VA) National Precision Oncology Program (NPOP) has provided somatic molecular testing for nearly 36,000 Veterans with cancer. Automated tools to monitor test use (locally and nationally) have only been available for NGS testing in advanced stage lung and prostate cancer. To better track utilization of NPOP supported testing across all cancer indications, and to support strategies to promote wider adoption, we developed an automated data architecture to monitor program operations. Here, we describe the development of the NPOP data mart and summarize the core components of the NPOP Somatic Molecular Testing dashboards.
METHODS
SQL Server Integration Services was used to build the backend data mart, which required the ingestion of vendor-specific XML data and subsequent harmonization with data found in the VA’s Corporate Data Warehouse (CDW). The NPOP somatic testing dashboards, developed using Power BI, are securely hosted in the cloud, and accessible through SharePoint.
DATA ANALYSIS
The NPOP dashboard suite displays key measures using descriptive statistics, including counts, proportions, means, medians, and standard deviations. To support the visualization of comparisons we leveraged stacked and clustered bar charts, and violin plots.
RESULTS
The NPOP data mart refreshes nightly providing users with near real-time data. The NPOP somatic testing dashboards include an all vendor combined report and sub-reports organized by vendors: Foundation Medicine, Personalis, and Personal Genome Diagnostics and Tempus. All reports contain four views with the ability to toggle between tests ordered or completed. For current NPOP vendors, patient level data and turnaround time views were developed. Data are stratified by test category (i.e., NGS Solid, NGS Liquid, Heme, IHC) and can be viewed longitudinally (i.e., across time) and filtered by test date, VA facility, patient demographics, and cancer characteristics (diagnosis, stage). To date, over 50,000 tests have been ordered (90% through Foundation Medicine).
IMPLICATIONS
The NPOP data mart and operational dashboards synthesizes vast amounts of data into a visually consumable format that supports monitoring the uptake and variation of somatic molecular testing services being offered across the VA.
Development of an Informatics Infrastructure and Frontend Dashboard for Monitoring Clinical Operations of the National TeleOncology Service
Background
Since inception, the Veterans Affairs (VA) National TeleOncology (NTO) service has monitored clinical operations through data tools produced by the Veterans Health Administration Support Service Center (VSSC). Unfortunately, pertinent data are spread across multiple reports, making it difficult to continually harmonize needed information. Further, the VSSC does not account for NTO’s hub and spoke clinical model, leading to inaccuracies when attempting to analyze unique encounters. To address these challenges, NTO partnered with the VA Salt Lake City Health Care System Informatics, Decision-Enhancement, and Analytic Sciences Center (IDEAS) to develop an informatics architecture and frontend NTO Clinical Operations Dashboard (NCOD). Here, we summarize our dashboard development process and the finalized key reporting components of the NCOD.
Methods
The VA Corporate Data Warehouse (CDW) serves as the primary data source for the NCOD. SQL Server Integration Services was used to build the backend data architecture. Data from the CDW were isolated into a staging data mart for reporting purposes using an extract, transform, load (ETL) approach. The frontend user interface was developed using Power BI. We used a participatory approach1 in determining reporting requirements. Stakeholders included the IDEAS dashboard development team and potential end users from NTO, including leadership, program managers, support assistants, and telehealth coordinators.
Results
The NCOD ETL is scheduled to refresh the data nightly to provide end users with a near real-time experience. The NCOD is comprised of the following four data views: clinic availability, team productivity, patient summary, and encounter summary. The clinic availability view summarizes clinic capacity, no shows, overbookings, and percent utilization. Relative value unit- based productivity is summarized in the team productivity view. The patient summary view presents aggregated data for veterans served by NTO, including geographic distribution, with patient-level drill down displaying demographics, cancer characteristics, and treatment history. Lastly, the encounter view displays utilization trends by modality, while accurately accounting for the hub and spoke clinical model.
Conclusions
An informatics architecture and frontend information display that is capable of synthesizing EHR data into a consumable format has been pivotal in obtaining accurate and timely insight into the demand and capacity of services provided by NTO.
- Esquer Rochin MA, Gutierrez-Garcia JO, Rosales JH, Rodriguez LF. Design and evaluation of a dashboard to support the comprehension of the progression.
Background
Since inception, the Veterans Affairs (VA) National TeleOncology (NTO) service has monitored clinical operations through data tools produced by the Veterans Health Administration Support Service Center (VSSC). Unfortunately, pertinent data are spread across multiple reports, making it difficult to continually harmonize needed information. Further, the VSSC does not account for NTO’s hub and spoke clinical model, leading to inaccuracies when attempting to analyze unique encounters. To address these challenges, NTO partnered with the VA Salt Lake City Health Care System Informatics, Decision-Enhancement, and Analytic Sciences Center (IDEAS) to develop an informatics architecture and frontend NTO Clinical Operations Dashboard (NCOD). Here, we summarize our dashboard development process and the finalized key reporting components of the NCOD.
Methods
The VA Corporate Data Warehouse (CDW) serves as the primary data source for the NCOD. SQL Server Integration Services was used to build the backend data architecture. Data from the CDW were isolated into a staging data mart for reporting purposes using an extract, transform, load (ETL) approach. The frontend user interface was developed using Power BI. We used a participatory approach1 in determining reporting requirements. Stakeholders included the IDEAS dashboard development team and potential end users from NTO, including leadership, program managers, support assistants, and telehealth coordinators.
Results
The NCOD ETL is scheduled to refresh the data nightly to provide end users with a near real-time experience. The NCOD is comprised of the following four data views: clinic availability, team productivity, patient summary, and encounter summary. The clinic availability view summarizes clinic capacity, no shows, overbookings, and percent utilization. Relative value unit- based productivity is summarized in the team productivity view. The patient summary view presents aggregated data for veterans served by NTO, including geographic distribution, with patient-level drill down displaying demographics, cancer characteristics, and treatment history. Lastly, the encounter view displays utilization trends by modality, while accurately accounting for the hub and spoke clinical model.
Conclusions
An informatics architecture and frontend information display that is capable of synthesizing EHR data into a consumable format has been pivotal in obtaining accurate and timely insight into the demand and capacity of services provided by NTO.
Background
Since inception, the Veterans Affairs (VA) National TeleOncology (NTO) service has monitored clinical operations through data tools produced by the Veterans Health Administration Support Service Center (VSSC). Unfortunately, pertinent data are spread across multiple reports, making it difficult to continually harmonize needed information. Further, the VSSC does not account for NTO’s hub and spoke clinical model, leading to inaccuracies when attempting to analyze unique encounters. To address these challenges, NTO partnered with the VA Salt Lake City Health Care System Informatics, Decision-Enhancement, and Analytic Sciences Center (IDEAS) to develop an informatics architecture and frontend NTO Clinical Operations Dashboard (NCOD). Here, we summarize our dashboard development process and the finalized key reporting components of the NCOD.
Methods
The VA Corporate Data Warehouse (CDW) serves as the primary data source for the NCOD. SQL Server Integration Services was used to build the backend data architecture. Data from the CDW were isolated into a staging data mart for reporting purposes using an extract, transform, load (ETL) approach. The frontend user interface was developed using Power BI. We used a participatory approach1 in determining reporting requirements. Stakeholders included the IDEAS dashboard development team and potential end users from NTO, including leadership, program managers, support assistants, and telehealth coordinators.
Results
The NCOD ETL is scheduled to refresh the data nightly to provide end users with a near real-time experience. The NCOD is comprised of the following four data views: clinic availability, team productivity, patient summary, and encounter summary. The clinic availability view summarizes clinic capacity, no shows, overbookings, and percent utilization. Relative value unit- based productivity is summarized in the team productivity view. The patient summary view presents aggregated data for veterans served by NTO, including geographic distribution, with patient-level drill down displaying demographics, cancer characteristics, and treatment history. Lastly, the encounter view displays utilization trends by modality, while accurately accounting for the hub and spoke clinical model.
Conclusions
An informatics architecture and frontend information display that is capable of synthesizing EHR data into a consumable format has been pivotal in obtaining accurate and timely insight into the demand and capacity of services provided by NTO.
- Esquer Rochin MA, Gutierrez-Garcia JO, Rosales JH, Rodriguez LF. Design and evaluation of a dashboard to support the comprehension of the progression.
- Esquer Rochin MA, Gutierrez-Garcia JO, Rosales JH, Rodriguez LF. Design and evaluation of a dashboard to support the comprehension of the progression.