WO2016118527A1 - Systems and methods for response prediction to chemotherapy in high grade bladder cancer - Google Patents
Systems and methods for response prediction to chemotherapy in high grade bladder cancer Download PDFInfo
- Publication number
- WO2016118527A1 WO2016118527A1 PCT/US2016/013959 US2016013959W WO2016118527A1 WO 2016118527 A1 WO2016118527 A1 WO 2016118527A1 US 2016013959 W US2016013959 W US 2016013959W WO 2016118527 A1 WO2016118527 A1 WO 2016118527A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- weka
- linear
- treatment
- trees
- svmlight
- Prior art date
Links
Classifications
-
- C—CHEMISTRY; METALLURGY
- C12—BIOCHEMISTRY; BEER; SPIRITS; WINE; VINEGAR; MICROBIOLOGY; ENZYMOLOGY; MUTATION OR GENETIC ENGINEERING
- C12Q—MEASURING OR TESTING PROCESSES INVOLVING ENZYMES, NUCLEIC ACIDS OR MICROORGANISMS; COMPOSITIONS OR TEST PAPERS THEREFOR; PROCESSES OF PREPARING SUCH COMPOSITIONS; CONDITION-RESPONSIVE CONTROL IN MICROBIOLOGICAL OR ENZYMOLOGICAL PROCESSES
- C12Q1/00—Measuring or testing processes involving enzymes, nucleic acids or microorganisms; Compositions therefor; Processes of preparing such compositions
- C12Q1/68—Measuring or testing processes involving enzymes, nucleic acids or microorganisms; Compositions therefor; Processes of preparing such compositions involving nucleic acids
- C12Q1/6876—Nucleic acid products used in the analysis of nucleic acids, e.g. primers or probes
- C12Q1/6883—Nucleic acid products used in the analysis of nucleic acids, e.g. primers or probes for diseases caused by alterations of genetic material
- C12Q1/6886—Nucleic acid products used in the analysis of nucleic acids, e.g. primers or probes for diseases caused by alterations of genetic material for cancer
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16B—BIOINFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR GENETIC OR PROTEIN-RELATED DATA PROCESSING IN COMPUTATIONAL MOLECULAR BIOLOGY
- G16B20/00—ICT specially adapted for functional genomics or proteomics, e.g. genotype-phenotype associations
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16B—BIOINFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR GENETIC OR PROTEIN-RELATED DATA PROCESSING IN COMPUTATIONAL MOLECULAR BIOLOGY
- G16B20/00—ICT specially adapted for functional genomics or proteomics, e.g. genotype-phenotype associations
- G16B20/20—Allele or variant detection, e.g. single nucleotide polymorphism [SNP] detection
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16B—BIOINFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR GENETIC OR PROTEIN-RELATED DATA PROCESSING IN COMPUTATIONAL MOLECULAR BIOLOGY
- G16B30/00—ICT specially adapted for sequence analysis involving nucleotides or amino acids
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16B—BIOINFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR GENETIC OR PROTEIN-RELATED DATA PROCESSING IN COMPUTATIONAL MOLECULAR BIOLOGY
- G16B40/00—ICT specially adapted for biostatistics; ICT specially adapted for bioinformatics-related machine learning or data mining, e.g. knowledge discovery or pattern finding
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16B—BIOINFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR GENETIC OR PROTEIN-RELATED DATA PROCESSING IN COMPUTATIONAL MOLECULAR BIOLOGY
- G16B40/00—ICT specially adapted for biostatistics; ICT specially adapted for bioinformatics-related machine learning or data mining, e.g. knowledge discovery or pattern finding
- G16B40/20—Supervised data analysis
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
- G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
- G16H50/50—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for simulation or modelling of medical disorders
-
- C—CHEMISTRY; METALLURGY
- C12—BIOCHEMISTRY; BEER; SPIRITS; WINE; VINEGAR; MICROBIOLOGY; ENZYMOLOGY; MUTATION OR GENETIC ENGINEERING
- C12Q—MEASURING OR TESTING PROCESSES INVOLVING ENZYMES, NUCLEIC ACIDS OR MICROORGANISMS; COMPOSITIONS OR TEST PAPERS THEREFOR; PROCESSES OF PREPARING SUCH COMPOSITIONS; CONDITION-RESPONSIVE CONTROL IN MICROBIOLOGICAL OR ENZYMOLOGICAL PROCESSES
- C12Q2600/00—Oligonucleotides characterized by their use
- C12Q2600/106—Pharmacogenomics, i.e. genetic variability in individual responses to drugs and drug metabolism
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
- G06N20/20—Ensemble learning
Definitions
- the field of the invention is in silico systems and methods for prediction of treatment outcome for chemotherapy in bladder cancer.
- WO 2014/193982 describes systems and methods in which pathway elements (corresponding to cellular in vivo features) of a pathway model are modified in silico to simulate treatment of a cell with a drug.
- the modified model can then be used to help predict the effect of the drug on one or more pathways, and indirectly predict the effect of the drug on a diseased tissue. While such system has provided remarkable predictive power in certain circumstances, such system was based on cell culture data and as such did not fully reflect in vivo environments.
- simulation of the treatment was performed using a single model that was rooted in measured and assumed attributes and therefore relied on specific assumptions genuine to the model. The described approach fails to provide insight into mitigating risks associated with the specific assumptions of model.
- the so formed prediction model is then used to rank possible responses to treatment. Wei simply builds prediction outcome models to make an assessment of likely outcome based patient-specific profile information. Unfortunately, not all algorithms will be suitable for predictive analysis of drug treatment as each algorithm has built in assumptions that might not be valid for the specific disease and/or drug treatment.
- models that are maximized for a particular prediction will not necessarily provide the best accuracy as compared to a random event and/or other model.
- a predictive model for treatment outcome for high- grade bladder cancer can be derived from a collection of models that were prepared using various machine learning algorithms trained on previously known high-grade bladder cancer omics information that was associated with treatment outcome. Most preferably, prediction accuracy is improved by identification of a model with high accuracy gain and selection of omics parameters and associated weighting from the identified model.
- contemplated methods include a step of obtaining a plurality of omics data from the patient, and a further step of (a) using an accuracy gain metric to select at least a single model for prediction of the treatment outcome of high grade bladder cancer treatment or (b) selecting at least a single model on the basis of a previously determined accuracy gain metric for prediction of the treatment outcome of high grade bladder cancer treatment.
- Models may be selected from among a large number, for example, from among at least 10 trained models or from among at least 100 trained models or even more.
- an analysis engine calculates a prediction outcome (e.g., complete response to treatment, partial response to treatment, stable non-response to treatment, and progressive non-response to treatment) using the single model and the plurality of omics data from the patient.
- a prediction outcome e.g., complete response to treatment, partial response to treatment, stable non-response to treatment, and progressive non-response to treatment
- the omics data include whole genome differential objects, exome differential objects, SNP data, copy number data, RNA transcription data, protein expression data, and/or protein activity data
- the accuracy gain metric may be an accuracy gain, an accuracy gain distribution, an area under curve metric, an R 2 metric, a p- value metric, a silhouette coefficient, and/or a confusion matrix. While not limiting the inventive subject matter, it is also contemplated that the accuracy gain metric of the single model is within the upper quartile of all models, or within the top 5% of all models, or wherein the accuracy gain metric of the single model exceeds all other models.
- the single model may be generated using a machine learning algorithm that uses a classifier selected form the group consisting of NMFpredictor (linear), SVMlight (linear), SVMlight first order polynomial kernel (degree-d polynomial), SVMlight second order polynomial kernel (degree-d polynomial), WEKA SMO (linear), WEKA j48 trees (trees -based), WEKA hyper pipes (distribution-based), WEKA random forests (trees -based), WEKA naive Bayes (probabilistic/bayes), WEKA JRip (rules-based), glmnet lasso (sparse linear), glmnet ridge regression (sparse linear), and glmnet elastic nets (sparse linear).
- a classifier selected form the group consisting of NMFpredictor (linear), SVMlight (linear), SVMlight first order polynomial kernel (degree-d polynomial), SVMlight
- the step of calculating comprises a step of selecting features of the single model having minimum absolute predetermined weights (e.g., within the top quartile of all weights in the single model). While numerous features may be suitable, it is contemplated that the step of calculating uses at least 10 distinct selected features in the single model.
- the features of the single model are RNA transcription values for genes selected from the group consisting of PCDHGA4, PCDHGB 1, HSP90AB2P, SPAG9, DDI2, TOP1P2, AGAP1 , BBS9, FNIP2, LOC647121, NFIC, TGFBRAP1 , EPRS, C9orfl29, SARS, RBM28, NACC2, GTPBP5, PRKAR2A, CDK8, FAM24B, CRK, RAB2A, SMAD2, ELP2, WWP1, KIF5B, RPL39, PSEN1 , SURF4, TTC35, TOM1, TES, VWA1, GOLGA2, ARHGAP21, FLJ37201,
- RNA transcription values for the genes are calculated with respective factors, that the respective factors are weighted, and that (using absolute values), the weights are in the order of PCDHGA4, PCDHGB 1, HSP90AB2P, SPAG9, DDI2, TOP1P2, AGAP1 , BBS9, FNIP2, LOC647121, NFIC, TGFBRAP1 , EPRS, C9orfl29, SARS, RBM28, NACC2, GTPBP5, PRKAR2A, CDK8, FAM24B, CRK, RAB2A, SMAD2, ELP2, WWP1, KIF5B, RPL39, PSEN1 , SURF4, TTC35, TOM1, TES, VWA1, GOLGA2, ARHGAP21, FLJ37201,
- the inventors therefore also contemplate a method of predicting treatment outcome for a patient having high-grade bladder cancer.
- Such methods will preferably include a step of obtaining plurality of RNA transcription data of the patient, and a further step of calculating, by an analysis engine and using the plurality of RNA transcription data of the patient, a treatment outcome score using a model.
- the model uses RNA transcription values for genes selected from the group consisting of PCDHGA4, PCDHGB1, HSP90AB2P, SPAG9, DDI2, TOP1P2, AGAP1, BBS9, FNIP2, LOC647121, NFIC, TGFBRAPl, EPRS, C9orfl29, SARS, RBM28, NACC2, GTPBP5, PRKAR2A, CDK8, FAM24B, CRK, RAB2A, SMAD2, ELP2, WWP1, KIF5B, RPL39, PSEN1, SURF4, TTC35, TOM1, TES, VWA1, GOLGA2, ARHGAP21, FLJ37201, KIAA1429, AZIN1, SCAMP2, H1F0, PYCR1, SEC24D, FLNB, PATL1, HDLBP, RRBP1, OXR1, GLB 1, NPEPPS, KIF1C, DDB 1, and GSN.
- genes selected from the group consisting of PCDHGA4, PCDHGB1, H
- the plurality of RNA transcription data are obtained from polyA RNA, and/or the treatment outcome score is indicative of a complete response to treatment, a partial response to treatment, a stable non-response to treatment, or a progressive non- response to treatment.
- the model was generated using a machine learning algorithm that uses a classifier selected form the group consisting of NMFpredictor (linear), SVMlight (linear), SVMlight first order polynomial kernel (degree-d polynomial), SVMlight second order polynomial kernel (degree-d polynomial), WEKA SMO (linear), WEKA j48 trees (trees-based), WEKA hyper pipes (distribution-based), WEKA random forests (trees-based), WEKA naive Bayes
- RNA transcription values for the genes are calculated with respective factors, and wherein the respective factors are weighted, using absolute values, in the order of PCDHGA4,
- RNA transcription values are values for at least two genes selected from the group consisting of PCDHGA4, PCDHGB 1, HSP90AB2P, SPAG9, DDI2, TOP1P2, AGAP1, BBS9, FNIP2, LOC647121, NFIC, TGFBRAPl, EPRS, C9orfl29, SARS, RBM28, NACC2, GTPBP5, PRKAR2A, CDK8, FAM24B, CRK, RAB2A, SMAD2, ELP2, WWP1, KIF5B, RPL39, PSEN1, SURF4, TTC35, TOM1, TES, VWA1, GOLGA2, ARHGAP21, FLJ37201,
- RNA transcription values in a model generated by a machine learning algorithm to so predict treatment outcome for the patient.
- the machine learning algorithm uses a classifier selected form the group consisting of
- NMFpredictor linear
- SVMlight linear
- SVMlight first order polynomial kernel degree-d polynomial
- SVMlight second order polynomial kernel degree-d polynomial
- WEKA SMO linear
- WEKA j48 trees trees-based
- WEKA hyper pipes distributed-based
- WEKA random forests trees-based
- WEKA naive Bayes probabilistic/bayes
- WEKA JRip (rules- based), glmnet lasso (sparse linear), glmnet ridge regression (sparse linear), and glmnet elastic nets (sparse linear).
- RNA transcription values for the genes are calculated with respective factors, and that the respective factors are weighted, using absolute values, in the order of PCDHGA4, PCDHGB1, HSP90AB2P, SPAG9, DDI2, TOP1P2, AGAPl, BBS9, FNIP2, LOC647121, NFIC, TGFBRAPl, EPRS, C9orfl29, SARS, RBM28, NACC2, GTPBP5, PRKAR2A, CDK8, FAM24B, CRK, RAB2A, SMAD2, ELP2, WWP1, KIF5B, RPL39, PSEN1, SURF4, TTC35, TOM1, TES, VWA1, GOLGA2,
- ARHGAP21 ARHGAP21, FLJ37201, KIAA1429, AZIN1, SCAMP2, H1F0, PYCR1, SEC24D, FLNB, PATL1, HDLBP, RRBP1, OXR1, GLB1, NPEPPS, KIF1C, DDB 1, and GSN.
- RNA transcription values for prediction of the treatment outcome of high grade bladder cancer treatment, wherein the prediction uses a single model obtained from a machine learning algorithm, and wherein the RNA transcription values are for genes selected from the group consisting of PCDHGA4,
- the RNA transcription values for the genes are calculated with respective factors, and wherein the respective factors are weighted, using absolute values, in the order of PCDHGA4, PCDHGB 1 , HSP90AB2P, SPAG9, DDI2, TOP1P2, AGAP1, BBS9, FNIP2, LOC647121, NFIC, TGFBRAP1 , EPRS, C9orfl29, SARS, RBM28, NACC2, GTPBP5, PRKAR2A, CDK8, FAM24B, CRK, RAB2A, SMAD2, ELP2, WWP1, KIF5B, RPL39, PSEN1 , SURF4, TTC35, TOM1, TES, VWA1 , GOLGA2, ARHGAP21, FLJ37201, KIAA1429, AZIN1 , SCAMP2, H1F0, PYCR1,
- Figure 1 is an exemplary table of features and feature weights derived from a model with high accuracy gain using TCGA high-grade bladder cancer data.
- Figure 2 is an exemplary heat map of RNA transcription values from TCGA high- grade bladder cancer data for responders to drug treatment and non-responders.
- the inventive subject matter is directed to various computer systems and methods in which genomic information for a relatively large class of patients suffering from a particular neoplastic disease (e.g., bladder cancer) is subjected to a relatively large number of machine learning algorithms to so identify a corresponding large number of predictive models.
- the predictive models are then analyzed for accuracy gain, and the model(s) with the highest accuracy gain will then be used to identify relevant omics factors for the prediction.
- contemplated systems and methods are neither based on prediction optimization of a singular model nor based on identification of best correlations of selected omics parameters with a treatment prediction. Instead, it should be recognized that contemplated systems and methods rely on the identification of omics parameters and associated weighting factors that are derived from one or more implementations of machine learning algorithms that result in trained models having a predetermined or minimum accuracy gain. Notably, the so identified omics parameters will typically have no statistically predictive power by themselves and as such would not be used in any omics based test system.
- omics parameters are used in the context of a trained model that has high accuracy gain
- multiple omics parameters will provide a system with high predictive power, particularly when applied in the system using weighting factors associated with the trained model.
- model and omics parameters and weightings are unique to the particular training sets and/or type of outcome prediction, and that other diseases (e.g., lung cancer) and/or outcome predictions (e.g., survival time past 5 years) may lead to entirely different models, omics parameters, and weightings.
- the inventor is considered to have discovered weightings and/or trained models that have high predictive power associated with high-grade bladder cancer.
- treatment prediction can be validated from the a priori identified pathway(s) and/or pathway element(s), or identified pathways and/or pathway elements by in silico modulation using known pathway modeling system and methods to so help confirm treatment strategy predicted by the system.
- RNA transcription values and associated meta data are subject to training and validation splitting in a preprocessing step that then provides the data to different machine-learning packages for analysis.
- the focus of the disclosed inventive subject matter is to enable construction or configuration of a computing device(s) to operate on vast quantities of digital data, beyond the capabilities of a human.
- the digital data can represent machine- trained computer models of omics data and treatment outcomes
- the digital data is a representation of one or more digital models of such real-world items, not the actual items.
- the computing devices are able to manage the digital data or models in a manner that would be beyond the capability of a human.
- the computing devices lack a priori capabilities without such configuration.
- the present inventive subject matter significantly improves/alleviates problems inherent to computational analysis of complex omics calculations.
- any language directed to a computer should be read to include any suitable combination of computing devices, including servers, interfaces, systems, databases, agents, peers, engines, controllers, modules, or other types of computing devices operating individually or collectively.
- the computing devices comprise a processor configured to execute software instructions stored on a tangible, non- transitory computer readable storage medium (e.g., hard drive, FPGA, PLA, solid state drive, RAM, flash, ROM, etc.).
- the software instructions configure or otherwise program the computing device to provide the roles, responsibilities, or other functionality as discussed below with respect to the disclosed apparatus.
- the disclosed technologies can be embodied as a computer program product that includes a non-transitory computer readable medium storing the software instructions that causes a processor to execute the disclosed steps associated with implementations of computer-based algorithms, processes, methods, or other instructions.
- the various servers, systems, databases, or interfaces exchange data using standardized protocols or algorithms, possibly based on HTTP, HTTPS, AES, public-private key exchanges, web service APIs, known financial transaction protocols, or other electronic information exchanging methods.
- Data exchanges among devices can be conducted over a packet- switched network, the Internet, LAN, WAN, VPN, or other type of packet switched network, circuit switched network, and/or cell switched network.
- chemotherapy responder If a patient had a chemotherapy agent with Complete or Partial Response recorded, they were considered a "chemotherapy responder”. If they had Clinical Progressive or Stable disease, they were considered a "chemotherapy non-responder”. A total of 33 patients had a chemotherapy response recorded (15 non-responders and 18 responders). All 33 patients were confirmed to be high-grade bladder cancer patients using further TCGA clinical information.
- accuracy gain was calculated by comparison of the correct prediction percentage using the validation data set against the percentage (frequency) of occurrence of the majority classifier (here: treatment is responsive). For example, where responsive treatment frequency is 60% in the known data set and where the model correctly predicts 85% of the treatment outcome as responsive, the accuracy gain is 25%. Notably, over all models constructed, the best model building process was 88% accurate in cross- validation testing folds (which was 33% better than majority) and used an elastic net classifier. The final fully-trained model that used the most accurate build process was selected from the 72 candidate models.
- each type of model includes inherent biases or assumptions, which may influence how a resulting trained model would operate relative to other types of trained models, even when trained on identical data. Accordingly, different models will produce different predictions/accuracy gains when using the same training data set.
- single machine learning algorithms were optimized to increase correct prediction on the same data set.
- accuracy i.e., accurate prediction capability against 'coin flip'
- Such bias can be overcome by training numerous diverse models with different underlying principles and classifiers on disease-specific data sets with associated metadata and by selecting from the so trained models those with desirable accuracy gain or robustness.
- Figure 1 exemplarily depicts a collection of genes encoding an RNA where the omics data from a patient are RNA transcription data (transcription strength).
- the predictive model was built as described above from the a priori known TGCA data using RNA transcription levels from the gene expression panel. The best predictive model had 88% accuracy in cross-validation testing folds and the top 53 genes with highest weighting factor are shown.
- the PCDHGA4 gene (Protocadherin Gamma Subfamily A, 4) had a weighting factor of -121543.6202 with respect to the RNA expression, with further genes and weighting factors listed below the PCDHGA4 gene.
- RNA transcription data as training data resulting in the best models (i.e., models having the highest accuracy gain) relative to other trained models that were trained on other types of omic data (e.g., WGS, SNP copy number, proteomics, etc.).
- Figure 2 exemplarily illustrates a heat map for the actual patient data where each row in the map corresponds to a single patient, and each column to a specific gene (here, the genes listed in the graph of Figure 1.
- the patient data are grouped into responders (categorized in CR: complete response and PR: partial response) and non-responders (categorized in Prog: with disease progression and Stable: without disease progression).
- Color depth/grayscale value corresponds to measured transcription level and is expressed as color/gray scale value between -1.8 and 1.8.
- the final predictive score for each patient is the sum of the expression value of Figure 2 for each gene multiplied by the weighting factor.
- RNA transcription can be clamped off in silico in the pathway modeling system, and activities are re-inferred, which in effect simulates in silico the anticipated effect of a drug intervention in vivo.
- the prediction model can then be used to reassess the newly inferred post-intervention data.
- RNA transcription data one or more other (or additional) omics data are also suitable for use in conjunction with the teachings herein.
- suitable alternative or additional omics data include whole genome differential object data, exome differential object data, SNP data, copy number data, protein expression data, and/or protein activity data.
- meta data associated with the omics data need not be limited to treatment outcome, but may include a large number of alternative patient or care -relevant metrics.
- contemplated metadata may include treatment cost, likelihood of resistance, likelihood of metastatic disease, 5-year survival, suitability for immunotherapy, patient demographic information, etc.
- contemplated classifiers include one or more of a linear classifier, an NMF-based classifier, a graphical-based classifier, a tree -based classifier, a Bayesian-based classifier, a rules-based classifier, a net-based classifier, and a kNN classifier.
- especially preferred algorithms will include those that use a classifier selected form the group consisting of NMFpredictor (linear), SVMlight (linear), SVMlight first order polynomial kernel (degree-d polynomial), SVMlight second order polynomial kernel (degree-d polynomial), WEKA SMO (linear), WEKA j48 trees (trees-based), WEKA hyper pipes (distribution-based), WEKA random forests (trees-based), WEKA naive Bayes
- ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇
- the inventor contemplates that at least 5, at least 10, at least 20, at least 50, at least 100, at least 500, at least 1,000, at least 5,000, or at least 10,000 trained models are created. Depending on the number of possible training data sets, the number of validations, and the number of types of algorithms, the number of resulting trained models could even exceed 1,000,000 trained models.
- model quality is assessed and most preferably models are retained that have a prediction power that exceeds random selection. Viewed from a different perspective, models will be assessed on their gain in accuracy. There are numerous manners of assessing accuracy, and the particular choice may depend at least in part on the algorithm used.
- suitable metrics include an accuracy value, an accuracy gain, a performance metric, or other measure of the corresponding model. Additional example metrics include an area under curve metric, an R 2 , a p-value metric, a silhouette coefficient, a confusion matrix, or other metric that relates to the nature of the model or its corresponding model template.
- accuracy of a model can be derived through use of known data sets and corresponding known clinical outcome data.
- a number of evaluation models can be built that are both trained and validated against the input known data sets (e.g., k-fold cross validation).
- a trained model can be trained based on 80% of the input data. Once the evaluation model has been trained, the remaining 20% of the genomic data can be run through the evaluation model to see if it generates prediction data similar to or closet to the remaining 20% of the known clinical outcome data. The accuracy of the trained evaluation model is then considered to be the ratio of the number of correct predictions to the total number of outcomes.
- a RNA transcription data set/clinical outcome data set represents a cohort of 500 patients.
- the data sets can then be partitioned into one or more groups of evaluation training sets, e.g., containing 400 patient samples.
- Models are then created based on the 400 patient samples, and the so trained models are validated by executing the model on the remaining 100 patients' transcription data set to generate 100 prediction outcomes.
- the 100 prediction outcomes are then compared to the actual 100 outcomes from the patient data in the clinical outcome data set.
- the accuracy of the trained model is the number of correct prediction outcomes relative to the total number of outcomes. If, out of the 100 prediction outcomes, the trained evaluation model generates 85 correct outcomes that match the actual or known clinical outcomes from the patient data, then the accuracy of the trained evaluation model is considered 85%.
- the accuracy gain would be 25% (i.e., 25% above randomly observed results; predicted event occurs at 60%, correct prediction at 85%, accuracy gain is 25%)
- the model used for prediction may be selected as the top model (having highest accuracy gain, or highest accuracy score, etc.), or as being in the top n-tile (tertile, quartile, quintile, etc.), or as being in the top n% of all models (top 5%, top 10%, etc.).
- suitable models have may have an accuracy gain metric that exceeds all other models.
- the prediction based on the top (or other selected single) model may be based on all of the omics data that were part of the input data (i.e., uses all RNA expression levels used for training the models) or only a fraction of the omics data.
- the omics data with the highest or minimum absolute predetermined weight (e.g., top quartile of all weights in the single model) in the model will be generally preferred as is shown in the selected features (genes) of Figure 1.
- suitable models will employ at least 5, or at least 10, or at least 20, or at least 50, or at least 100 features in the prediction.
- features may be used, preferably in combination, in an gene expression array rather than in a predictive algorithm (e.g., significant features in Figure 2).
Abstract
Description
Claims
Priority Applications (9)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP16740607.3A EP3248127A4 (en) | 2015-01-20 | 2016-01-19 | Systems and methods for response prediction to chemotherapy in high grade bladder cancer |
US15/543,418 US11101038B2 (en) | 2015-01-20 | 2016-01-19 | Systems and methods for response prediction to chemotherapy in high grade bladder cancer |
CN201680008725.2A CN107548498A (en) | 2015-01-20 | 2016-01-19 | System and method for the chemotherapy in the high-level carcinoma of urinary bladder of response prediction |
CA2974199A CA2974199A1 (en) | 2015-01-20 | 2016-01-19 | Systems and methods for response prediction to chemotherapy in high grade bladder cancer |
KR1020177023267A KR102116485B1 (en) | 2015-01-20 | 2016-01-19 | Systems and methods for predicting response of highly differentiated bladder cancer to chemotherapy |
AU2016209478A AU2016209478B2 (en) | 2015-01-20 | 2016-01-19 | Systems and methods for response prediction to chemotherapy in high grade bladder cancer |
JP2017537902A JP2018507470A (en) | 2015-01-20 | 2016-01-19 | System and method for predicting response to chemotherapy for high-grade bladder cancer |
IL253550A IL253550B (en) | 2015-01-20 | 2017-07-18 | Systems and methods for response prediction to chemotherapy in high grade bladder cancer |
AU2019203295A AU2019203295A1 (en) | 2015-01-20 | 2019-05-10 | Systems and methods for response prediction to chemotherapy in high grade bladder cancer |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201562105697P | 2015-01-20 | 2015-01-20 | |
US62/105,697 | 2015-01-20 | ||
US201562127546P | 2015-03-03 | 2015-03-03 | |
US62/127,546 | 2015-03-03 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2016118527A1 true WO2016118527A1 (en) | 2016-07-28 |
Family
ID=56417658
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2016/013959 WO2016118527A1 (en) | 2015-01-20 | 2016-01-19 | Systems and methods for response prediction to chemotherapy in high grade bladder cancer |
Country Status (9)
Country | Link |
---|---|
US (1) | US11101038B2 (en) |
EP (1) | EP3248127A4 (en) |
JP (1) | JP2018507470A (en) |
KR (1) | KR102116485B1 (en) |
CN (1) | CN107548498A (en) |
AU (2) | AU2016209478B2 (en) |
CA (1) | CA2974199A1 (en) |
IL (1) | IL253550B (en) |
WO (1) | WO2016118527A1 (en) |
Cited By (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2019089393A1 (en) * | 2017-10-30 | 2019-05-09 | Nantomics, Llc | Temozolomide response predictor and methods |
WO2020077352A1 (en) * | 2018-10-12 | 2020-04-16 | Human Longevity, Inc. | Multi-omic search engine for integrative analysis of cancer genomic and clinical data |
EP3494504A4 (en) * | 2016-08-03 | 2020-07-22 | Nantomics, LLC | Dasatinib response prediction models and methods therefor |
US11071774B2 (en) | 2016-06-30 | 2021-07-27 | Nantcell, Inc. | Nant cancer vaccine |
US11101038B2 (en) | 2015-01-20 | 2021-08-24 | Nantomics, Llc | Systems and methods for response prediction to chemotherapy in high grade bladder cancer |
US11564980B2 (en) | 2018-04-23 | 2023-01-31 | Nantcell, Inc. | Tumor treatment method with an individualized peptide vaccine |
US11590217B2 (en) | 2018-04-23 | 2023-02-28 | Nantcell, Inc. | Neoepitope vaccine and immune stimulant combinations and methods |
US11823773B2 (en) | 2018-04-13 | 2023-11-21 | Nant Holdings Ip, Llc | Nant cancer vaccine strategies |
Families Citing this family (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN109543203B (en) * | 2017-09-22 | 2023-04-18 | 山东建筑大学 | Building cold and heat load prediction method based on random forest |
US20210228128A1 (en) * | 2018-05-08 | 2021-07-29 | Abbott Diabetes Care Inc. | Sensing systems and methods for identifying emotional stress events |
CN108611416B (en) * | 2018-05-09 | 2020-12-29 | 中国科学院昆明动物研究所 | Cervical cancer personalized prognosis evaluation method based on polygene expression profile |
US20200118691A1 (en) * | 2018-10-10 | 2020-04-16 | Lukasz R. Kiljanek | Generation of Simulated Patient Data for Training Predicted Medical Outcome Analysis Engine |
CN109671499B (en) * | 2018-10-22 | 2023-06-13 | 南方医科大学 | Method for constructing rectal toxicity prediction system |
EP3912007A4 (en) * | 2019-01-10 | 2022-11-02 | Travera LLC | Identifying cancer therapies |
US10515715B1 (en) | 2019-06-25 | 2019-12-24 | Colgate-Palmolive Company | Systems and methods for evaluating compositions |
US11368890B2 (en) * | 2020-06-30 | 2022-06-21 | At&T Intellectual Property I, L.P. | Predicting small cell capacity and coverage to facilitate offloading of macrocell capacity |
CN115565610A (en) * | 2022-09-29 | 2023-01-03 | 四川大学 | Method and system for establishing recurrence transfer analysis model based on multiple sets of mathematical data |
CN115631847B (en) * | 2022-10-19 | 2023-07-14 | 哈尔滨工业大学 | Early lung cancer diagnosis system, storage medium and equipment based on multiple groups of chemical characteristics |
CN116013528B (en) * | 2023-01-10 | 2023-11-24 | 中山大学孙逸仙纪念医院 | Bladder cancer postoperative recurrence risk prediction method, device and medium combining with FISH detection |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2005008213A2 (en) * | 2003-07-10 | 2005-01-27 | Genomic Health, Inc. | Expression profile algorithm and test for cancer prognosis |
US20070128636A1 (en) * | 2005-12-05 | 2007-06-07 | Baker Joffre B | Predictors Of Patient Response To Treatment With EGFR Inhibitors |
WO2012009382A2 (en) * | 2010-07-12 | 2012-01-19 | The Regents Of The University Of Colorado | Molecular indicators of bladder cancer prognosis and prediction of treatment response |
WO2013090620A1 (en) * | 2011-12-13 | 2013-06-20 | Genomedx Biosciences, Inc. | Cancer diagnostics using non-coding transcripts |
WO2014043803A1 (en) * | 2012-09-20 | 2014-03-27 | Genomedx Biosciences, Inc. | Thyroid cancer diagnostics |
Family Cites Families (37)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
AU1760001A (en) | 1999-11-10 | 2001-06-06 | Quest Diagnostics Investments Incorporated | Use of computationally derived protein structures of genetic polymorphisms in pharmacogenomics and clinical applications |
JP2005521138A (en) | 2002-03-15 | 2005-07-14 | パシフィック エッジ バイオテクノロジー リミティド | Medical application of adaptive learning system using gene expression data |
WO2004038376A2 (en) | 2002-10-24 | 2004-05-06 | Duke University | Binary prediction tree modeling with many predictors and its uses in clinical and genomic applications |
US9342657B2 (en) | 2003-03-24 | 2016-05-17 | Nien-Chih Wei | Methods for predicting an individual's clinical treatment outcome from sampling a group of patient's biological profiles |
US20060195266A1 (en) * | 2005-02-25 | 2006-08-31 | Yeatman Timothy J | Methods for predicting cancer outcome and gene signatures for use therein |
US20050210015A1 (en) | 2004-03-19 | 2005-09-22 | Zhou Xiang S | System and method for patient identification for clinical trials using content-based retrieval and learning |
CA2848463A1 (en) | 2004-04-09 | 2005-10-27 | Genomic Health, Inc. | Gene expression markers for predicting response to chemotherapy |
AU2005321925A1 (en) | 2004-12-30 | 2006-07-06 | Proventys, Inc. | Methods, systems, and computer program products for developing and using predictive models for predicting a plurality of medical outcomes, for evaluating intervention strategies, and for simultaneously validating biomarker causality |
TWI363309B (en) | 2006-11-30 | 2012-05-01 | Navigenics Inc | Genetic analysis systems, methods and on-line portal |
US7899764B2 (en) | 2007-02-16 | 2011-03-01 | Siemens Aktiengesellschaft | Medical ontologies for machine learning and decision support |
US20080228699A1 (en) | 2007-03-16 | 2008-09-18 | Expanse Networks, Inc. | Creation of Attribute Combination Databases |
WO2009073478A2 (en) | 2007-11-30 | 2009-06-11 | Applied Genomics, Inc. | Tle3 as a marker for chemotherapy |
US8386401B2 (en) | 2008-09-10 | 2013-02-26 | Digital Infuzion, Inc. | Machine learning methods and systems for identifying patterns in data using a plurality of learning machines wherein the learning machine that optimizes a performance function is selected |
US8484225B1 (en) | 2009-07-22 | 2013-07-09 | Google Inc. | Predicting object identity using an ensemble of predictors |
US20110262921A1 (en) * | 2010-04-23 | 2011-10-27 | Sabichi Anita L | Test for the Detection of Bladder Cancer |
US10192641B2 (en) | 2010-04-29 | 2019-01-29 | The Regents Of The University Of California | Method of generating a dynamic pathway map |
CA3007805C (en) | 2010-04-29 | 2019-11-26 | The Regents Of The University Of California | Pathway recognition algorithm using data integration on genomic models (paradigm) |
US9646134B2 (en) | 2010-05-25 | 2017-05-09 | The Regents Of The University Of California | Bambam: parallel comparative analysis of high-throughput sequencing data |
WO2011149534A2 (en) | 2010-05-25 | 2011-12-01 | The Regents Of The University Of California | Bambam: parallel comparative analysis of high-throughput sequencing data |
WO2012030840A2 (en) * | 2010-08-30 | 2012-03-08 | Myriad Genetics, Inc. | Gene signatures for cancer diagnosis and prognosis |
EP2681709A4 (en) | 2011-03-04 | 2015-05-06 | Kew Group Llc | Personalized medical management system, networks, and methods |
EP2718485A4 (en) | 2011-06-07 | 2015-05-06 | Caris Mpi Inc | Molecular profiling for cancer |
WO2012168421A1 (en) | 2011-06-10 | 2012-12-13 | Deutsches Krebsforschungszentrum | Prediction of recurrence for bladder cancer by a protein signature in tissue samples |
JP5897823B2 (en) * | 2011-06-17 | 2016-03-30 | 東レ株式会社 | Bladder cancer diagnostic composition and method |
US20140199273A1 (en) | 2011-08-05 | 2014-07-17 | Nodality, Inc. | Methods for diagnosis, prognosis and methods of treatment |
US9934361B2 (en) | 2011-09-30 | 2018-04-03 | Univfy Inc. | Method for generating healthcare-related validated prediction models from multiple sources |
MX352274B (en) | 2011-10-21 | 2017-11-16 | Nestec Sa | Methods for improving inflammatory bowel disease diagnosis. |
EP2644705A1 (en) | 2012-03-30 | 2013-10-02 | RWTH Aachen | Biomarker for bladder cancer |
US9767526B2 (en) | 2012-05-11 | 2017-09-19 | Health Meta Llc | Clinical trials subject identification system |
EP2669682B1 (en) | 2012-05-31 | 2017-04-19 | Heinrich-Heine-Universität Düsseldorf | Novel prognostic and predictive biomarkers (tumor markers) for human breast cancer |
AU2013329319B2 (en) | 2012-10-09 | 2019-03-14 | Five3 Genomics, Llc | Systems and methods for learning and identification of regulatory interactions in biological pathways |
US20140143188A1 (en) | 2012-11-16 | 2014-05-22 | Genformatic, Llc | Method of machine learning, employing bayesian latent class inference: combining multiple genomic feature detection algorithms to produce an integrated genomic feature set with specificity, sensitivity and accuracy |
US20140279754A1 (en) | 2013-03-15 | 2014-09-18 | The Cleveland Clinic Foundation | Self-evolving predictive model |
KR20200043486A (en) | 2013-05-28 | 2020-04-27 | 파이브3 제노믹스, 엘엘씨 | Paradigm drug response networks |
EP3095055A4 (en) * | 2014-01-17 | 2017-10-11 | Ontario Institute For Cancer Research | Biopsy-driven genomic signature for prostate cancer prognosis |
US11101038B2 (en) | 2015-01-20 | 2021-08-24 | Nantomics, Llc | Systems and methods for response prediction to chemotherapy in high grade bladder cancer |
US20180039731A1 (en) | 2015-03-03 | 2018-02-08 | Nantomics, Llc | Ensemble-Based Research Recommendation Systems And Methods |
-
2016
- 2016-01-19 US US15/543,418 patent/US11101038B2/en active Active
- 2016-01-19 CN CN201680008725.2A patent/CN107548498A/en not_active Withdrawn
- 2016-01-19 AU AU2016209478A patent/AU2016209478B2/en active Active
- 2016-01-19 CA CA2974199A patent/CA2974199A1/en not_active Withdrawn
- 2016-01-19 JP JP2017537902A patent/JP2018507470A/en not_active Ceased
- 2016-01-19 WO PCT/US2016/013959 patent/WO2016118527A1/en active Application Filing
- 2016-01-19 EP EP16740607.3A patent/EP3248127A4/en not_active Withdrawn
- 2016-01-19 KR KR1020177023267A patent/KR102116485B1/en active IP Right Grant
-
2017
- 2017-07-18 IL IL253550A patent/IL253550B/en not_active IP Right Cessation
-
2019
- 2019-05-10 AU AU2019203295A patent/AU2019203295A1/en not_active Abandoned
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2005008213A2 (en) * | 2003-07-10 | 2005-01-27 | Genomic Health, Inc. | Expression profile algorithm and test for cancer prognosis |
US20070128636A1 (en) * | 2005-12-05 | 2007-06-07 | Baker Joffre B | Predictors Of Patient Response To Treatment With EGFR Inhibitors |
WO2012009382A2 (en) * | 2010-07-12 | 2012-01-19 | The Regents Of The University Of Colorado | Molecular indicators of bladder cancer prognosis and prediction of treatment response |
WO2013090620A1 (en) * | 2011-12-13 | 2013-06-20 | Genomedx Biosciences, Inc. | Cancer diagnostics using non-coding transcripts |
WO2014043803A1 (en) * | 2012-09-20 | 2014-03-27 | Genomedx Biosciences, Inc. | Thyroid cancer diagnostics |
Non-Patent Citations (1)
Title |
---|
See also references of EP3248127A4 * |
Cited By (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11101038B2 (en) | 2015-01-20 | 2021-08-24 | Nantomics, Llc | Systems and methods for response prediction to chemotherapy in high grade bladder cancer |
US11071774B2 (en) | 2016-06-30 | 2021-07-27 | Nantcell, Inc. | Nant cancer vaccine |
US11207392B2 (en) | 2016-06-30 | 2021-12-28 | Nantcell Inc. | Coordinated multi-prong cancer therapy |
US11439697B2 (en) | 2016-06-30 | 2022-09-13 | Nantcell, Inc. | Nant cancer vaccine |
EP4101447A1 (en) | 2016-06-30 | 2022-12-14 | Nant Holdings IP, LLC | Nant cancer vaccine |
EP3494504A4 (en) * | 2016-08-03 | 2020-07-22 | Nantomics, LLC | Dasatinib response prediction models and methods therefor |
WO2019089393A1 (en) * | 2017-10-30 | 2019-05-09 | Nantomics, Llc | Temozolomide response predictor and methods |
US11823773B2 (en) | 2018-04-13 | 2023-11-21 | Nant Holdings Ip, Llc | Nant cancer vaccine strategies |
US11564980B2 (en) | 2018-04-23 | 2023-01-31 | Nantcell, Inc. | Tumor treatment method with an individualized peptide vaccine |
US11590217B2 (en) | 2018-04-23 | 2023-02-28 | Nantcell, Inc. | Neoepitope vaccine and immune stimulant combinations and methods |
WO2020077352A1 (en) * | 2018-10-12 | 2020-04-16 | Human Longevity, Inc. | Multi-omic search engine for integrative analysis of cancer genomic and clinical data |
JP2022504916A (en) * | 2018-10-12 | 2022-01-13 | ヒューマン ロンジェヴィティ インコーポレイテッド | Multi-omics search engine for integrated analysis of cancer genes and clinical data |
Also Published As
Publication number | Publication date |
---|---|
US20180004905A1 (en) | 2018-01-04 |
EP3248127A1 (en) | 2017-11-29 |
AU2019203295A1 (en) | 2019-05-30 |
US11101038B2 (en) | 2021-08-24 |
IL253550A0 (en) | 2017-09-28 |
IL253550B (en) | 2020-05-31 |
AU2016209478B2 (en) | 2019-03-07 |
KR20180010176A (en) | 2018-01-30 |
CN107548498A (en) | 2018-01-05 |
JP2018507470A (en) | 2018-03-15 |
CA2974199A1 (en) | 2016-07-28 |
KR102116485B1 (en) | 2020-05-28 |
EP3248127A4 (en) | 2018-08-08 |
AU2016209478A1 (en) | 2017-08-10 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11101038B2 (en) | Systems and methods for response prediction to chemotherapy in high grade bladder cancer | |
Kelley et al. | Sequential regulatory activity prediction across chromosomes with convolutional neural networks | |
AU2016280074B2 (en) | Systems and methods for patient-specific prediction of drug responses from cell line genomics | |
AU2017202808B2 (en) | Paradigm drug response networks | |
JP2020501240A (en) | Methods and systems for predicting DNA accessibility in pan-cancer genomes | |
US20180039732A1 (en) | Dasatinib response prediction models and methods therefor | |
CN105144178A (en) | Systems and methods for clinical decision support | |
Hossain et al. | Application of skew-normal distribution for detecting differential expression to microRNA data | |
Haddon | Evaluation of Some Statistical Methods for the Identification of Differentially Expressed Genes |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 16740607 Country of ref document: EP Kind code of ref document: A1 |
|
DPE1 | Request for preliminary examination filed after expiration of 19th month from priority date (pct application filed from 20040101) | ||
WWE | Wipo information: entry into national phase |
Ref document number: 15543418 Country of ref document: US |
|
ENP | Entry into the national phase |
Ref document number: 2974199 Country of ref document: CA Ref document number: 2017537902 Country of ref document: JP Kind code of ref document: A |
|
WWE | Wipo information: entry into national phase |
Ref document number: 253550 Country of ref document: IL |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
REEP | Request for entry into the european phase |
Ref document number: 2016740607 Country of ref document: EP |
|
ENP | Entry into the national phase |
Ref document number: 2016209478 Country of ref document: AU Date of ref document: 20160119 Kind code of ref document: A |
|
ENP | Entry into the national phase |
Ref document number: 20177023267 Country of ref document: KR Kind code of ref document: A |