US20220093215A1 - Discovering genomes to use in machine learning techniques - Google Patents

Discovering genomes to use in machine learning techniques Download PDF

Info

Publication number
US20220093215A1
US20220093215A1 US17/339,583 US202117339583A US2022093215A1 US 20220093215 A1 US20220093215 A1 US 20220093215A1 US 202117339583 A US202117339583 A US 202117339583A US 2022093215 A1 US2022093215 A1 US 2022093215A1
Authority
US
United States
Prior art keywords
genome
feature
genomes
machine learning
features
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
US17/339,583
Inventor
Paul Grouchy
Timothy Burton
Ali Khosousi
Abhinav Doomra
Sunny GUPTA
Ian Shadforth
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Analytics 4 Life Inc
Original Assignee
Analytics 4 Life Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Analytics 4 Life Inc filed Critical Analytics 4 Life Inc
Priority to US17/339,583 priority Critical patent/US20220093215A1/en
Publication of US20220093215A1 publication Critical patent/US20220093215A1/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G16INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16BBIOINFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR GENETIC OR PROTEIN-RELATED DATA PROCESSING IN COMPUTATIONAL MOLECULAR BIOLOGY
    • G16B40/00ICT specially adapted for biostatistics; ICT specially adapted for bioinformatics-related machine learning or data mining, e.g. knowledge discovery or pattern finding
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/12Computing arrangements based on biological models using genetic models
    • G06N3/126Evolutionary algorithms, e.g. genetic algorithms or genetic programming
    • GPHYSICS
    • G01MEASURING; TESTING
    • G01NINVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N33/00Investigating or analysing materials by specific methods not covered by groups G01N1/00 - G01N31/00
    • G01N33/48Biological material, e.g. blood, urine; Haemocytometers
    • G01N33/50Chemical analysis of biological material, e.g. blood, urine; Testing involving biospecific ligand binding methods; Immunological testing
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00Machine learning
    • G06N20/20Ensemble learning
    • GPHYSICS
    • G16INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16BBIOINFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR GENETIC OR PROTEIN-RELATED DATA PROCESSING IN COMPUTATIONAL MOLECULAR BIOLOGY
    • G16B5/00ICT specially adapted for modelling or simulations in systems biology, e.g. gene-regulatory networks, protein interaction networks or metabolic networks
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00Machine learning
    • G06N20/10Machine learning using kernel methods, e.g. support vector machines [SVM]
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/08Learning methods
    • G06N3/086Learning methods using evolutionary algorithms, e.g. genetic algorithms or genetic programming
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N5/00Computing arrangements using knowledge-based models
    • G06N5/01Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
    • GPHYSICS
    • G16INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16ZINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS, NOT OTHERWISE PROVIDED FOR
    • G16Z99/00Subject matter not provided for in other main groups of this subclass

Definitions

  • Machine learning techniques predict outcomes based on sets of input data. For example, machine learning techniques are being used to predict weather patterns, geological activity, provide medical diagnoses, and so on. Machine learning techniques rely on a set of features generated using a training set of data (i.e., a data set of observations, in each of which an outcome to be predicted is known), each of which represents some measurable aspect of observed data, to generate and tune one or more predictive models. For example, observed signals (e.g., heartbeat signals from a number of subjects) may be analyzed to collect frequency, average values, and other statistical information about these signals.
  • a training set of data i.e., a data set of observations, in each of which an outcome to be predicted is known
  • observed signals e.g., heartbeat signals from a number of subjects
  • a machine learning technique may use these features to generate and tune a model that relates these features to one or more conditions, such as some form of cardiovascular disease (CVD), including coronary artery disease (CAD), and then apply that model to data sources with unknown outcomes, such as an undiagnosed patient or future weather patterns, and so on.
  • CVD cardiovascular disease
  • CAD coronary artery disease
  • these features are manually selected and combined by data scientists working with domain experts.
  • FIG. 1 is a block diagram illustrating an environment in which the facility operates in some embodiments.
  • FIG. 2 is a flow diagram illustrating the processing of a discover features component in some embodiments.
  • FIG. 3 is a flow diagram illustrating the processing of a process component in some embodiments.
  • FIG. 4 is a flow diagram illustrating the processing of an apply feature generators component in some embodiments.
  • FIG. 5 is a flow diagram illustrating the processing of an identify novel feature vectors component in some embodiments.
  • FIG. 6 is a flow diagram illustrating the processing of a discover genomes component in accordance with some embodiments.
  • FIG. 7 is a flow diagram illustrating the processing of a spawn genome component in accordance with some embodiments.
  • FIG. 8 is a flow diagram illustrating the processing of an identify high-performing genomes component in accordance with some embodiments.
  • machine learning techniques rely on features and/or combinations of features, the process of feature selection and combination typically is an important part of a machine learning process.
  • machine learning algorithms e.g., decision trees, artificial neural networks (ANNs), deep ANNs, genetic (and meta-genetic) algorithms, and so on
  • the choice of algorithm and any associated parameters can also be important.
  • different machine learning algorithms or family of machine learn algorithms
  • different machine learning algorithms may present various tradeoffs with respect to resources (e.g., memory, processor utilization), speed, accuracy, and so on.
  • models are trained using machine learning algorithms, features, and parameters selected by individuals based on the preferences of those individuals and/or criteria specified by those individuals.
  • the inventors have recognized that it can be expensive and time-consuming manually to identify features, machine learning algorithms, and corresponding parameters and even more difficult to produce features, machine learning algorithms, and corresponding parameters that produce more accurate models and, therefore, more accurate predictions. Accordingly, the inventors have conceived and reduced to practice a facility that performs automatic discovery of combinations of features, machine learning algorithms, and/or machine learning parameters.
  • the facility operates as part of a machine learning pipeline that constructs and evaluates predictive models, such as those for disease diagnosis, based on time-series and/or other signals, such as physiological signals.
  • the machine learning process uses features to identify patterns within a training set of data and, based on these patterns, generates predictive models. These predictive models can be validated using validation data sets (i.e., data sets for which an outcome is known but that were not used to train the model) and applied to new input data in order to predict outcomes from the input data, such as providing a diagnosis for a medical condition, etc.
  • validation data sets i.e., data sets for which an outcome is known but that were not used to train the model
  • new input data i.e., data sets for which an outcome is known but that were not used to train the model
  • new input data i.e., data sets for which an outcome is known but that were not used to train the model
  • new input data i.e., data sets for which an outcome is known but that were not used to train the model
  • the facility seeks to identify combinations of features and machine learning algorithm parameters where each combination can be used to train one or more models.
  • a combination of features and/or machine learning parameters is sometimes referred herein to as a “genome.”
  • the facility evaluates each genome based on the ability of a model trained using a machine learning algorithm and that genome to produce accurate results when applied to a validation data set by, for example, generating a fitness or validation score for the trained model and the corresponding genome used to train the model.
  • multiple models may be trained using a genome and the resulting fitness scores can be aggregated to generate an aggregated fitness score for the genome.
  • the facility for identifying combinations of features and machine learning algorithm parameters can be used for a medical diagnosis predictive modeling task.
  • the facility receives, for a number of patients or subjects, one or more sets of physiological data that relate to some type of physiological output or condition of the patient over a period of time (e.g., less than a second, on the order of a few seconds, about ten seconds, about 30 seconds and up to about five minutes, about an hour or more, etc.), such as electroencephalograms, and so on.
  • a period of time e.g., less than a second, on the order of a few seconds, about ten seconds, about 30 seconds and up to about five minutes, about an hour or more, etc.
  • electroencephalograms e.g., electroencephalograms, and so on.
  • the facility discards certain portions of the signal to ensure that the signals from each patient commence at a stable and consistent initial condition.
  • the data may be normalized to remove potentially misleading information.
  • the facility can normalize the amplitude of signal data (e.g., transforming to a z-score), to account for variations in signal strength caused by sensor contact or other non-physiological data.
  • the facility can perform a peak search and discard any data before a first heartbeat identified in the signal and after a last heartbeat identified in the signal.
  • the facility applies a set of feature generators to a set of signals to generate, for each combination of a signal and a feature generator, a feature value for the signal.
  • each feature value is representative of some property of the underlying signal data.
  • the facility receives patient data for each of 1000 patients and applies one or more feature generators to the data to generate, for each application of a feature generator to the data of a single patient, a feature value (or set of feature values).
  • the facility collects the feature values generated by a single feature generator in a “feature vector,” such that the feature vector stores one feature value per patient. Once the feature vectors are generated, they can be compared to determine how different each is relative to each of the other feature vectors.
  • the facility computes a distance metric for each feature vector to assess the novelty of the corresponding feature generator. Based on the assessed novelty, the facility (1) provides the feature generators that produced the novel feature vectors to the machine learning process for the purpose of basing new predictive models on the provided feature generators and (2) modifies these feature generators to create a new generation of feature generators. The facility repeats this evolutionary process to identify even more novel features for use by the machine learning process.
  • the facility computes or identifies separate sets of one or more values from the data. For example, in the case of data generated as part of an electrocardiogram, the facility identifies global and local maxima and minima within the data, computes frequency/period information from the data, calculates average values of the data over a certain period of time (e.g., the average duration and values generated during a QRS complex), and so on. In some cases, the facility transforms the received data and extracts sets of one or more values from the transformed data.
  • the facility can transform received signal data in any number of ways, such as taking one or more (successive) derivatives of the data, taking one or more partial derivatives of the data, integrating the data, calculating the gradient of the data, applying a function to the data, applying a Fourier transform, applying linear or matrix transformations, generating topology metrics/features, generating computational geometry metrics/features, generating differential manifold metrics/features, and so on. In this manner, the facility generates multiple perspectives of the data in order to yield a diverse set of features. While these transformations are provided by way of example, one of ordinary skill will recognize that the data can be transformed in any number of ways.
  • the facility receives multiple input signals (e.g., input signals collected by different electrodes or leads connected to a patient, multimodal signals, such as signals from leads of wide-band biopotential measuring equipment and a channel of S p O 2 (blood oxygen saturation), and so on) and/or transformed signals and extracts values from the signal data by computing, for each signal, an average value of the signal over the sampling period.
  • multiple input signals e.g., input signals collected by different electrodes or leads connected to a patient
  • multimodal signals such as signals from leads of wide-band biopotential measuring equipment and a channel of S p O 2 (blood oxygen saturation), and so on
  • S p O 2 blood oxygen saturation
  • the facility receives multiple input signals (e.g., input signals collected by different electrodes or leads connected to a patient, multimodal signals, such as signals from leads of wide-band biopotential measuring equipment and a channel of S p O 2 (blood oxygen saturation), and so on) and/or transformed signals and extracts values from the signal data by computing, for each signal
  • Table 1 represents a set of average signal values (A, B, C, and D) for each of n patients. Although average values have been used here, one of ordinary skill in the art will recognize that any type of data can be extracted or computed from the underlying data signals, such as the amount of time that a signal exceeded a threshold value, the values for one signal while the value of another signal exceeded a threshold value, and so on.
  • the facility applies one or more feature generators to the received or generated data, such as the extracted data, the raw or preprocessed signal data, the transformed data, and so on.
  • a feature generator receives as input at least a portion or representation of the signal data and produces a corresponding output value (or set of values) (i.e., a “feature”).
  • One set of feature generators includes the following equations:
  • each of A, B, C, and D represents a value extracted from a specific patient's data and S(t) represents, for each signal, the value of the signal at time t.
  • F1 represents the name of the feature while the equation A+C ⁇ D represents the corresponding feature generator.
  • the facility employs composite feature generators in which one feature generator serves as an input to another feature generator, such as:
  • F ⁇ 4 F ⁇ 1 * F ⁇ 2 F ⁇ ⁇ 3 3 + . 0 ⁇ 57 ( Eq ⁇ ⁇ 4 )
  • the facility applies feature generators to the extracted data of each patient represented in Table 1 to generate, for each feature generator, a feature vector of three values (one for each patient), such as those represented in Table 2 below:
  • each feature generator F1, F2, and F3 has applied each feature generator F1, F2, and F3 to the extracted data shown in Table 1 to generate, for each feature generator, a corresponding feature vector that includes a value for each patient.
  • the feature vector generated by applying feature generator Fl to the extracted data includes a value of ⁇ 29.76 for Patient 1, a value of ⁇ 0.6 for patient 2, and so on.
  • each feature vector represents, for a specific feature generator, a signature (not necessarily unique) for the corresponding feature generator based on at least a portion of each patient's physiological data (i.e., the patients represented in the physiological data to which the feature generators were applied).
  • feature generators are expressed using different structures or models, such as expression trees, neural networks, etc.
  • the facility may employ any number of feature generators and any number of sets of physiological data (or portions thereof) in the generation of feature vectors.
  • the facility randomly selects a number of previously-generated feature generators for use in generating feature vectors rather than employing each and every available feature generator.
  • the facility creates and/or modifies feature generators by, for example, randomly generating expression trees, randomly assigning weights to connections within a neural network, and so on.
  • the facility employs some form of novelty search to identify the most “novel” feature vectors among the generated feature vectors. Novelty corresponds to how different a particular feature vector is from each of a comparison set of other feature vectors (made up of any feature vectors generated by the facility during a current iteration and feature vectors produced by feature generators selected in any earlier iteration); the greater the difference from the feature vectors of the comparison set, the greater the novelty.
  • the facility uses a form of distance as a measure of novelty (i.e., how “far” each feature vector is from the other feature vectors).
  • the facility calculates the distance between that feature vector and each of the other generated feature vectors and performs an aggregation of the generated distance values, such as calculating an average or mean (e.g., arithmetic, geometric, harmonic, etc.) distance value for the feature vector, or a total (sum) distance between the feature vector and each of the other generated feature vectors, identifying a mode distance value, a median distance value, a maximum distance value for the feature vector, and so on.
  • an average or mean e.g., arithmetic, geometric, harmonic, etc.
  • a total (sum) distance between the feature vector and each of the other generated feature vectors identifying a mode distance value, a median distance value, a maximum distance value for the feature vector, and so on.
  • the facility includes feature vectors produced by feature generators selected in an earlier generation.
  • the facility applies a weight, such as a randomly generated weight, to each of the feature vectors and/or normalizes each set of feature vectors prior to comparison.
  • a weight such as a randomly generated weight
  • the facility identifies the most “novel” feature vectors based on the calculated distances, which act as a “novelty score” or “fitness score” for each of the feature vectors.
  • the facility identifies the feature vectors with the greatest average distance to other vectors (e.g., the feature vector generated by F3), the feature vectors with the greatest MAX distance (e.g., the feature vectors generated by F1 and F3), and so on.
  • the number of novel feature vectors identified is fixed (or capped) at a predetermined number, such as five, ten, 100, 500, etc.
  • the number of novel feature vectors to be identified is determined dynamically, such as the top 10% of analyzed feature vectors based on novelty scores, any feature vectors having a novelty scores that is more than a predetermined number of standard deviations beyond a mean novelty score for the analyzed feature vectors, and so on.
  • the feature generators that produced each of these identified novel feature vectors can then be added to the set of features available for use as inputs to models constructed and evaluated by the machine learning pipeline. Those models can be applied to patient data for, e.g., diagnostic, predictive, therapeutic, or other analytic, scientific, health-related or other purposes.
  • the facility randomly mutates or modifies the feature generators used to generate the identified novel feature vectors. Each mutation effects some change in the corresponding feature generator and creates a new version of the feature generator that can be used to contribute to a new generation of feature generators. The facility uses this new feature generator to generate new feature vectors, and then assesses the novelty of the new feature vectors. Moreover, the corresponding feature generator can be further mutated to continue this process of feature vector and feature generation creation.
  • F1 0 represents F1 above (Eq 1) at generation 0 (i.e., the first generation)
  • F1 1 represents a mutated version of F1 at generation 1 (i.e., the second generation)
  • the facility obtains features in different ways.
  • the facility may receive from a user, such as a domain expert, a set of features (and corresponding feature generators) that the user has identified as being optimal and/or that the user desires to be tested.
  • the features may be editorially selected from one or more feature stores.
  • features automatically generated by the facility can be combined with other features to create various hybrid features. Even features of unknown provenance may be used.
  • the facility identifies genomes to train models, identifies, from among these genomes, the “best” (highest rated) genomes, and mutates the identified genomes to produce even more genomes that can be used to train models. After using a genome to train one or more models, the facility applies each trained model to a validation data set so that the trained model can be scored (e.g., how well does the trained model correctly identify and/or classify subjects in the underlying validation data set).
  • the facility mutates the genomes that produce the best results (e.g., have the highest validation or fitness scores), trains new models using these mutated genomes, and repeats this process until one or more termination criteria are met (e.g., a predetermined number of generations, no additional high scoring (higher than a predetermined or dynamically generated threshold) genomes are generated during a predetermined or dynamically number (e.g., 1, 5, 8, 17, etc.) of previous generations, a combination thereof, etc.).
  • termination criteria e.g., a predetermined number of generations, no additional high scoring (higher than a predetermined or dynamically generated threshold) genomes are generated during a predetermined or dynamically number (e.g., 1, 5, 8, 17, etc.) of previous generations, a combination thereof, etc.).
  • the facility uses previously identified or generated genomes as a first set of genomes (i.e., a first generation) from which to discover genomes for machine learning algorithms.
  • the facility automatically generates a first generation of genomes by, for each genome, randomly (with or without replacement) selecting one or more feature vectors from one or more previously generated sets of feature vectors (e.g., a feature vector produced by applying a feature generator to a set of training data).
  • a genome may also include one or more machine learning algorithm parameters to the machine learning algorithm, such as the number of predictors (e.g., regressors, classifiers, the number and/or the maximum number of decision trees to use for a machine learning algorithm, etc.) to use for an underlying ensemble method associated with the algorithm, a maximum depth for a machine learning algorithm (e.g., maximum depth for decision trees), and so on.
  • the genome can be configured to define a value for each machine learning parameter associated with that machine learning algorithm.
  • one of the elements of the genome selects among different machine learning algorithms and may be mutated so that the genome and its corresponding parameter values are used with different machine learning algorithms to train models over the evolutionary process.
  • a genome may identify a machine learning algorithm that relies on decision trees while a mutated version of that same genome identifies a machine learning algorithm that uses one or more support vector machines, linear models, etc.
  • the genome may specify a modeling parameter for each and every machine learning algorithm that may be combined with the genome to train a model.
  • a single genome may include machine learning parameters for multiple machine learning algorithms.
  • a genome need not include each and every modeling parameter for a corresponding machine learning algorithm.
  • the facility can retrieve a default value for these parameters from, for example, a machine learning parameter store.
  • a set of genomes may be represented as:
  • each row corresponds to a different genome (named in the first column from the left) from among a first generation of selected or generated genomes and identifies a machine learning algorithm (“MLA”; second column from the left) to use to train a model using the genome, such as an index into a machine learning algorithm store.
  • MLA machine learning algorithm store
  • each non-bolded region (to the right of the second column) identifies a different feature.
  • a genome can also include a corresponding feature generator or a reference to the corresponding feature generator, such as a link to feature generator store. As discussed above, these features may be generated automatically by the facility and/or retrieved from another source.
  • each bolded region in Table 4 represents a value for a particular machine learning parameter.
  • machine learning parameters are represented by an indicator or reference (e.g., P6:1) followed by an equals sign and a corresponding value.
  • machine learning algorithm parameter P6:1 has a corresponding value of 8 in genome G20 1 .
  • each machine learning parameter is presented as an index into a two-dimensional array, such that “P6:1” represents the “first” machine learning parameter of the “sixth” machine learning algorithm (i.e., the machine learning parameter with an index of 1 for the machine learning algorithm with an index of 6).
  • a genome may specify values for any or all machine learning parameters that may be used to training a model using the genome (or a mutated version of that genome).
  • genomes may be of varying length.
  • genome G1 1 includes values for six features and zero machine learning parameters while gnome G2 1 includes values for two features and three machine learning parameters. Accordingly, the facility may employ variable-length genomes in the machine learning processes.
  • the facility may filter features from within genomes and/or filters genomes themselves to avoid redundancy among each.
  • the facility In order to filter features and/or genomes, the facility generates correlation values for each pair and discards one item of the pair.
  • the facility To identify and filter correlated features from a genome, the facility generates, for each of the features, a feature vector by applying a feature generator associated with the feature to a training set of data to produce a set of values. The facility compares each of the generated feature vectors to the other generated feature vectors to determine whether any of the feature vectors are “highly” correlated (i.e., not “novel” within the selected set of feature vectors).
  • the component may calculate a distance value for each of the generated feature vectors relative to the other feature vectors (as discussed above with respect to identifying novel feature generators) and, if the distance between any pair (set of two) is less than or equal to a distance threshold (i.e., “highly” correlated or not “novel”), discard a feature corresponding to one of the pair of feature vectors.
  • a distance threshold i.e., “highly” correlated or not “novel”
  • the facility may replace the discarded feature with a new feature, such as a randomly-selected feature.
  • the facility may identify and discard redundant genomes by generating, for each feature of the genome, a feature vector, calculating distance metrics for each pair (set of two) of genomes based on the generated feature vectors, and identifying pairs of genomes whose calculated distances do not exceed a genome distance threshold. For each identified pair of genomes, the facility may discard or mutate one or both of the genomes to reduce correlation and redundancy among a group of genomes.
  • distance is used in this example as a metric for determining a correlation between two vectors or sets of vectors, one of ordinary skill in the art will recognize that correlations between two or sets of vectors can be calculated in other ways, such as normalized cross-correlation, and so on.
  • the facility may employ additional or other techniques to filter genomes, such as generating a graph where features represent vertices in the graph which are connected via edges in the graph. An edge between two features is generated if, for example, a correlation value between the two features exceeds a predetermined correlation threshold and/or the distance between the two features is less than a predetermined distance threshold.
  • the facility removes connected vertices (features) from the graph until no edges remain in the graph (an edge being removed when a connected vertex is removed) and selects the remaining non-connected vertices (features) for inclusion in the “filtered” genome.
  • the facility may randomly select connected vertices for removal.
  • the facility may perform this process multiple times for a set of vertices (features) and then select a preferred “filtered” genome, such as the genome with the most or least vertices (features) removed.
  • the facility trains at least one model using the features, machine learning parameters, and/or machine learning algorithm(s) of that genome.
  • the facility can use AdaBoost (“Adaptive Boosting”) techniques to train a model using the corresponding features, machine learning parameters, machine learning algorithm, and a training set of data.
  • AdaBoost Adaptive Boosting
  • the facility applies the trained model to one or more sets of validation data to assess how well the trained model identifies and/or classifies previously-identified or classified subjects within the validation data set.
  • a genome may be generated to train models to identify patients represented in a data set who are likely to have diabetes.
  • the trained model can be applied to a validation set of data to determine a validation score that reflects how well the trained model identifies patients from the validation set that are known to have or now have diabetes; scoring (adding) one “point” for every correct determination (e.g. true positives and true negatives) and losing (subtracting) one “point” for every incorrect determination (e.g., false positives and false negatives).
  • an overall score for the trained model can be determined based on how many “points” the trained model scores when applied to one or more sets of validation data.
  • a fitness score for a trained model may be generated by calculating the area under a corresponding receiver operating characteristic (ROC) curve, calculating a mean squared prediction error, f scores, sensitivity, specificity, negative and positive predictive values, diagnostic odds ratios, and so on.
  • the generated fitness score may be similarly attributed to the genome.
  • the genome may be used to train multiple machine learning algorithms and each of those trained machine learning algorithms may be applied to multiple validation sets to produce, for each genome used to train machine algorithms, multiple fitness scores.
  • the facility generates a fitness score for the corresponding genome by aggregating each of the fitness scores generated for the machine learning algorithms trained using the genome.
  • the generated fitness scores may be aggregated and/or filtered prior to aggregation.
  • the facility identifies the “best” genomes based on these fitness scores. For example, the facility can establish a fitness threshold based on the produced fitness scores and identify the “best” genomes as those genomes whose resulting fitness scores exceed the fitness threshold.
  • the fitness threshold may be generated or determined in any number of ways, such as receiving a fitness threshold from a user, calculating a fitness threshold based on the set of fitness scores (e.g., average, average plus 15%, top fifteen, top n-th percentile (where n is provided by a user or generated automatically by the facility)), and so on.
  • the facility stores each of the genomes in association with their corresponding fitness scores and selects the genomes identified as “best” for mutation (i.e., the genomes having a fitness score that exceeds a fitness threshold).
  • the facility mutates a genome by adding, removing, or changing any one or more of the feature vectors or machine learning parameters of the genome.
  • Table 5 represents a number of mutations to the genomes represented above in Table 4.
  • each row corresponds to a different genome (named in the first column from the left) from among a second generation of genomes selected for mutation.
  • Genome G1 1 has been mutated (represented as G1 2 ) by removing three feature vectors (represented with a strikethrough) and changing the references machine learning algorithm index from 4 to 5.
  • the facility has mutated genome G2 1 by 1) removing feature vector F9701, 2) adding feature vector F584, and 3) adjusting machine learning parameter P9 1 from 7 to 12; genome G4 1 by adding features F24 and F982; and genome Gn 1 by multiplying values generated by F65 by values generated by F14.
  • mutated genomes can then be used to train one or more machine learning algorithms, scored by applying the trained machine learning algorithm to one or more validation data sets, selected for mutation, mutated, and so on.
  • the facility performs this process until a termination point is reached, such as when a predetermined number of generations has been produced (e.g., six, 30, 100,000, etc.), and so on.
  • FIG. 1 is a block diagram illustrating an environment 100 in which the facility operates in accordance with some embodiments of the disclosed technology.
  • environment 100 includes service provider 110 , signal recorder 140 (e.g., a wide-band biopotential measuring equipment), data providers 150 , patient 160 , and network 160 .
  • service provider includes facility 120 , which includes discover features component 121 , process component 122 , apply feature generators component 123 , identify novel feature vectors component 124 , discover genomes component 125 , spawn genome component 126 , identify high-performing genomes component 127 , patient data store 130 , model store 131 , feature vector store 132 , and feature generator store 133 .
  • Discover features component 121 is invoked by the facility to identify and mutate feature generators based on received data.
  • Process component 122 is invoked by the discover features component 121 to process and transform patient signal data, such as raw signal data from signal recorder 140 (e.g., one or more measurement devices and/or systems used to collect the underlying data such as wide-band biopotential measuring equipment, etc.), 3-D image data, etc.
  • Apply feature generators component 123 is invoked by the discover features component to apply a set of one or more feature generators to the processed and transformed patient signal data.
  • Identify novel feature vectors component 124 is invoked by the discover features component to identify the most novel feature vectors from among a group of feature vectors generated by, for example, one or more feature generators.
  • Discover genomes component 125 is invoked by the facility 120 to generate, analyze, and mutate genomes for use by machine learning algorithms.
  • Spawn genome component 126 is invoked by the discover genomes component to generate a genome comprising any number of feature vectors and/or machine learning parameters.
  • Identify high-performing genomes component 127 is invoked by the discover genomes component to identify, from among a group of genomes, the genomes that have a corresponding fitness score that exceeds a fitness threshold.
  • Patient data store 130 includes physiological patient data, such as raw physiological data (including but not limited to data obtained via, e.g., signal recorder 140 ), transformed physiological data, biographical information, demographic information, etc.
  • Model store 131 stores information about models generated by applying machine learning techniques to training data, such as the machine learning techniques described in Christopher M. Bishop, Pattern Recognition and Machine Learning (2006) (Library of Congress Control Number: 2006922522; ISBN-10: 0-387-31073-8), which is herein incorporated by reference in its entirety.
  • Feature vector store 132 stores sets of feature vectors generated by applying one or more feature generators to a set of physiological data.
  • Feature generator store 133 stores sets of feature generators that can be applied to patient physiological data and can include multiple generations of feature generators.
  • Genome store 134 stores generated and/or mutated genomes created by the facility and/or other sources.
  • Machine learning parameters store 135 stores, for each of a number of machine learning algorithms, a set of parameters that can serve as inputs to that machine learning algorithm and additional information related to the parameter, such as a maximum value for a corresponding parameter, a minimum value for a corresponding parameter, a default value for a corresponding parameter, etc.
  • Machine learning algorithm store 133 stores the logic for each of a number of machine learning algorithms, each of which can be selectively trained and validated by the facility.
  • a signal recorder 140 is connected to patient 160 via electrodes 145 and includes facility 120 , one or more output devices 142 , such as a monitor, printer, speaker, etc., and one or more input devices 144 , such as settings controls, keyboard, biometric data reader, etc.
  • the facility can be configured to operate remotely from a patient and other diagnostics equipment and/or in conjunction with or part of the diagnostics equipment such as a wide-band biopotential measuring equipment (i.e., any device configured to capture unfiltered electrophysiological signals, including those with spectral components that are not altered).
  • the facility can be configured to operate in real-time with the reading of physiological data and/or can be applied to previously recorded physiological data.
  • Data providers 150 each of which includes a data store 152 , can provide information for analysis or use by the facility such as, physiological patient data recorded off-site (e.g., at a hospital or clinic without access to a facility on premises, third-party data providers, etc.), feature vectors and/or feature generators produced or generated elsewhere, and so on.
  • Network 170 represents communications links over which the various elements of environment 100 may communicate, such as the internet, a local area network, and so on.
  • these computer systems and other devices can include server computer systems, desktop computer systems, laptop computer systems, netbooks, tablets, mobile phones, personal digital assistants, televisions, cameras, automobile computers, electronic media players, appliances, wearable devices, other hardware, and/or the like.
  • the facility 120 may operate on specific-purpose computing systems, such as wide-band biopotential measuring equipment (or any device configured to capture unfiltered electrophysiological signals, including electrophysiological signals with unaltered spectral components), electroencephalogram equipment, radiology equipment, sound recording equipment, and so on.
  • the computer systems and devices include one or more of each of the following: a central processing unit (“CPU”) configured to execute computer programs; a computer memory configured to store programs and data while they are being used, including a multithreaded program being tested, a debugger, the facility, an operating system including a kernel, and device drivers; a persistent storage device, such as a hard drive or flash drive configured to persistently store programs and data (e.g., firmware and the like); a computer-readable storage media drive, such as a floppy, flash, CD-ROM, or DVD drive, configured to read programs and data stored on a computer-readable storage medium, such as a floppy disk, flash memory device, CD-ROM, or DVD; and a network connection configured to connect the computer system to other computer systems to send and/or receive data, such as via the internet, a Local Area Network (LAN), a Wide Area Network (WAN), a point-to-point dial-up connection, a cell phone network, or another network and its networking hardware in various examples including routers
  • While computer systems configured as described above may be used to support the operation of the facility, those skilled in the art will readily appreciate that the facility may be implemented using devices of various types and configurations, and having various components. Elements of the facility may be described in the general context of computer-executable instructions, such as program modules, executed by one or more computers or other devices.
  • program modules include routines, programs, objects, components, data structures, and/or the like configured to perform particular tasks or implement particular abstract data types and may be encrypted.
  • the functionality of the program modules may be combined or distributed as desired in various examples.
  • display pages may be implemented in any of various ways, such as in C++ or as web pages in XML (Extensible Markup Language), HTML (HyperText Markup Language), JavaScript, AJAX (Asynchronous JavaScript and XML) techniques, or any other scripts or methods of creating displayable data, such as the Wireless Access Protocol (WAP).
  • WAP Wireless Access Protocol
  • the functionality of the program modules may be combined or distributed as desired in various embodiments, including cloud-based implementations, web applications, mobile applications for mobile devices, and so on.
  • aspects of the disclosed technology can be embodied in a special purpose computer or data processor, such as application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), graphics processing units (GPU), manycore processors, and so on, that is specifically programmed, configured, or constructed to perform one or more of the computer-executable instructions explained in detail herein. While aspects of the disclosed technology, such as certain functions, are described as being performed exclusively on a single device, the disclosed technology can also be practiced in distributed computing environments where functions or modules are shared among disparate processing devices, which are linked through a communications network such as a Local Area Network (LAN), Wide Area Network (WAN), or the internet. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.
  • ASIC application-specific integrated circuits
  • FPGA field-programmable gate arrays
  • GPU graphics processing units
  • manycore processors and so on
  • aspects of the disclosed technology may be stored or distributed on tangible computer-readable media, including magnetically or optically readable computer discs, hard-wired or preprogrammed chips (e.g., EEPROM semiconductor chips), nanotechnology memory, biological memory, or other computer-readable storage media.
  • computer-implemented instructions, data structures, screen displays, and other data under aspects of the disclosed technology may be distributed over the internet or over other networks (including wireless networks), on a propagated signal on a propagation medium (e.g., electromagnetic wave(s), sound wave, etc.) over a period of time, or they may be provided on any analog or digital network (packet switched, circuit switched, or other scheme).
  • the term computer-readable storage medium does not encompass signals (e.g., propagating signals) or transitory media.
  • FIG. 2 is a flow diagram illustrating the processing of a discover features component 121 in accordance with some embodiments of the disclosed technology.
  • the discover features component is invoked by the facility to identify novel feature vectors based on selected patient data.
  • the component receives physiological signal data, such as raw signal data received directly from signal recorder, previously-generated physiological signal from another device or site, etc.
  • physiological signals e.g., electrophysiological signals, biosignals
  • diagnostic and other purposes including, for example, activity trackers, echocardiograms, wide-band biopotential measuring equipment, electroencephalograms, electromyograms, electrooculography, galvanic skin response, heart rate monitors, magnetic resonance imaging, magnetoencephalograms, mechanomyograms, wearable technology devices (e.g., FITBITs), and so on.
  • activity trackers e.g., echocardiograms, wide-band biopotential measuring equipment, electroencephalograms, electromyograms, electrooculography, galvanic skin response, heart rate monitors, magnetic resonance imaging, magnetoencephalograms, mechanomyograms, wearable technology devices (e.g., FITBITs), and so on.
  • FITBITs wearable technology devices
  • the component invokes a process signal data component to process and transform the received signal data, which can produce multiple sets of data and transformed data.
  • the component sets a generation value equal to 0.
  • the component generates one or more feature generators by, for example, randomly generating an expression tree, randomly generating a set of weights for a neural network, randomly mutating one or more of a set of previously-generated feature generators, and so on.
  • the component invokes an apply feature generators component to apply the generated feature generators to one or more sets of the processed signal data to produce a set of feature vectors.
  • the component invokes an identify novel feature vectors component to identify the most novel feature vectors from among the group of feature vectors generated by the feature generators.
  • the component stores the feature generators that produced the identified feature vectors in, for example, a feature generator store.
  • the component increments the generation variable.
  • decision block 245 if the generation variable is greater than or equal to a generation threshold, then the component completes, else the component continues at block 250 .
  • the component may also use other stopping conditions, such as a number of generations of feature generators that do not produce at least a threshold number of novel feature vectors.
  • the component copies and mutates the identified feature generators and then loops back to block 225 to apply the mutated feature generators to one or more sets of the processed signal data.
  • the component may apply any type or types of mutations to a feature generator, such as applying multiple point mutations and/or a random recombination to one or more expression trees, randomly generating a set of connection weights for a neural network, and so on.
  • FIG. 3 is a flow diagram illustrating the processing of a process component 122 in accordance with some embodiments of the disclosed technology.
  • the process component is invoked by the discover features component to process and transform patient signal data.
  • the component loops through each signal (or data set) of a set of received signals (or set of data sets), each signal representing physiological data received from a patient.
  • the component pre-processes the received signal, such as applying one or more signal filters to the signal, performing a peak search on the data and discarding extraneous information, down-sampling the received signal, up-sampling the received signal, sub-sampling the received signal, converting an analog signal to a digital signal, converting image data to signal data, and so on.
  • the component stores the pre-processed signal in, for example, a patient data store.
  • the signal data may be stored anonymously (i.e., without explicitly or implicitly identifying the corresponding patient, etc.). However, different instances of signal data associated with the same patient may be associated with an anonymized unique identifier so that multiple signals from a single patient can be used in conjunction for training and diagnostic purposes.
  • the component extracts one or more values from the stored signal data.
  • the component stores the one or more extracted values.
  • the component identifies any transformations to be applied to the signal.
  • the facility may store an indication of a set of transformations or transformation functions (e.g., Fourier transforms, functions to apply to the signal, derivatives, partial derivatives, and so on) to apply to a particular signal.
  • the facility may randomly select, from among a catalog of transformations, one or more transformations to apply to the signal data.
  • the component loops through each of the transformations and applies the transformation to the signal.
  • the component applies the transformation to the signal (e.g., calculating the third derivative with respect to a particular variable, calculating the result of a composite function generated by applying one function to the signal data (i.e., a function representative of the signal data), etc.).
  • the component stores the transformed signal data in, for example, a patient data store.
  • the component extracts one or more values from the transformed signal data.
  • the component stores the one or more extracted values.
  • the component selects the next transformation and loops back to block 335 to apply the transformation to the signal data, else the component continues at block 365 .
  • the component selects the next signal and loops back to block 305 to process the next signal, else the component completes.
  • FIG. 4 is a flow diagram illustrating the processing of an apply feature generators component 123 in accordance with some embodiments of the disclosed technology.
  • the apply feature generators component is invoked by the discover features component 121 to apply a set of one or more feature generators to signal data, such as pre-processed and transformed signal data, modeled signal data, etc.
  • the component loops through each of a received set of feature generators and applies the feature generator to each signal in a received set of signal data.
  • the received signal data can include multiple sets of signal data for each of multiple patients, multiple transformations of that data, and so on.
  • the component loops through each of the signals to apply the feature generators to the signal data.
  • the component applies the currently-selected feature generator to the currently-selected signal data.
  • the component may apply the feature generator to each of a pre-processed version of the currently-selected data signal and any transformed version of that data.
  • the component “plugs in” or substitutes coefficients generated by modeled signal data into a feature generator with a set of variables to produce an output feature value.
  • the component can apply one or more elements of modeled signal data to a neural network to produce an output feature value.
  • the component stores the output value.
  • the component selects the next signal and loops back to block 420 to process the next signal, else the component continues at block 460 .
  • the component generates a feature vector that includes each of the generated feature values and stores the feature vector in association with the feature generator in, for example, a feature vector store.
  • the feature vector may comprise an array of features and a link to, or identifier of, the corresponding feature generator.
  • the component may also associate the feature vector with the signal data used to generate the feature vector.
  • the component selects the next feature generator and loops back to block 410 to process the feature generator, else the component completes.
  • FIG. 5 is a flow diagram illustrating the processing of an identify novel feature vectors component 124 in accordance with some embodiments of the disclosed technology.
  • the facility receives a set of feature vectors and, for each feature vector, information related to the corresponding feature generator, such as an identifier for the feature generator.
  • the component collects a comparison set of feature vectors that includes, for example, feature vectors generated by feature generators of earlier generations that were found to be novel and the feature vectors generated by a current generation of feature vectors. For example, the component can randomly select a set of novel feature vectors from a feature store.
  • a request to retrieve feature vectors includes upper and lower limits on the number of features values for each feature vector to be retrieved, such as no less than 50 (lower threshold) and no more than 5000 (upper threshold).
  • the component loops through each feature vector of a current generation of feature generators to determine how different each of their corresponding feature vectors is to each of the feature vectors of the comparison set of feature vectors.
  • the component loops through each feature vector of the comparison set of feature vectors to compare each feature vector to the feature vector of the currently-selected feature generator.
  • the component calculates a difference value between the currently-selected feature vector of the comparison set and the feature vector of the currently-selected feature generator.
  • the component can calculate a distance value between each of the feature vectors.
  • the component stores the calculated difference value.
  • the component selects the next feature vector and loops back to block 515 to process the feature vector, else the component continues at block 535 .
  • the component calculates a novelty score for the currently-selected feature generator based on the stored difference values, such as an average or maximum distance, and stores the novelty score in association with the feature generator (e.g., in a feature generator store).
  • the component selects the next feature generator and loops back to block 515 to process the feature generator, else the component continues at block 545 .
  • the component tests whether each of the feature vectors is novel, based on the calculated novelty scores, and identifies any corresponding feature generators.
  • decision block 550 if the novelty score for the currently-selected feature generator is greater than a novelty threshold, then the component continues at block 555 , else the component continues at block 560 .
  • the novelty threshold may be generated or determined in any number of ways, such as receiving a novelty threshold from a user, calculating a novelty threshold based on the set of novelty scores (e.g., average, average plus 25%, top n (where n is provided by a user or generated automatically by the facility), top tenth percentile), and so on.
  • the novelty threshold may change dynamically (e.g., from generation to generation) based on, for example, the number of generations without a new feature generator that exceeds the current novelty threshold to ensure that the facility is producing and testing new feature generators and corresponding features.
  • the component identifies the currently-selected feature vector as novel.
  • the component selects the next feature vector and loops back to block 545 to process the feature vector, else the component completes.
  • FIG. 6 is a flow diagram illustrating the processing of a discover genomes component 126 in accordance with some embodiments of the disclosed technology.
  • the facility invokes the discover genomes component to generate and analyze genomes for use by machine learning algorithms.
  • the facility initializes a generation variable equal to 0.
  • the component determines a number (n) of genomes to generate based on, for example, user input, system parameters, or randomly.
  • the component invokes a spawn genome component n time(s) to spawn the appropriate number of genomes.
  • the component invokes an identify high-performing genomes component to identify, from among the spawned genomes, the genomes that have a fitness score that exceeds a fitness threshold.
  • the component increments the generation variable.
  • decision block 660 if the generation variable is greater than or equal to a generation threshold, then processing of the component completes, else the component continues at block 670 .
  • the component mutates the high-performing genomes and then loops back to block 640 to identify high-performing genomes from among the mutated genomes (e.g., the mutated genomes having fitness scores that exceed a fitness threshold).
  • the component may mutate the genome by adding, changing, or removing (or any combination thereof) one or more elements of the variable-length genome. For example, the component may mutate one genome by replacing one feature with another feature and adding a new feature to the mutated genome.
  • the component may select a new machine learning algorithm to associate with the genome.
  • the component may also remove or mutate any irrelevant machine learning algorithm parameters and/or replace them with machine learning parameter values for the newly selected machine learning algorithm.
  • genomes may use sexual reproduction techniques as a form of mutation by randomly selecting elements of multiple genomes and combining those elements to form a new genome.
  • one or more elements of a genome may be configured so that they remain fixed (i.e., are not changed) during the evolutionary process described herein.
  • FIG. 7 is a flow diagram illustrating the processing of a spawn genome component 126 in accordance with some embodiments of the disclosed technology.
  • the discover genomes component 125 invokes the spawn genome component to generate a genome identifying any number of features, machine learning parameters, and/or machine learning algorithms.
  • the component identifies a set of available features, such as features referenced in one or more feature generator stores.
  • the component determines the number of features to include in the genome to be generated. For example, the component may determine the number of features to include in the genome to be generated based on user input, system parameters, or randomly.
  • the component randomly selects, from among the identified features, the determined number of features.
  • the component replaces correlated features from among the selected features with randomly selected features.
  • the component identifies a set of available machine learning parameters. For example, the component may identify, for each machine learning algorithm available to the facility, a set of parameters associated with that machine learning algorithm, which may be stored in a list or other data structure available to the component (e.g., a machine learning parameter store).
  • a genome may be generated for a single machine learning algorithm (or fixed set of machine learning algorithms). In this case, the component may identify only those machine learning parameters (or a proper subset thereof) that are associated with the single or fixed set of machine learning algorithms.
  • a genome may include an element that identifies a machine learning algorithm and that can be mutated.
  • the component may identify any or all machine learning parameters of machine learning algorithms that are within the scope of this mutation (i.e., the parameters of the machine learning algorithms that the genome and its descendants may be associated with to train models during the evolutionary process described herein).
  • the component determines the number of machine learning parameters to include in the genome to be generated. For example, the component may determine the number of machine learning parameters to include in the genome to be generated based on user input, system parameters, or randomly. For example, one genome may include each and every machine learning parameter associated with a particular machine learning algorithm or a set of machine learning algorithms while another genome includes only a proper subset of machine learning parameters associated with a particular machine learning algorithm.
  • the component randomly selects, from among the identified machine learning parameters, the determined number of machine learning parameters and assigns a value to the parameter based on any associated constraints, such as randomly selecting a value between a minimum value and a maximum value associated with the parameter.
  • the component stores each of the selected features and machine learning parameters in a genome data structure and then returns the genome data structure.
  • FIG. 8 is a flow diagram illustrating the processing of an identify high-performing genomes component 127 in accordance with some embodiments of the disclosed technology.
  • the discover genomes component invokes the identify high-performing genomes component to identify, from among a group of genomes, the genomes that have a corresponding fitness score that exceeds a fitness threshold (i.e., the genomes that are “high-performing”).
  • the component loops through a set of genomes provided to the component, such a first generation set of genomes, a mutated set of genomes, or some combination thereof.
  • the component trains one or more models using the currently-selected genome, including its features, machine learning parameters, and any specified machine learning algorithm.
  • the facility (or the machine learning algorithm itself) may provide a default value retrieved from, for example, a machine learning parameter store.
  • the component generates a validation or fitness score for the currently-selected genome by, for example, applying the trained model to a set of validation data and assessing the ability of the trained model to correctly identify or classify subjects from among the validation data.
  • the component stores the generated score(s) and/or an aggregation thereof produced for the trained models in association with the currently-selected genome.
  • the component selects the next genome and loops back to block 810 , else the component continues at block 860 .
  • the component assesses the score generated for each genome and selects the “best” genomes for mutation. In this example, the “best” genomes are those that produce validation or fitness scores that exceed a fitness threshold.
  • decision block 870 if the score generated for the currently-selected genome exceeds a fitness threshold, then the component continues at block 880 , else the component continues at block 890 .
  • the component flags the currently-selected genome for mutation.
  • the component may select genomes for mutation based on criteria other than, or in addition to, fitness scores. For example, the component may use novelty scores or other scores to select genomes for mutation.
  • the component may employ a tournament selection process in which a number of genomes are randomly selected from the population and the genome with the highest score among this “tournament” is selected for reproduction. In this example, if only low scoring genomes appear in a tournament, a low-scoring genome will be selected for reproduction.
  • block 890 if there are any genomes yet to be processed, then the component selects the next genome and loops back to block 860 , else the component returns the flagged genomes and processing of the component completes.

Abstract

A facility for identifying combinations of feature and machine learning algorithm parameters, where each combination can be combined with one or more machine learning algorithms to train a model, is disclosed. The facility evaluates each genome based on the ability of a model trained using that genome and a machine learning algorithm to produce accurate results when applied to a validation data set by, for example, generating a fitness or validation score for the trained model and the corresponding genome used to train the model. Genomes that produce fitness scores that exceed a fitness threshold are selected for mutation, mutated, and the process is repeated. These trained models can then be applied to new data to generate predictions for the underlying subject matter.

Description

    CROSS-REFERENCE TO RELATED APPLICATION(S)
  • This application is a continuation of U.S. patent application Ser. No. 15/653,441, filed on July 18, 2017, entitled “DISCOVERING GENOMES TO USE IN MACHINE LEARNING TECHNIQUES,” which is incorporated by reference herein in its entirety. This application is related to U.S. patent application Ser. No. 13/970,580, filed on Aug. 19, 2013, entitled “NON-INVASIVE METHOD AND SYSTEM FOR CHARACTERIZING CARDIOVASCULAR SYSTEMS,” now U.S. Pat. No. 9,289,150; U.S. patent application Ser. No. 15/061,090, filed on Mar. 4, 2016, entitled “NON-INVASIVE METHOD AND SYSTEM FOR CHARACTERIZING CARDIOVASCULAR SYSTEMS,” now U.S. Pat. No. 9,655,536 U.S. patent application Ser. No. 15/588,148, filed on May 5, 2017, entitled “NON-INVASIVE METHOD AND SYSTEM FOR CHARACTERIZING CARDIOVASCULAR SYSTEMS,” now U.S. Pat. No. 9,968,275; U.S. patent application Ser. No. 13/605,364, filed on Sep. 6, 2012, entitled “SYSTEM AND METHOD FOR EVALUATING AN ELECTROPHYSIOLOGICAL SIGNAL,” now U.S. Pat. No. 8,923,958; U.S. patent application Ser. No. 13/970,582, filed on Aug. 19, 2013, entitled “NON-INVASIVE METHOD AND SYSTEM FOR CHARACTERIZING CARDIOVASCULAR SYSTEMS FOR ALL-CAUSE MORTALITY AND SUDDEN CARDIAC DEATH RISK,” now U.S. Pat. No. 9,408,543; U.S. patent application Ser. No. 15/207,214, filed on Jul. 11, 2016, entitled “NON-INVASIVE METHOD AND SYSTEM FOR CHARACTERIZING CARDIOVASCULAR SYSTEMS FOR ALL-CAUSE MORTALITY AND SUDDEN CARDIAC DEATH RISK,” now U.S. Pat. No. 9,955,883; U.S. patent application Ser. No. 14/295,615, filed on Jun. 4, 2014, entitled “NONINVASIVE ELECTROCARDIOGRAPHIC METHOD FOR ESTIMATING MAMMALIAN CARDIAC CHAMBER SIZE AND MECHANICAL FUNCTION,” now U.S. Pat. No. 9,737,229; U.S. patent application Ser. No. 14/077,993, filed on Nov. 12, 2013, entitled “NONINVASIVE ELECTROCARDIOGRAPHIC METHOD FOR ESTIMATING MAMMALIAN CARDIAC CHAMBER SIZE AND MECHANICAL FUNCTION,” now U.S. Pat. No. 10,039,468; U.S. patent application Ser. No. 14/596,541, filed on Jan. 14, 2015, entitled “NONINVASIVE METHOD FOR ESTIMATING GLUCOSE, GLYCOSYLATED HEMOGLOBIN AND OTHER BLOOD CONSTITUENTS,” now U.S. Pat. No. 9,597,021; U.S. patent application Ser. No. 15/460,341, filed on Mar. 16, 2017, entitled “NONINVASIVE METHOD FOR ESTIMATING GLUCOSE, GLYCOSYLATED HEMOGLOBIN AND OTHER BLOOD CONSTITUENTS,” now U.S. Pat. No. 10,765,350; U.S. patent application Ser. No. 14/620,388, filed on Feb. 12, 2015, entitled “METHOD AND SYSTEM FOR CHARACTERIZING CARDIOVASCULAR SYSTEMS FROM SINGLE CHANNEL DATA,” now U.S. patent application Ser. No. 14/620,388; U.S. patent application Ser. No. 15/192,639, filed on Jun. 24, 2016, entitled “METHODS AND SYSTEMS USING MATHEMATICAL ANALYSIS AND MACHINE LEARNING TO DIAGNOSE DISEASE,” now U.S. Pat. No. 9,910,964; U.S. patent application Ser. No. 15/248,838, filed on Aug. 26, 2016, entitled “BIOSIGNAL ACQUISITION DEVICE,” now U.S. Pat. No. 10,542,897; U.S. Provisional Patent Application No. 62/397,895, filed on Sep. 21, 2016, entitled “GRAPHICAL USER INTERFACE FOR CARDIAC PHASE-SPACE TOMOGRAPHY,” U.S. patent application Ser. No. 15/633,330, filed Jun. 26, 2017, entitled “NON-INVASIVE METHOD AND SYSTEM FOR MEASURING MYOCARDIAL ISCHEMIA, STENOSIS IDENTIFICATION, LOCALIZATION AND FRACTIONAL FLOW RESERVE ESTIMATION,” now U.S. Pat. No. 10,362,950; and U.S. patent application Ser. No. 15/653,433 (Attorney Docket No. 124077-8001.US00), filed on Jul. 18, 2017, entitled “DISCOVERING NOVEL FEATURES TO USE IN MACHINE LEARNING TECHNIQUES, SUCH AS MACHINE LEARNING TECHNIQUES FOR DIAGNOSING MEDICAL CONDITIONS.” Each of the above-identified applications and issued patents is hereby incorporated by reference in its entirety.
  • BACKGROUND
  • Machine learning techniques predict outcomes based on sets of input data. For example, machine learning techniques are being used to predict weather patterns, geological activity, provide medical diagnoses, and so on. Machine learning techniques rely on a set of features generated using a training set of data (i.e., a data set of observations, in each of which an outcome to be predicted is known), each of which represents some measurable aspect of observed data, to generate and tune one or more predictive models. For example, observed signals (e.g., heartbeat signals from a number of subjects) may be analyzed to collect frequency, average values, and other statistical information about these signals. A machine learning technique may use these features to generate and tune a model that relates these features to one or more conditions, such as some form of cardiovascular disease (CVD), including coronary artery disease (CAD), and then apply that model to data sources with unknown outcomes, such as an undiagnosed patient or future weather patterns, and so on. Conventionally, these features are manually selected and combined by data scientists working with domain experts.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • FIG. 1 is a block diagram illustrating an environment in which the facility operates in some embodiments.
  • FIG. 2 is a flow diagram illustrating the processing of a discover features component in some embodiments.
  • FIG. 3 is a flow diagram illustrating the processing of a process component in some embodiments.
  • FIG. 4 is a flow diagram illustrating the processing of an apply feature generators component in some embodiments.
  • FIG. 5 is a flow diagram illustrating the processing of an identify novel feature vectors component in some embodiments.
  • FIG. 6 is a flow diagram illustrating the processing of a discover genomes component in accordance with some embodiments.
  • FIG. 7 is a flow diagram illustrating the processing of a spawn genome component in accordance with some embodiments.
  • FIG. 8 is a flow diagram illustrating the processing of an identify high-performing genomes component in accordance with some embodiments.
  • DETAILED DESCRIPTION
  • Because machine learning techniques rely on features and/or combinations of features, the process of feature selection and combination typically is an important part of a machine learning process. Moreover, because a large number of diverse machine learning algorithms exist (e.g., decision trees, artificial neural networks (ANNs), deep ANNs, genetic (and meta-genetic) algorithms, and so on), the choice of algorithm and any associated parameters can also be important. For example, different machine learning algorithms (or family of machine learn algorithms) may be best suited for different types of data and/or the types of predictions to be made. Furthermore, different machine learning algorithms may present various tradeoffs with respect to resources (e.g., memory, processor utilization), speed, accuracy, and so on. Typically, models are trained using machine learning algorithms, features, and parameters selected by individuals based on the preferences of those individuals and/or criteria specified by those individuals. The inventors have recognized that it can be expensive and time-consuming manually to identify features, machine learning algorithms, and corresponding parameters and even more difficult to produce features, machine learning algorithms, and corresponding parameters that produce more accurate models and, therefore, more accurate predictions. Accordingly, the inventors have conceived and reduced to practice a facility that performs automatic discovery of combinations of features, machine learning algorithms, and/or machine learning parameters.
  • In some embodiments, the facility operates as part of a machine learning pipeline that constructs and evaluates predictive models, such as those for disease diagnosis, based on time-series and/or other signals, such as physiological signals. The machine learning process uses features to identify patterns within a training set of data and, based on these patterns, generates predictive models. These predictive models can be validated using validation data sets (i.e., data sets for which an outcome is known but that were not used to train the model) and applied to new input data in order to predict outcomes from the input data, such as providing a diagnosis for a medical condition, etc. As new data and new features are produced or acquired, the machine learning process improves upon the predictive capabilities of these models by incorporating new features and, in some cases, discarding others, such as those that are determined to be too similar to other features.
  • In particular, the facility seeks to identify combinations of features and machine learning algorithm parameters where each combination can be used to train one or more models. A combination of features and/or machine learning parameters is sometimes referred herein to as a “genome.” The facility evaluates each genome based on the ability of a model trained using a machine learning algorithm and that genome to produce accurate results when applied to a validation data set by, for example, generating a fitness or validation score for the trained model and the corresponding genome used to train the model. In some cases, the facility uses the validation score as a fitness score while in other cases the validation score is an element of a fitness score (e.g., fitness score=training score+validation score). In some cases, multiple models may be trained using a genome and the resulting fitness scores can be aggregated to generate an aggregated fitness score for the genome.
  • By way of example, the facility for identifying combinations of features and machine learning algorithm parameters can be used for a medical diagnosis predictive modeling task. In this example, the facility receives, for a number of patients or subjects, one or more sets of physiological data that relate to some type of physiological output or condition of the patient over a period of time (e.g., less than a second, on the order of a few seconds, about ten seconds, about 30 seconds and up to about five minutes, about an hour or more, etc.), such as electroencephalograms, and so on. These data may be received in real-time or near real-time, concurrent or nearly concurrent with the operation of the facility, or they may be received at an earlier time. In some cases, the facility discards certain portions of the signal to ensure that the signals from each patient commence at a stable and consistent initial condition. Furthermore, the data may be normalized to remove potentially misleading information. For example, the facility can normalize the amplitude of signal data (e.g., transforming to a z-score), to account for variations in signal strength caused by sensor contact or other non-physiological data. As another example, in the case of a cardiac signal, the facility can perform a peak search and discard any data before a first heartbeat identified in the signal and after a last heartbeat identified in the signal.
  • In some embodiments, the facility applies a set of feature generators to a set of signals to generate, for each combination of a signal and a feature generator, a feature value for the signal. Thus, each feature value is representative of some property of the underlying signal data. In one example, the facility receives patient data for each of 1000 patients and applies one or more feature generators to the data to generate, for each application of a feature generator to the data of a single patient, a feature value (or set of feature values). The facility collects the feature values generated by a single feature generator in a “feature vector,” such that the feature vector stores one feature value per patient. Once the feature vectors are generated, they can be compared to determine how different each is relative to each of the other feature vectors. The facility computes a distance metric for each feature vector to assess the novelty of the corresponding feature generator. Based on the assessed novelty, the facility (1) provides the feature generators that produced the novel feature vectors to the machine learning process for the purpose of basing new predictive models on the provided feature generators and (2) modifies these feature generators to create a new generation of feature generators. The facility repeats this evolutionary process to identify even more novel features for use by the machine learning process.
  • In some embodiments, for each received set of data, the facility computes or identifies separate sets of one or more values from the data. For example, in the case of data generated as part of an electrocardiogram, the facility identifies global and local maxima and minima within the data, computes frequency/period information from the data, calculates average values of the data over a certain period of time (e.g., the average duration and values generated during a QRS complex), and so on. In some cases, the facility transforms the received data and extracts sets of one or more values from the transformed data. The facility can transform received signal data in any number of ways, such as taking one or more (successive) derivatives of the data, taking one or more partial derivatives of the data, integrating the data, calculating the gradient of the data, applying a function to the data, applying a Fourier transform, applying linear or matrix transformations, generating topology metrics/features, generating computational geometry metrics/features, generating differential manifold metrics/features, and so on. In this manner, the facility generates multiple perspectives of the data in order to yield a diverse set of features. While these transformations are provided by way of example, one of ordinary skill will recognize that the data can be transformed in any number of ways.
  • In one example, the facility receives multiple input signals (e.g., input signals collected by different electrodes or leads connected to a patient, multimodal signals, such as signals from leads of wide-band biopotential measuring equipment and a channel of SpO2 (blood oxygen saturation), and so on) and/or transformed signals and extracts values from the signal data by computing, for each signal, an average value of the signal over the sampling period. In this example, four signals per patient are represented, although one of ordinary skill in the art will recognize that any number of signals may be monitored and/or received for processing and further analysis by the facility. Thus, in this example, the extracted data of each patient can be represented as a set of these average values over time, such as:
  • TABLE 1
    Patient A B C D
    1 0.24 0 0 30
    2 0.2 0.6 4.2 5
    ...
    n .32 2 4 .02

    Table 1 represents a set of average signal values (A, B, C, and D) for each of n patients. Although average values have been used here, one of ordinary skill in the art will recognize that any type of data can be extracted or computed from the underlying data signals, such as the amount of time that a signal exceeded a threshold value, the values for one signal while the value of another signal exceeded a threshold value, and so on.
  • In some embodiments, after data have been extracted from the received signal, the facility applies one or more feature generators to the received or generated data, such as the extracted data, the raw or preprocessed signal data, the transformed data, and so on. A feature generator receives as input at least a portion or representation of the signal data and produces a corresponding output value (or set of values) (i.e., a “feature”). One set of feature generators includes the following equations:
  • F 1 = A + C - D , ( Eq 1 ) F 2 = A * S ( 4 ) * B D + C + D , and ( Eq 2 ) F 3 = S ( 1 ) * D , ( Eq 3 )
  • where each of A, B, C, and D represents a value extracted from a specific patient's data and S(t) represents, for each signal, the value of the signal at time t. In Eq 1, for example, F1 represents the name of the feature while the equation A+C−D represents the corresponding feature generator. In some cases, the facility employs composite feature generators in which one feature generator serves as an input to another feature generator, such as:
  • F 4 = F 1 * F 2 F 3 3 + . 0 57 ( Eq 4 )
  • In this example, the facility applies feature generators to the extracted data of each patient represented in Table 1 to generate, for each feature generator, a feature vector of three values (one for each patient), such as those represented in Table 2 below:
  • TABLE 2
    Patient F1 F2 F3
    1 −29.76 5.48 905.83
    2 −0.6 6.67 9.57
    ...
    n 4.3 185.74 0.04
  • In this example, the facility has applied each feature generator F1, F2, and F3 to the extracted data shown in Table 1 to generate, for each feature generator, a corresponding feature vector that includes a value for each patient. For example, the feature vector generated by applying feature generator Fl to the extracted data includes a value of −29.76 for Patient 1, a value of −0.6 for patient 2, and so on. Thus, each feature vector represents, for a specific feature generator, a signature (not necessarily unique) for the corresponding feature generator based on at least a portion of each patient's physiological data (i.e., the patients represented in the physiological data to which the feature generators were applied). In some examples, feature generators are expressed using different structures or models, such as expression trees, neural networks, etc. One of ordinary skill in the art will recognize that the facility may employ any number of feature generators and any number of sets of physiological data (or portions thereof) in the generation of feature vectors. In some embodiments, the facility randomly selects a number of previously-generated feature generators for use in generating feature vectors rather than employing each and every available feature generator. In some embodiments, the facility creates and/or modifies feature generators by, for example, randomly generating expression trees, randomly assigning weights to connections within a neural network, and so on.
  • In some embodiments, after the facility generates a number of feature vectors, the facility employs some form of novelty search to identify the most “novel” feature vectors among the generated feature vectors. Novelty corresponds to how different a particular feature vector is from each of a comparison set of other feature vectors (made up of any feature vectors generated by the facility during a current iteration and feature vectors produced by feature generators selected in any earlier iteration); the greater the difference from the feature vectors of the comparison set, the greater the novelty. The facility uses a form of distance as a measure of novelty (i.e., how “far” each feature vector is from the other feature vectors). In this case, for each generated feature vector, the facility calculates the distance between that feature vector and each of the other generated feature vectors and performs an aggregation of the generated distance values, such as calculating an average or mean (e.g., arithmetic, geometric, harmonic, etc.) distance value for the feature vector, or a total (sum) distance between the feature vector and each of the other generated feature vectors, identifying a mode distance value, a median distance value, a maximum distance value for the feature vector, and so on. For example, using the feature vectors of Table 2 (for patients 1, 2, and n), the distances for each set of feature vectors could be calculated as such:
  • F 1 - F 2 distance : ( - 2 9.76 - 5.48 ) 2 + ( - 0.6 - 6.67 ) 2 + ( 4.3 - 1 8 5 . 7 4 ) 2 = 1 8 4 . 9 7. F 1 - F 3 distance : ( - 2 9.76 - 905.83 ) 2 + ( - 0.6 - 9.57 ) 2 + ( 4.3 - 0.04 ) 2 = 93 6 . 2 3 F 2 - F 3 distance : ( 5.48 - 905.83 ) 2 + ( 6.67 - 9.57 ) 2 + ( 185.74 - 0.04 ) 2 = 91 9 . 7 0 .
  • In this example, the total Euclidean distance between each of the feature vectors has been calculated as a means for calculating a difference between each of two vectors. In addition to the feature vectors generated by a current set (i.e., a current generation) of feature generators, the facility includes feature vectors produced by feature generators selected in an earlier generation. In some examples, the facility applies a weight, such as a randomly generated weight, to each of the feature vectors and/or normalizes each set of feature vectors prior to comparison. Thus, the distance measurements for each of the feature vectors in this example are as follows:
  • TABLE 3
    Feature Distance Distance Distance Average MAX
    Generator to F1 to F2 to F3 Distance Distance
    F1 184.97 936.23 560.60 936.23
    F2 184.97 919.70 552.34 919.70
    F3 936.23 919.70 927.97 936.23
  • In this example, the facility identifies the most “novel” feature vectors based on the calculated distances, which act as a “novelty score” or “fitness score” for each of the feature vectors. The facility identifies the feature vectors with the greatest average distance to other vectors (e.g., the feature vector generated by F3), the feature vectors with the greatest MAX distance (e.g., the feature vectors generated by F1 and F3), and so on. In some examples, the number of novel feature vectors identified is fixed (or capped) at a predetermined number, such as five, ten, 100, 500, etc. In other examples, the number of novel feature vectors to be identified is determined dynamically, such as the top 10% of analyzed feature vectors based on novelty scores, any feature vectors having a novelty scores that is more than a predetermined number of standard deviations beyond a mean novelty score for the analyzed feature vectors, and so on. The feature generators that produced each of these identified novel feature vectors can then be added to the set of features available for use as inputs to models constructed and evaluated by the machine learning pipeline. Those models can be applied to patient data for, e.g., diagnostic, predictive, therapeutic, or other analytic, scientific, health-related or other purposes.
  • In some embodiments, in addition to providing the feature generators used to generate the identified novel feature vectors for use by the machine learning process, the facility randomly mutates or modifies the feature generators used to generate the identified novel feature vectors. Each mutation effects some change in the corresponding feature generator and creates a new version of the feature generator that can be used to contribute to a new generation of feature generators. The facility uses this new feature generator to generate new feature vectors, and then assesses the novelty of the new feature vectors. Moreover, the corresponding feature generator can be further mutated to continue this process of feature vector and feature generation creation. For example, a feature generator expressed in the form of an equation, such as F10=A+C−D, can be mutated by randomly selecting one or more element(s) of the equation and replacing the selected element(s) with other elements (e.g., randomly selected elements). In this example, the equation can be changed by replacing A with B to create F11=B+C−D or replacing C−D with
  • C - B 2 3
  • to create
  • F 1 1 = B + C - B 2 3 .
  • In this case, the subscripted 0 and 1 have been included to represent a generational marker or count for each of the feature generators. In other words, F10 represents F1 above (Eq 1) at generation 0 (i.e., the first generation), F11 represents a mutated version of F1 at generation 1 (i.e., the second generation), and so on. In some cases, an earlier generation (or a transformation thereof) is included as an element in subsequent generations, such as F21=√{square root over (F20)}+C2 or √{square root over (F2n−1)}+C2 (n≠0).
  • In some embodiments, the facility obtains features in different ways. For example, the facility may receive from a user, such as a domain expert, a set of features (and corresponding feature generators) that the user has identified as being optimal and/or that the user desires to be tested. As another example, the features may be editorially selected from one or more feature stores. In some cases, features automatically generated by the facility can be combined with other features to create various hybrid features. Even features of unknown provenance may be used.
  • In some embodiments, the facility identifies genomes to train models, identifies, from among these genomes, the “best” (highest rated) genomes, and mutates the identified genomes to produce even more genomes that can be used to train models. After using a genome to train one or more models, the facility applies each trained model to a validation data set so that the trained model can be scored (e.g., how well does the trained model correctly identify and/or classify subjects in the underlying validation data set). The facility mutates the genomes that produce the best results (e.g., have the highest validation or fitness scores), trains new models using these mutated genomes, and repeats this process until one or more termination criteria are met (e.g., a predetermined number of generations, no additional high scoring (higher than a predetermined or dynamically generated threshold) genomes are generated during a predetermined or dynamically number (e.g., 1, 5, 8, 17, etc.) of previous generations, a combination thereof, etc.).
  • In some embodiments, the facility uses previously identified or generated genomes as a first set of genomes (i.e., a first generation) from which to discover genomes for machine learning algorithms. In other examples, the facility automatically generates a first generation of genomes by, for each genome, randomly (with or without replacement) selecting one or more feature vectors from one or more previously generated sets of feature vectors (e.g., a feature vector produced by applying a feature generator to a set of training data). A genome may also include one or more machine learning algorithm parameters to the machine learning algorithm, such as the number of predictors (e.g., regressors, classifiers, the number and/or the maximum number of decision trees to use for a machine learning algorithm, etc.) to use for an underlying ensemble method associated with the algorithm, a maximum depth for a machine learning algorithm (e.g., maximum depth for decision trees), and so on. In the event that the genome is configured to be used with one specific machine learning algorithm, the genome can be configured to define a value for each machine learning parameter associated with that machine learning algorithm. In other cases, one of the elements of the genome selects among different machine learning algorithms and may be mutated so that the genome and its corresponding parameter values are used with different machine learning algorithms to train models over the evolutionary process. For example, during a first generation, a genome may identify a machine learning algorithm that relies on decision trees while a mutated version of that same genome identifies a machine learning algorithm that uses one or more support vector machines, linear models, etc. In these cases, the genome may specify a modeling parameter for each and every machine learning algorithm that may be combined with the genome to train a model. Thus, a single genome may include machine learning parameters for multiple machine learning algorithms. However, a genome need not include each and every modeling parameter for a corresponding machine learning algorithm. In the event that a model is to be trained using a particular machine learning algorithm and a genome that does not include a value for a machine learning parameter of that machine learning algorithm, the facility can retrieve a default value for these parameters from, for example, a machine learning parameter store.
  • For example, a set of genomes may be represented as:
  • TABLE 4
    G11 MLA = 4 F23 F78798 F32 F55 F453 F234
    G21 MLA = 9 F9701 F223 F1 F63 F349 P9:1 = 7
    G31 MLA = 2 F823 F525 F732 F525 F125
    G41 MLA = 6 F597 F135 F404 F31 P6:1 = 5 P6:2 = 150
    . . .
    G201 MLA = 1 F43 F65 P1:1 = 8 P1:2 = 218 P1:3 = 0.3

    where each row corresponds to a different genome (named in the first column from the left) from among a first generation of selected or generated genomes and identifies a machine learning algorithm (“MLA”; second column from the left) to use to train a model using the genome, such as an index into a machine learning algorithm store. For example, genome G31 specifies a machine learning algorithm corresponding to index 2 in a machine learning algorithm store (MLA=2). In this example, each non-bolded region (to the right of the second column) identifies a different feature. A genome can also include a corresponding feature generator or a reference to the corresponding feature generator, such as a link to feature generator store. As discussed above, these features may be generated automatically by the facility and/or retrieved from another source.
  • Furthermore, each bolded region in Table 4 represents a value for a particular machine learning parameter. In this example set of genomes, machine learning parameters are represented by an indicator or reference (e.g., P6:1) followed by an equals sign and a corresponding value. For example, machine learning algorithm parameter P6:1 has a corresponding value of 8 in genome G201. In this example set of genomes, each machine learning parameter is presented as an index into a two-dimensional array, such that “P6:1” represents the “first” machine learning parameter of the “sixth” machine learning algorithm (i.e., the machine learning parameter with an index of 1 for the machine learning algorithm with an index of 6). As discussed above, a genome may specify values for any or all machine learning parameters that may be used to training a model using the genome (or a mutated version of that genome). Moreover, as is clear from Table 4, genomes may be of varying length. For example, genome G11 includes values for six features and zero machine learning parameters while gnome G21 includes values for two features and three machine learning parameters. Accordingly, the facility may employ variable-length genomes in the machine learning processes.
  • In some embodiments, the facility may filter features from within genomes and/or filters genomes themselves to avoid redundancy among each. In order to filter features and/or genomes, the facility generates correlation values for each pair and discards one item of the pair. To identify and filter correlated features from a genome, the facility generates, for each of the features, a feature vector by applying a feature generator associated with the feature to a training set of data to produce a set of values. The facility compares each of the generated feature vectors to the other generated feature vectors to determine whether any of the feature vectors are “highly” correlated (i.e., not “novel” within the selected set of feature vectors). For example, the component may calculate a distance value for each of the generated feature vectors relative to the other feature vectors (as discussed above with respect to identifying novel feature generators) and, if the distance between any pair (set of two) is less than or equal to a distance threshold (i.e., “highly” correlated or not “novel”), discard a feature corresponding to one of the pair of feature vectors. Moreover, the facility may replace the discarded feature with a new feature, such as a randomly-selected feature. Similarly, the facility may identify and discard redundant genomes by generating, for each feature of the genome, a feature vector, calculating distance metrics for each pair (set of two) of genomes based on the generated feature vectors, and identifying pairs of genomes whose calculated distances do not exceed a genome distance threshold. For each identified pair of genomes, the facility may discard or mutate one or both of the genomes to reduce correlation and redundancy among a group of genomes. Although distance is used in this example as a metric for determining a correlation between two vectors or sets of vectors, one of ordinary skill in the art will recognize that correlations between two or sets of vectors can be calculated in other ways, such as normalized cross-correlation, and so on. In some embodiments, the facility may employ additional or other techniques to filter genomes, such as generating a graph where features represent vertices in the graph which are connected via edges in the graph. An edge between two features is generated if, for example, a correlation value between the two features exceeds a predetermined correlation threshold and/or the distance between the two features is less than a predetermined distance threshold. Once the graph is generated, the facility removes connected vertices (features) from the graph until no edges remain in the graph (an edge being removed when a connected vertex is removed) and selects the remaining non-connected vertices (features) for inclusion in the “filtered” genome. In some cases, the facility may randomly select connected vertices for removal. Moreover, the facility may perform this process multiple times for a set of vertices (features) and then select a preferred “filtered” genome, such as the genome with the most or least vertices (features) removed.
  • In order to test the fitness or validity of each genome, the facility trains at least one model using the features, machine learning parameters, and/or machine learning algorithm(s) of that genome. For example, the facility can use AdaBoost (“Adaptive Boosting”) techniques to train a model using the corresponding features, machine learning parameters, machine learning algorithm, and a training set of data. However, one of ordinary skill in the art will recognize that many different techniques can be used to train one or more models given a genome or a set of genomes. After the model is trained, the facility applies the trained model to one or more sets of validation data to assess how well the trained model identifies and/or classifies previously-identified or classified subjects within the validation data set. For example, a genome may be generated to train models to identify patients represented in a data set who are likely to have diabetes. Once a model is trained using one of these genomes, the trained model can be applied to a validation set of data to determine a validation score that reflects how well the trained model identifies patients from the validation set that are known to have or now have diabetes; scoring (adding) one “point” for every correct determination (e.g. true positives and true negatives) and losing (subtracting) one “point” for every incorrect determination (e.g., false positives and false negatives). Thus, an overall score for the trained model can be determined based on how many “points” the trained model scores when applied to one or more sets of validation data. One of ordinary skill in the art will recognize that several techniques may be used to generate a fitness score for a trained model, such as calculating the area under a corresponding receiver operating characteristic (ROC) curve, calculating a mean squared prediction error, f scores, sensitivity, specificity, negative and positive predictive values, diagnostic odds ratios, and so on. In this example, where a single machine learning algorithm is trained using the genome, the generated fitness score may be similarly attributed to the genome. In other case, the genome may be used to train multiple machine learning algorithms and each of those trained machine learning algorithms may be applied to multiple validation sets to produce, for each genome used to train machine algorithms, multiple fitness scores. In these cases, the facility generates a fitness score for the corresponding genome by aggregating each of the fitness scores generated for the machine learning algorithms trained using the genome. In some cases, the generated fitness scores may be aggregated and/or filtered prior to aggregation.
  • In some embodiments, after the facility has produced fitness scores for each of the genomes, the facility identifies the “best” genomes based on these fitness scores. For example, the facility can establish a fitness threshold based on the produced fitness scores and identify the “best” genomes as those genomes whose resulting fitness scores exceed the fitness threshold. The fitness threshold may be generated or determined in any number of ways, such as receiving a fitness threshold from a user, calculating a fitness threshold based on the set of fitness scores (e.g., average, average plus 15%, top fifteen, top n-th percentile (where n is provided by a user or generated automatically by the facility)), and so on. The facility then stores each of the genomes in association with their corresponding fitness scores and selects the genomes identified as “best” for mutation (i.e., the genomes having a fitness score that exceeds a fitness threshold).
  • In some embodiments, the facility mutates a genome by adding, removing, or changing any one or more of the feature vectors or machine learning parameters of the genome. For example, Table 5 below represents a number of mutations to the genomes represented above in Table 4.
  • TABLE 5
    G12 MLA = 5 F23 F78798 F32
    Figure US20220093215A1-20220324-P00001
    Figure US20220093215A1-20220324-P00002
    Figure US20220093215A1-20220324-P00003
    G22 MLA = 9
    Figure US20220093215A1-20220324-P00004
    F223 F1 F63 F349 F584 P9:1 = 12
    G42 MLA = 6 F597 F135 F404 F31 F24 F982 P6:1 = 5 P6:2 = 150
    . . .
    G202 MLA = 1 F43 F65*F14 P1:1 = 8 P1:2 = 218 P1:3 = 0.3

    In this example, each row corresponds to a different genome (named in the first column from the left) from among a second generation of genomes selected for mutation. In this example, based on its low fitness score, the facility did not select genome G31 for mutation and, therefore, Table 5 does not include a corresponding entry for a mutated version of this genome. Moreover, genome G11 has been mutated (represented as G12) by removing three feature vectors (represented with a strikethrough) and changing the references machine learning algorithm index from 4 to 5. Furthermore, the facility has mutated genome G21 by 1) removing feature vector F9701, 2) adding feature vector F584, and 3) adjusting machine learning parameter P91 from 7 to 12; genome G41 by adding features F24 and F982; and genome Gn1 by multiplying values generated by F65 by values generated by F14. These mutated genomes can then be used to train one or more machine learning algorithms, scored by applying the trained machine learning algorithm to one or more validation data sets, selected for mutation, mutated, and so on. The facility performs this process until a termination point is reached, such as when a predetermined number of generations has been produced (e.g., six, 30, 100,000, etc.), and so on.
  • FIG. 1 is a block diagram illustrating an environment 100 in which the facility operates in accordance with some embodiments of the disclosed technology. In this example, environment 100 includes service provider 110, signal recorder 140 (e.g., a wide-band biopotential measuring equipment), data providers 150, patient 160, and network 160. In this example, service provider includes facility 120, which includes discover features component 121, process component 122, apply feature generators component 123, identify novel feature vectors component 124, discover genomes component 125, spawn genome component 126, identify high-performing genomes component 127, patient data store 130, model store 131, feature vector store 132, and feature generator store 133. Discover features component 121 is invoked by the facility to identify and mutate feature generators based on received data. Process component 122 is invoked by the discover features component 121 to process and transform patient signal data, such as raw signal data from signal recorder 140 (e.g., one or more measurement devices and/or systems used to collect the underlying data such as wide-band biopotential measuring equipment, etc.), 3-D image data, etc. Apply feature generators component 123 is invoked by the discover features component to apply a set of one or more feature generators to the processed and transformed patient signal data. Identify novel feature vectors component 124 is invoked by the discover features component to identify the most novel feature vectors from among a group of feature vectors generated by, for example, one or more feature generators. Discover genomes component 125 is invoked by the facility 120 to generate, analyze, and mutate genomes for use by machine learning algorithms. Spawn genome component 126 is invoked by the discover genomes component to generate a genome comprising any number of feature vectors and/or machine learning parameters. Identify high-performing genomes component 127 is invoked by the discover genomes component to identify, from among a group of genomes, the genomes that have a corresponding fitness score that exceeds a fitness threshold. Patient data store 130 includes physiological patient data, such as raw physiological data (including but not limited to data obtained via, e.g., signal recorder 140), transformed physiological data, biographical information, demographic information, etc. These data may be stored anonymously to protect the privacy of each of the corresponding patients and may be processed and encrypted to ensure that its transmission and storage complies with any governing laws and their implementing regulations, such as the U.S. Health Insurance Portability and Accountability Act of 1996 (as amended), the European Data Protection Directive, the Canadian Personal Information Protection and Electronics Documents Act, the Australian Privacy Act of 1988, Japan's Personal Information Protection Act of 2015 (as amended), state and provincial laws and regulations, and so on. Model store 131 stores information about models generated by applying machine learning techniques to training data, such as the machine learning techniques described in Christopher M. Bishop, Pattern Recognition and Machine Learning (2006) (Library of Congress Control Number: 2006922522; ISBN-10: 0-387-31073-8), which is herein incorporated by reference in its entirety. Feature vector store 132 stores sets of feature vectors generated by applying one or more feature generators to a set of physiological data. Feature generator store 133 stores sets of feature generators that can be applied to patient physiological data and can include multiple generations of feature generators. Genome store 134 stores generated and/or mutated genomes created by the facility and/or other sources. Machine learning parameters store 135 stores, for each of a number of machine learning algorithms, a set of parameters that can serve as inputs to that machine learning algorithm and additional information related to the parameter, such as a maximum value for a corresponding parameter, a minimum value for a corresponding parameter, a default value for a corresponding parameter, etc. Machine learning algorithm store 133 stores the logic for each of a number of machine learning algorithms, each of which can be selectively trained and validated by the facility. In this example, a signal recorder 140 is connected to patient 160 via electrodes 145 and includes facility 120, one or more output devices 142, such as a monitor, printer, speaker, etc., and one or more input devices 144, such as settings controls, keyboard, biometric data reader, etc. Thus, as in this example, the facility can be configured to operate remotely from a patient and other diagnostics equipment and/or in conjunction with or part of the diagnostics equipment such as a wide-band biopotential measuring equipment (i.e., any device configured to capture unfiltered electrophysiological signals, including those with spectral components that are not altered). Accordingly, the facility can be configured to operate in real-time with the reading of physiological data and/or can be applied to previously recorded physiological data. Data providers 150, each of which includes a data store 152, can provide information for analysis or use by the facility such as, physiological patient data recorded off-site (e.g., at a hospital or clinic without access to a facility on premises, third-party data providers, etc.), feature vectors and/or feature generators produced or generated elsewhere, and so on. Network 170 represents communications links over which the various elements of environment 100 may communicate, such as the internet, a local area network, and so on.
  • In various examples, these computer systems and other devices can include server computer systems, desktop computer systems, laptop computer systems, netbooks, tablets, mobile phones, personal digital assistants, televisions, cameras, automobile computers, electronic media players, appliances, wearable devices, other hardware, and/or the like. In some embodiments, the facility 120 may operate on specific-purpose computing systems, such as wide-band biopotential measuring equipment (or any device configured to capture unfiltered electrophysiological signals, including electrophysiological signals with unaltered spectral components), electroencephalogram equipment, radiology equipment, sound recording equipment, and so on. In various examples, the computer systems and devices include one or more of each of the following: a central processing unit (“CPU”) configured to execute computer programs; a computer memory configured to store programs and data while they are being used, including a multithreaded program being tested, a debugger, the facility, an operating system including a kernel, and device drivers; a persistent storage device, such as a hard drive or flash drive configured to persistently store programs and data (e.g., firmware and the like); a computer-readable storage media drive, such as a floppy, flash, CD-ROM, or DVD drive, configured to read programs and data stored on a computer-readable storage medium, such as a floppy disk, flash memory device, CD-ROM, or DVD; and a network connection configured to connect the computer system to other computer systems to send and/or receive data, such as via the internet, a Local Area Network (LAN), a Wide Area Network (WAN), a point-to-point dial-up connection, a cell phone network, or another network and its networking hardware in various examples including routers, switches, and various types of transmitters, receivers, or computer-readable transmission media. While computer systems configured as described above may be used to support the operation of the facility, those skilled in the art will readily appreciate that the facility may be implemented using devices of various types and configurations, and having various components. Elements of the facility may be described in the general context of computer-executable instructions, such as program modules, executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, and/or the like configured to perform particular tasks or implement particular abstract data types and may be encrypted. Furthermore, the functionality of the program modules may be combined or distributed as desired in various examples. Moreover, display pages may be implemented in any of various ways, such as in C++ or as web pages in XML (Extensible Markup Language), HTML (HyperText Markup Language), JavaScript, AJAX (Asynchronous JavaScript and XML) techniques, or any other scripts or methods of creating displayable data, such as the Wireless Access Protocol (WAP). Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments, including cloud-based implementations, web applications, mobile applications for mobile devices, and so on.
  • The following discussion provides a brief, general description of a suitable computing environment in which the disclosed technology can be implemented. Although not required, aspects of the disclosed technology are described in the general context of computer-executable instructions, such as routines executed by a general-purpose data processing device, e.g., a server computer, wireless device, or personal computer. Those skilled in the relevant art will appreciate that aspects of the disclosed technology can be practiced with other communications, data processing, or computer system configurations, including: internet or otherwise network-capable appliances, hand-held devices (including personal digital assistants (PDAs)), wearable computers (e.g., fitness-oriented wearable computing devices), all manner of cellular or mobile phones (including Voice over IP (VoIP) phones), dumb terminals, media players, gaming devices, multi-processor systems, microprocessor-based or programmable consumer electronics, set-top boxes, network PCs, mini-computers, mainframe computers, and the like. Indeed, the terms “computer,” “server,” “host,” “host system,” and the like are generally used interchangeably herein, and refer to any of the above devices and systems, as well as any data processor.
  • Aspects of the disclosed technology can be embodied in a special purpose computer or data processor, such as application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), graphics processing units (GPU), manycore processors, and so on, that is specifically programmed, configured, or constructed to perform one or more of the computer-executable instructions explained in detail herein. While aspects of the disclosed technology, such as certain functions, are described as being performed exclusively on a single device, the disclosed technology can also be practiced in distributed computing environments where functions or modules are shared among disparate processing devices, which are linked through a communications network such as a Local Area Network (LAN), Wide Area Network (WAN), or the internet. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.
  • Aspects of the disclosed technology may be stored or distributed on tangible computer-readable media, including magnetically or optically readable computer discs, hard-wired or preprogrammed chips (e.g., EEPROM semiconductor chips), nanotechnology memory, biological memory, or other computer-readable storage media. Alternatively, computer-implemented instructions, data structures, screen displays, and other data under aspects of the disclosed technology may be distributed over the internet or over other networks (including wireless networks), on a propagated signal on a propagation medium (e.g., electromagnetic wave(s), sound wave, etc.) over a period of time, or they may be provided on any analog or digital network (packet switched, circuit switched, or other scheme). Furthermore, the term computer-readable storage medium does not encompass signals (e.g., propagating signals) or transitory media.
  • FIG. 2 is a flow diagram illustrating the processing of a discover features component 121 in accordance with some embodiments of the disclosed technology. The discover features component is invoked by the facility to identify novel feature vectors based on selected patient data. In block 205, the component receives physiological signal data, such as raw signal data received directly from signal recorder, previously-generated physiological signal from another device or site, etc. Several techniques exist for collecting and analyzing physiological signals (e.g., electrophysiological signals, biosignals) from patients for diagnostic and other purposes including, for example, activity trackers, echocardiograms, wide-band biopotential measuring equipment, electroencephalograms, electromyograms, electrooculography, galvanic skin response, heart rate monitors, magnetic resonance imaging, magnetoencephalograms, mechanomyograms, wearable technology devices (e.g., FITBITs), and so on. While data provided by these systems can be helpful in identifying medical concerns and diagnosing medical conditions, they are often only a starting point for the diagnosis process. Moreover, given the specific nature of most of these systems, the data they analyze is often over-filtered to reduce complexity for the system itself or for, e.g., a technician, physician, or other health care provider (in such cases, to reduce visual complexity, etc.) thereby eliminating data that potentially have untapped diagnostic value. In block 210, the component invokes a process signal data component to process and transform the received signal data, which can produce multiple sets of data and transformed data. In block 215, the component sets a generation value equal to 0. In block 220, the component generates one or more feature generators by, for example, randomly generating an expression tree, randomly generating a set of weights for a neural network, randomly mutating one or more of a set of previously-generated feature generators, and so on. In block 225, the component invokes an apply feature generators component to apply the generated feature generators to one or more sets of the processed signal data to produce a set of feature vectors. In block 230, the component invokes an identify novel feature vectors component to identify the most novel feature vectors from among the group of feature vectors generated by the feature generators. In block 235, the component stores the feature generators that produced the identified feature vectors in, for example, a feature generator store. In block 240, the component increments the generation variable. In decision block 245, if the generation variable is greater than or equal to a generation threshold, then the component completes, else the component continues at block 250. The component may also use other stopping conditions, such as a number of generations of feature generators that do not produce at least a threshold number of novel feature vectors. In block 250, the component copies and mutates the identified feature generators and then loops back to block 225 to apply the mutated feature generators to one or more sets of the processed signal data. As discussed above, the component may apply any type or types of mutations to a feature generator, such as applying multiple point mutations and/or a random recombination to one or more expression trees, randomly generating a set of connection weights for a neural network, and so on.
  • FIG. 3 is a flow diagram illustrating the processing of a process component 122 in accordance with some embodiments of the disclosed technology. The process component is invoked by the discover features component to process and transform patient signal data. In blocks 305 through 365, the component loops through each signal (or data set) of a set of received signals (or set of data sets), each signal representing physiological data received from a patient. In block 310, the component pre-processes the received signal, such as applying one or more signal filters to the signal, performing a peak search on the data and discarding extraneous information, down-sampling the received signal, up-sampling the received signal, sub-sampling the received signal, converting an analog signal to a digital signal, converting image data to signal data, and so on. In block 315, the component stores the pre-processed signal in, for example, a patient data store. The signal data may be stored anonymously (i.e., without explicitly or implicitly identifying the corresponding patient, etc.). However, different instances of signal data associated with the same patient may be associated with an anonymized unique identifier so that multiple signals from a single patient can be used in conjunction for training and diagnostic purposes. In block 320, the component extracts one or more values from the stored signal data. In block 325, the component stores the one or more extracted values. In block 330, the component identifies any transformations to be applied to the signal. For example, the facility may store an indication of a set of transformations or transformation functions (e.g., Fourier transforms, functions to apply to the signal, derivatives, partial derivatives, and so on) to apply to a particular signal. As another example, the facility may randomly select, from among a catalog of transformations, one or more transformations to apply to the signal data. In blocks 335 through 360, the component loops through each of the transformations and applies the transformation to the signal. In block 340, the component applies the transformation to the signal (e.g., calculating the third derivative with respect to a particular variable, calculating the result of a composite function generated by applying one function to the signal data (i.e., a function representative of the signal data), etc.). In block 345, the component stores the transformed signal data in, for example, a patient data store. In block 350, the component extracts one or more values from the transformed signal data. In block 355, the component stores the one or more extracted values. In block 360, if there are any identified transformations yet to be applied, then the component selects the next transformation and loops back to block 335 to apply the transformation to the signal data, else the component continues at block 365. In block 365, if there are any signals yet to be analyzed, then the component selects the next signal and loops back to block 305 to process the next signal, else the component completes.
  • FIG. 4 is a flow diagram illustrating the processing of an apply feature generators component 123 in accordance with some embodiments of the disclosed technology. The apply feature generators component is invoked by the discover features component 121 to apply a set of one or more feature generators to signal data, such as pre-processed and transformed signal data, modeled signal data, etc. In blocks 410 through 470, the component loops through each of a received set of feature generators and applies the feature generator to each signal in a received set of signal data. For example, the received signal data can include multiple sets of signal data for each of multiple patients, multiple transformations of that data, and so on. In blocks 420 through 450, the component loops through each of the signals to apply the feature generators to the signal data. In block 430, the component applies the currently-selected feature generator to the currently-selected signal data. For example, the component may apply the feature generator to each of a pre-processed version of the currently-selected data signal and any transformed version of that data. As another example, the component “plugs in” or substitutes coefficients generated by modeled signal data into a feature generator with a set of variables to produce an output feature value. As another example, the component can apply one or more elements of modeled signal data to a neural network to produce an output feature value. In block 440, the component stores the output value. In block 450, if there are any signals yet to be analyzed, then the component selects the next signal and loops back to block 420 to process the next signal, else the component continues at block 460. In block 460, the component generates a feature vector that includes each of the generated feature values and stores the feature vector in association with the feature generator in, for example, a feature vector store. For example, the feature vector may comprise an array of features and a link to, or identifier of, the corresponding feature generator. The component may also associate the feature vector with the signal data used to generate the feature vector. In block 470, if there are any feature generators yet to be processed, then the component selects the next feature generator and loops back to block 410 to process the feature generator, else the component completes.
  • FIG. 5 is a flow diagram illustrating the processing of an identify novel feature vectors component 124 in accordance with some embodiments of the disclosed technology. In this example, the facility receives a set of feature vectors and, for each feature vector, information related to the corresponding feature generator, such as an identifier for the feature generator. In block 505, the component collects a comparison set of feature vectors that includes, for example, feature vectors generated by feature generators of earlier generations that were found to be novel and the feature vectors generated by a current generation of feature vectors. For example, the component can randomly select a set of novel feature vectors from a feature store. In some cases, a request to retrieve feature vectors includes upper and lower limits on the number of features values for each feature vector to be retrieved, such as no less than 50 (lower threshold) and no more than 5000 (upper threshold). In blocks 510 through 540, the component loops through each feature vector of a current generation of feature generators to determine how different each of their corresponding feature vectors is to each of the feature vectors of the comparison set of feature vectors. In blocks 515 through 530, the component loops through each feature vector of the comparison set of feature vectors to compare each feature vector to the feature vector of the currently-selected feature generator. In block 520, the component calculates a difference value between the currently-selected feature vector of the comparison set and the feature vector of the currently-selected feature generator. For example, the component can calculate a distance value between each of the feature vectors. In block 525, the component stores the calculated difference value. In block 530, if there are any feature vectors yet to be compared, then the component selects the next feature vector and loops back to block 515 to process the feature vector, else the component continues at block 535. In block 535, the component calculates a novelty score for the currently-selected feature generator based on the stored difference values, such as an average or maximum distance, and stores the novelty score in association with the feature generator (e.g., in a feature generator store). In block 540, if there are any feature generators yet to be assessed, then the component selects the next feature generator and loops back to block 515 to process the feature generator, else the component continues at block 545. In blocks 545 through 560, the component tests whether each of the feature vectors is novel, based on the calculated novelty scores, and identifies any corresponding feature generators. In decision block 550, if the novelty score for the currently-selected feature generator is greater than a novelty threshold, then the component continues at block 555, else the component continues at block 560. The novelty threshold may be generated or determined in any number of ways, such as receiving a novelty threshold from a user, calculating a novelty threshold based on the set of novelty scores (e.g., average, average plus 25%, top n (where n is provided by a user or generated automatically by the facility), top tenth percentile), and so on. In this manner, the novelty threshold may change dynamically (e.g., from generation to generation) based on, for example, the number of generations without a new feature generator that exceeds the current novelty threshold to ensure that the facility is producing and testing new feature generators and corresponding features. In block 555, the component identifies the currently-selected feature vector as novel. In block 560, if there are any feature vectors yet to be processed, then the component selects the next feature vector and loops back to block 545 to process the feature vector, else the component completes.
  • FIG. 6 is a flow diagram illustrating the processing of a discover genomes component 126 in accordance with some embodiments of the disclosed technology. The facility invokes the discover genomes component to generate and analyze genomes for use by machine learning algorithms. In block 610, the facility initializes a generation variable equal to 0. In block 620, the component determines a number (n) of genomes to generate based on, for example, user input, system parameters, or randomly. In block 630, the component invokes a spawn genome component n time(s) to spawn the appropriate number of genomes. In block 640, the component invokes an identify high-performing genomes component to identify, from among the spawned genomes, the genomes that have a fitness score that exceeds a fitness threshold. In block 650, the component increments the generation variable. In decision block 660, if the generation variable is greater than or equal to a generation threshold, then processing of the component completes, else the component continues at block 670. In block 670, the component mutates the high-performing genomes and then loops back to block 640 to identify high-performing genomes from among the mutated genomes (e.g., the mutated genomes having fitness scores that exceed a fitness threshold). The component may mutate the genome by adding, changing, or removing (or any combination thereof) one or more elements of the variable-length genome. For example, the component may mutate one genome by replacing one feature with another feature and adding a new feature to the mutated genome. In another example, the component may select a new machine learning algorithm to associate with the genome. In this case, the component may also remove or mutate any irrelevant machine learning algorithm parameters and/or replace them with machine learning parameter values for the newly selected machine learning algorithm. As another example, genomes may use sexual reproduction techniques as a form of mutation by randomly selecting elements of multiple genomes and combining those elements to form a new genome. Moreover, one or more elements of a genome may be configured so that they remain fixed (i.e., are not changed) during the evolutionary process described herein.
  • FIG. 7 is a flow diagram illustrating the processing of a spawn genome component 126 in accordance with some embodiments of the disclosed technology. The discover genomes component 125 invokes the spawn genome component to generate a genome identifying any number of features, machine learning parameters, and/or machine learning algorithms. In block 710, the component identifies a set of available features, such as features referenced in one or more feature generator stores. In block 720, the component determines the number of features to include in the genome to be generated. For example, the component may determine the number of features to include in the genome to be generated based on user input, system parameters, or randomly. In block 730, the component randomly selects, from among the identified features, the determined number of features. In block 740, the component replaces correlated features from among the selected features with randomly selected features. In block 750, the component identifies a set of available machine learning parameters. For example, the component may identify, for each machine learning algorithm available to the facility, a set of parameters associated with that machine learning algorithm, which may be stored in a list or other data structure available to the component (e.g., a machine learning parameter store). In some cases, a genome may be generated for a single machine learning algorithm (or fixed set of machine learning algorithms). In this case, the component may identify only those machine learning parameters (or a proper subset thereof) that are associated with the single or fixed set of machine learning algorithms. In other cases, a genome may include an element that identifies a machine learning algorithm and that can be mutated. In this case, the component may identify any or all machine learning parameters of machine learning algorithms that are within the scope of this mutation (i.e., the parameters of the machine learning algorithms that the genome and its descendants may be associated with to train models during the evolutionary process described herein). In block 760, the component determines the number of machine learning parameters to include in the genome to be generated. For example, the component may determine the number of machine learning parameters to include in the genome to be generated based on user input, system parameters, or randomly. For example, one genome may include each and every machine learning parameter associated with a particular machine learning algorithm or a set of machine learning algorithms while another genome includes only a proper subset of machine learning parameters associated with a particular machine learning algorithm. In block 770, the component randomly selects, from among the identified machine learning parameters, the determined number of machine learning parameters and assigns a value to the parameter based on any associated constraints, such as randomly selecting a value between a minimum value and a maximum value associated with the parameter. In block 780, the component stores each of the selected features and machine learning parameters in a genome data structure and then returns the genome data structure.
  • FIG. 8 is a flow diagram illustrating the processing of an identify high-performing genomes component 127 in accordance with some embodiments of the disclosed technology. The discover genomes component invokes the identify high-performing genomes component to identify, from among a group of genomes, the genomes that have a corresponding fitness score that exceeds a fitness threshold (i.e., the genomes that are “high-performing”). In blocks 810 through 850, the component loops through a set of genomes provided to the component, such a first generation set of genomes, a mutated set of genomes, or some combination thereof. In block 820, the component trains one or more models using the currently-selected genome, including its features, machine learning parameters, and any specified machine learning algorithm. In the event that a machine learning parameter of the genome is not associated with the machine learning algorithm used to train the model, that machine learning parameter may be ignored. Similarly, if a particular machine learning algorithm requires as input a particular machine learning parameter that is not included in the currently-selected genome, the facility (or the machine learning algorithm itself) may provide a default value retrieved from, for example, a machine learning parameter store. In block 830, the component generates a validation or fitness score for the currently-selected genome by, for example, applying the trained model to a set of validation data and assessing the ability of the trained model to correctly identify or classify subjects from among the validation data. In block 840, the component stores the generated score(s) and/or an aggregation thereof produced for the trained models in association with the currently-selected genome. In block 850, if there are any genomes yet to be scored, then the component selects the next genome and loops back to block 810, else the component continues at block 860. In blocks 860 through 890, the component assesses the score generated for each genome and selects the “best” genomes for mutation. In this example, the “best” genomes are those that produce validation or fitness scores that exceed a fitness threshold. In decision block 870, if the score generated for the currently-selected genome exceeds a fitness threshold, then the component continues at block 880, else the component continues at block 890. In block 880, the component flags the currently-selected genome for mutation. In some embodiments, the component may select genomes for mutation based on criteria other than, or in addition to, fitness scores. For example, the component may use novelty scores or other scores to select genomes for mutation. In some cases, the component may employ a tournament selection process in which a number of genomes are randomly selected from the population and the genome with the highest score among this “tournament” is selected for reproduction. In this example, if only low scoring genomes appear in a tournament, a low-scoring genome will be selected for reproduction. In block 890, if there are any genomes yet to be processed, then the component selects the next genome and loops back to block 860, else the component returns the flagged genomes and processing of the component completes.
  • From the foregoing, it will be appreciated that specific embodiments of the disclosed technology have been described herein for purposes of illustration, but that various modifications may be made without deviating from the scope of the disclosed technology. For example, the disclosed techniques can be applied to fields outside of the medical field, such as predicting weather patterns, geological activity, or any other field in which predictions are made based on sampled input data. To reduce the number of claims, certain aspects of the disclosed technology are presented below in certain claim forms, but applicants contemplate the various aspects of the disclosed technology in any number of claim forms. Accordingly, the disclosed technology is not limited except as by the appended claims.

Claims (21)

1. A system, having a memory and a processor, for discovering machine learning genomes, the system comprising:
a first component configured to generate a plurality of genomes, wherein each genome identifies at least one feature and at least one parameter for at least one machine learning algorithm, wherein generating a first genome of the plurality of genomes comprises:
randomly selecting, from among a set of features, one or more of the features,
randomly selecting, from among a set of parameters for at least one machine learning algorithm, one or more of the parameters, and
assigning at least one random value to each of the selected parameters;
a second component configured to, for each generated genome,
train a one or more models using the generated genome, and
for each model trained using the generated genome,
calculate a fitness score for the trained model at least in part by applying the trained model to a validation data set, and
produce a fitness score for the generated genome based at least in part on the fitness scores generated for the models trained using the generated genome;
a third component configured to identify, from among the generated genomes, a plurality of genomes having a fitness score that exceeds a fitness threshold; and
a fourth component configured to, for each of the identified genomes, mutate the identified genome,
wherein at least one of the components comprises computer-executable instructions stored in the memory for execution by the system.
2. The system of claim 1, further comprising:
a fifth component configured to, for the first genome comprising a first set of features, identify correlated features from among the first set of features at least in part by:
for each feature of the first set of features,
applying a feature generator associated with the feature to a training set of data to generate a feature vector for the feature,
for at least one pair of feature vectors,
calculating a distance between each feature vector of the pair of feature vectors,
determining that the calculated distance is less than a distance threshold,
in response to determining that the calculated distance is less than a distance threshold, removing, from the first genome, a feature corresponding to at least one feature vector of the pair of feature vectors,
wherein each feature vector includes, for each of a plurality of patients, a single value generated by applying a first feature generator to at least one representation of physiological data representative of the patient.
3. The system of claim 2, wherein the removing, from the first genome, of at least one feature corresponding to at least one feature vector of a first pair of feature vectors comprises:
randomly selecting one of feature vector of the first pair of feature vectors,
identifying, from among the features of the first genome, a feature corresponding to the randomly selected feature vector; and
removing, from the first genome, the identified feature.
4. The system of claim 1, further comprising:
a fifth component configured to, for the first genome comprising a first set of features, generate a graph comprising a vertex for each feature of the first set of features;
a sixth component configure to generate an edge between vertices whose corresponding features have a correlation value that exceeds a correlation threshold or a distance value that is less than a distance threshold; and
a seventh component configured to remove vertices from the graph until no connected vertices remain in the graph.
5. The system of claim 1, further comprising:
a machine configured to receive physiological signal data from at least one patient;
a fifth component configured to, for each patient,
apply at least one of the trained models to at least a portion of the physiological signal data received for the patient by the machine, and
generate a prediction for the patient based at least in part on the application of the at least one of the trained models to at least a portion of the received physiological signal.
6. A method, performed by a computing system having a memory and a processor, for discovering machine learning genomes, the method comprising:
generating, with the processor, a plurality of genomes, wherein each genome identifies at least one feature and at least one parameter for at least one machine learning algorithm;
for each generated genome,
training at least one model using the generated genome, and
producing a fitness score for the genome based at least in part on the trained at least one model;
identifying, from among the generated genomes, at least one genome having a fitness score that exceeds a fitness threshold; and
mutating each identified genome.
7. The method of claim 6, wherein generating a first genome of the plurality of genomes comprises:
randomly selecting, from among a set of features, one or more of the features;
randomly selecting, from among a set of parameters for at least one machine learning algorithm, one or more of the parameters; and
assigning at least one value to each of the selected parameters.
8. The method of claim 7, wherein generating the first genome further comprises:
for each feature of the randomly selected features,
retrieving a feature vector for the feature based at least in part on a feature generator associated with the feature and a training set of data;
identifying pairs of correlated feature vectors from among the generated feature vectors; and
for each identified pair of correlated feature vectors,
identifying one feature vector of the pair of correlated feature vectors,
removing, from the first genome, the feature associated with the feature generator used to generate the identified feature vector;
randomly selecting, from among the set of features, a feature to add to the first genome, and
adding the randomly selected feature to the first genome.
9. The method of claim 8, wherein identifying pairs of correlated feature vectors comprises:
for each pair of feature vectors,
calculating a distance metric for the pair of feature vectors, and
determining whether the distance metric calculated for the pair of feature vectors is less than a distance threshold,
wherein the distance threshold is determined based at least in part on the calculated distance metrics determined for each pair of feature vectors.
10. The method of claim 6, wherein producing a fitness score for a first genome comprises:
identifying a number of false positives generated by applying, to two or more validation data sets, a model trained using the first genome; and
identifying a number of false negatives generated by applying, to two or more validation data sets, a model trained using the first genome.
11. The method of claim 6, wherein producing a fitness score for a first genome comprises:
generating, for at least one model trained using the first genome, a receiver operating characteristic curve; and
calculating an area under the generated receiver operating characteristic curve.
12. The method of claim 6, wherein producing a fitness score for a first genome comprises calculating, for at least one model trained using the first genome, one or more of the errors selected from the group comprising: mean squared prediction error, mean absolute error, interquartile error, and log loss error, receiver-operator characteristic curve error, and f-score error.
13. The method of claim 6, wherein mutating a first identified genome comprises:
selecting at least one feature of the first identified genome; and
removing, from the first identified genome, each of the selected features of the first identified genome.
14. The method of claim 6, wherein mutating the first identified genome further comprises:
randomly selecting, from among the set of features, a plurality of the features; and
adding, to the first identified genome, each of the randomly selected plurality of features.
15. The method of claim 6, wherein mutating a first identified genome comprises:
modifying at least one feature of the first identified genome.
16. The method of claim 5, wherein mutating a first identified genome comprises:
modifying at least one machine learning algorithm parameter of the first identified genome.
17. A computer-readable medium storing instructions that, if executed by a computing system having a memory and a processor, cause the computing system to perform a method for discovering machine learning genomes, the method comprising:
generating a plurality of genomes, wherein each genome identifies at least one feature;
for each generated genome,
training at least one model using the generated genome, and
producing a fitness score for the genome based at least in part on the trained at least one model; and
identifying, from among the generated genomes, one or more genomes having a fitness score that exceeds a fitness threshold.
18. The computer-readable medium of claim 17, wherein each genome further identifies at least one parameter for at least one machine learning algorithm.
19. The computer-readable medium of claim 17, the method further comprising:
mutating each identified genome having a fitness score that exceeds the fitness threshold.
20. The computer-readable medium of claim 17, the method further comprising:
computing the fitness threshold at least in part by, determining an overall fitness score based on the fitness scores produced for each of the generated genomes.
21-26. (canceled)
US17/339,583 2017-07-18 2021-06-04 Discovering genomes to use in machine learning techniques Pending US20220093215A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US17/339,583 US20220093215A1 (en) 2017-07-18 2021-06-04 Discovering genomes to use in machine learning techniques

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US15/653,441 US11062792B2 (en) 2017-07-18 2017-07-18 Discovering genomes to use in machine learning techniques
US17/339,583 US20220093215A1 (en) 2017-07-18 2021-06-04 Discovering genomes to use in machine learning techniques

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US15/653,441 Continuation US11062792B2 (en) 2017-07-18 2017-07-18 Discovering genomes to use in machine learning techniques

Publications (1)

Publication Number Publication Date
US20220093215A1 true US20220093215A1 (en) 2022-03-24

Family

ID=65015749

Family Applications (2)

Application Number Title Priority Date Filing Date
US15/653,441 Active 2040-03-13 US11062792B2 (en) 2017-07-18 2017-07-18 Discovering genomes to use in machine learning techniques
US17/339,583 Pending US20220093215A1 (en) 2017-07-18 2021-06-04 Discovering genomes to use in machine learning techniques

Family Applications Before (1)

Application Number Title Priority Date Filing Date
US15/653,441 Active 2040-03-13 US11062792B2 (en) 2017-07-18 2017-07-18 Discovering genomes to use in machine learning techniques

Country Status (6)

Country Link
US (2) US11062792B2 (en)
EP (1) EP3655894A4 (en)
JP (1) JP7337788B2 (en)
CN (1) CN111095232A (en)
CA (1) CA3069833A1 (en)
WO (1) WO2019016608A1 (en)

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10460440B2 (en) * 2017-10-24 2019-10-29 General Electric Company Deep convolutional neural network with self-transfer learning
US11514327B2 (en) * 2018-06-19 2022-11-29 Orbai Technologies, Inc. Apparatus and method for utilizing a parameter genome characterizing neural network connections as a building block to construct a neural network with feedforward and feedback paths
US11087170B2 (en) * 2018-12-03 2021-08-10 Advanced Micro Devices, Inc. Deliberate conditional poison training for generative models
US20200303060A1 (en) * 2019-03-18 2020-09-24 Nvidia Corporation Diagnostics using one or more neural networks
US11803172B2 (en) * 2019-05-10 2023-10-31 Mjnn Llc Efficient selection of experiments for enhancing performance in controlled environment agriculture
US11521115B2 (en) * 2019-05-28 2022-12-06 Microsoft Technology Licensing, Llc Method and system of detecting data imbalance in a dataset used in machine-learning
US11526701B2 (en) 2019-05-28 2022-12-13 Microsoft Technology Licensing, Llc Method and system of performing data imbalance detection and correction in training a machine-learning model
US11537941B2 (en) 2019-05-28 2022-12-27 Microsoft Technology Licensing, Llc Remote validation of machine-learning models for data imbalance

Family Cites Families (126)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5148513A (en) * 1988-05-20 1992-09-15 John R. Koza Non-linear genetic process for use with plural co-evolving populations
JP3594106B2 (en) * 1997-07-24 2004-11-24 オムロン株式会社 Automatic programming device, automatic programming method and recording medium
US7970718B2 (en) 2001-05-18 2011-06-28 Health Discovery Corporation Method for feature selection and for evaluating features identified as significant for classifying data
US6513025B1 (en) 1999-12-09 2003-01-28 Teradyne, Inc. Multistage machine learning process
JP2004529406A (en) 2000-11-10 2004-09-24 アフィノバ, インコーポレイテッド Method and apparatus for dynamic real-time market segmentation
US6917926B2 (en) * 2001-06-15 2005-07-12 Medical Scientists, Inc. Machine learning method
US20030088565A1 (en) 2001-10-15 2003-05-08 Insightful Corporation Method and system for mining large data sets
WO2004063831A2 (en) 2003-01-15 2004-07-29 Bracco Imaging S.P.A. System and method for optimization of a database for the training and testing of prediction algorithms
JP2007504576A (en) * 2003-01-17 2007-03-01 アヤラ,フランシスコ,ジェイ System and method for developing artificial intelligence
CA2542937A1 (en) 2003-07-01 2005-01-13 Cardiomag Imaging, Inc. (Cmi) Machine learning for classification of magneto cardiograms
WO2005036180A1 (en) 2003-10-08 2005-04-21 The Government Of The United States Of America As Represented By The Secretary Of Department Of Health And Human Services Analysis methods using biomarkers concentrated with biomarkers attractant molecules
WO2005048185A1 (en) 2003-11-17 2005-05-26 Auckland University Of Technology Transductive neuro fuzzy inference method for personalised modelling
US7480640B1 (en) 2003-12-16 2009-01-20 Quantum Leap Research, Inc. Automated method and system for generating models from data
WO2005086060A1 (en) * 2004-03-02 2005-09-15 Cloudmark, Inc. Method and apparatus to use a genetic algorithm to generate an improved statistical model
US7827011B2 (en) 2005-05-03 2010-11-02 Aware, Inc. Method and system for real-time signal classification
JP4618058B2 (en) 2005-09-01 2011-01-26 株式会社日立製作所 Background image generation method and apparatus, and image monitoring system
US7599893B2 (en) 2005-10-13 2009-10-06 Aureon Laboratories, Inc. Methods and systems for feature selection in machine learning based on feature contribution and model fitness
WO2007137047A2 (en) 2006-05-16 2007-11-29 Greer Douglas S Modeling the neocortex
WO2007147166A2 (en) 2006-06-16 2007-12-21 Quantum Leap Research, Inc. Consilence of data-mining
US8046200B2 (en) 2006-09-05 2011-10-25 Colorado State University Research Foundation Nonlinear function approximation over high-dimensional domains
US7792770B1 (en) 2007-08-24 2010-09-07 Louisiana Tech Research Foundation; A Division Of Louisiana Tech University Foundation, Inc. Method to indentify anomalous data using cascaded K-Means clustering and an ID3 decision tree
WO2009063463A2 (en) 2007-11-14 2009-05-22 Medasense Biometrics Ltd Pain monitoring using multidimensional analysis of physiological signals
US8065247B2 (en) 2007-11-21 2011-11-22 Inomaly, Inc. Systems and methods for multivariate influence analysis of heterogenous mixtures of categorical and continuous data
JP4591566B2 (en) * 2008-07-14 2010-12-01 ソニー株式会社 Information processing apparatus, information processing method, and program
US8340746B2 (en) 2008-07-17 2012-12-25 Massachusetts Institute Of Technology Motif discovery in physiological datasets: a methodology for inferring predictive elements
US8972410B2 (en) 2008-07-30 2015-03-03 Hewlett-Packard Development Company, L.P. Identifying related objects in a computer database
WO2010030794A1 (en) 2008-09-10 2010-03-18 Digital Infuzion, Inc. Machine learning methods and systems for identifying patterns in data
WO2010035163A1 (en) 2008-09-29 2010-04-01 Koninklijke Philips Electronics, N.V. Method for increasing the robustness of computer-aided diagnosis to image processing uncertainties
NZ572036A (en) 2008-10-15 2010-03-26 Nikola Kirilov Kasabov Data analysis and predictive systems and related methodologies
US20110295782A1 (en) 2008-10-15 2011-12-01 Alexander Stojadinovic Clinical Decision Model
US8016200B2 (en) 2009-08-31 2011-09-13 Symbol Technologies, Inc. Handle and activation assembly for portable electronic device
US11562323B2 (en) * 2009-10-01 2023-01-24 DecisionQ Corporation Application of bayesian networks to patient screening and treatment
CN101785672B (en) 2010-01-29 2011-08-31 重庆大学 Breast tumor diagnosis system based on magnetic resonance spectrum imaging
CN107742534A (en) * 2010-03-15 2018-02-27 新加坡保健服务集团有限公司 Patient's viability forecasting system
US20120029974A1 (en) 2010-07-30 2012-02-02 International Business Machines Corporation Complex service modeling
CN106198980B (en) * 2010-08-13 2018-09-07 私募蛋白质体公司 Cancer of pancreas biomarker and application thereof
US10321892B2 (en) 2010-09-27 2019-06-18 Siemens Medical Solutions Usa, Inc. Computerized characterization of cardiac motion in medical diagnostic ultrasound
WO2012103290A1 (en) 2011-01-26 2012-08-02 Google Inc. Dynamic predictive modeling platform
US8533224B2 (en) 2011-05-04 2013-09-10 Google Inc. Assessing accuracy of trained predictive models
US9336484B1 (en) 2011-09-26 2016-05-10 The United States Of America As Represented By The Administrator Of The National Aeronautics And Space Administration (Nasa) System and method for outlier detection via estimating clusters
US9934361B2 (en) 2011-09-30 2018-04-03 Univfy Inc. Method for generating healthcare-related validated prediction models from multiple sources
KR101725121B1 (en) 2011-10-19 2017-04-12 한국전자통신연구원 Feature vector classification device and method thereof
US8688603B1 (en) 2011-11-14 2014-04-01 Amazon Technologies, Inc. System and method for identifying and correcting marginal false positives in machine learning models
JP6226187B2 (en) 2012-01-30 2017-11-08 日本電気株式会社 Information processing system, information processing method, information processing apparatus and control method and control program thereof, communication terminal and control method and control program thereof
AU2013274606B2 (en) 2012-06-11 2015-09-17 Landmark Graphics Corporation Methods and related systems of building models and predicting operational outcomes of a drilling operation
US10373708B2 (en) * 2012-06-21 2019-08-06 Philip Morris Products S.A. Systems and methods for generating biomarker signatures with integrated dual ensemble and generalized simulated annealing techniques
CA2877429C (en) * 2012-06-21 2020-11-03 Philip Morris Products S.A. Systems and methods for generating biomarker signatures with integrated bias correction and class prediction
US20140108324A1 (en) 2012-10-12 2014-04-17 Nec Laboratories America, Inc. Data analytic engine towards the self-management of complex physical systems
US20140143188A1 (en) 2012-11-16 2014-05-22 Genformatic, Llc Method of machine learning, employing bayesian latent class inference: combining multiple genomic feature detection algorithms to produce an integrated genomic feature set with specificity, sensitivity and accuracy
US9576262B2 (en) 2012-12-05 2017-02-21 Microsoft Technology Licensing, Llc Self learning adaptive modeling system
WO2014144246A1 (en) 2013-03-15 2014-09-18 Cyberricade, Inc. Cyber security
CN103177114B (en) 2013-04-02 2016-01-27 浙江大学 Based on the shift learning sorting technique across data field differentiating stream shape
WO2014186387A1 (en) 2013-05-14 2014-11-20 The Regents Of The University Of California Context-aware prediction in medical systems
ES2774965T3 (en) 2013-09-27 2020-07-23 Codexis Inc Predictive modeling based on structure
WO2015066052A1 (en) 2013-10-28 2015-05-07 New York University Methods, computer-accessible medium and systems to model disease progression using biomedical data from multiple patients
US20150127595A1 (en) 2013-11-01 2015-05-07 Numenta, Inc. Modeling and detection of anomaly based on prediction
WO2015069827A2 (en) 2013-11-06 2015-05-14 H. Lee Moffitt Cancer Center And Research Institute, Inc. Pathology case review, analysis and prediction
US9652354B2 (en) 2014-03-18 2017-05-16 Microsoft Technology Licensing, Llc. Unsupervised anomaly detection for arbitrary time series
US10366346B2 (en) 2014-05-23 2019-07-30 DataRobot, Inc. Systems and techniques for determining the predictive value of a feature
US9886670B2 (en) * 2014-06-30 2018-02-06 Amazon Technologies, Inc. Feature processing recipes for machine learning
US10169715B2 (en) * 2014-06-30 2019-01-01 Amazon Technologies, Inc. Feature processing tradeoff management
WO2016022438A1 (en) 2014-08-08 2016-02-11 Icahn School Of Medicine At Mount Sinai Automatic disease diagnoses using longitudinal medical record data
US9697469B2 (en) 2014-08-13 2017-07-04 Andrew McMahon Method and system for generating and aggregating models based on disparate data from insurance, financial services, and public industries
US20160045120A1 (en) 2014-08-15 2016-02-18 Massachusetts Institute Of Technology Systems and methods for spike sorting
AU2015306693A1 (en) 2014-08-25 2017-03-02 Shl Us Llc Customizable machine learning models
SG10201503424YA (en) 2014-10-09 2016-05-30 Cloudradigm Pte Ltd A computer implemented method and system for automatically modelling a problem and orchestrating candidate algorithms to solve the problem
US9349178B1 (en) 2014-11-24 2016-05-24 Siemens Aktiengesellschaft Synthetic data-driven hemodynamic determination in medical imaging
US20160162457A1 (en) 2014-12-09 2016-06-09 Idibon, Inc. Optimization techniques for artificial intelligence
US9659259B2 (en) 2014-12-20 2017-05-23 Microsoft Corporation Latency-efficient multi-stage tagging mechanism
WO2016118513A1 (en) 2015-01-20 2016-07-28 The Broad Institute, Inc. Method and system for analyzing biological networks
EP3048563A1 (en) 2015-01-22 2016-07-27 Alcatel Lucent Method and system for incremental manifold learning
AU2016226162B2 (en) 2015-03-03 2017-11-23 Nantomics, Llc Ensemble-based research recommendation systems and methods
WO2016164680A2 (en) 2015-04-09 2016-10-13 Equifax, Inc. Automated model development process
WO2016164815A1 (en) 2015-04-10 2016-10-13 Applied Proteomics, Inc. Protein biomarker panels for detecting colorectal cancer and advanced adenoma
US9996804B2 (en) 2015-04-10 2018-06-12 Facebook, Inc. Machine learning model tracking platform
US20190057182A1 (en) 2015-05-22 2019-02-21 Csts Health Care Inc. Biomarker-driven molecularly targeted combination therapies based on knowledge representation pathway analysis
US10984338B2 (en) 2015-05-28 2021-04-20 Raytheon Technologies Corporation Dynamically updated predictive modeling to predict operational outcomes of interest
US20160358099A1 (en) * 2015-06-04 2016-12-08 The Boeing Company Advanced analytical infrastructure for machine learning
US11250342B2 (en) 2015-07-16 2022-02-15 SparkBeyond Ltd. Systems and methods for secondary knowledge utilization in machine learning
CN108471973B (en) 2015-08-26 2021-08-10 生命解析公司 Method and apparatus for wideband phase gradient signal acquisition
US11715029B2 (en) 2015-09-21 2023-08-01 Equifax Inc. Updating attribute data structures to indicate trends in attribute data provided to automated modeling systems
WO2017059022A1 (en) 2015-09-30 2017-04-06 Inform Genomics, Inc. Systems and methods for predicting treatment-regiment-related outcomes
US20170169180A1 (en) 2015-12-14 2017-06-15 International Business Machines Corporation Situation-dependent blending method for predicting the progression of diseases or their responses to treatments
WO2017120579A1 (en) 2016-01-10 2017-07-13 Presenso, Ltd. System and method for validating unsupervised machine learning models
US20170249434A1 (en) 2016-02-26 2017-08-31 Daniela Brunner Multi-format, multi-domain and multi-algorithm metalearner system and method for monitoring human health, and deriving health status and trajectory
CN105843896A (en) 2016-03-22 2016-08-10 中国科学院信息工程研究所 Redundant source synergistic reducing method of multi-source heterogeneous big data
CN105912500B (en) 2016-03-30 2017-11-14 百度在线网络技术(北京)有限公司 Machine learning model generation method and device
US20180032678A1 (en) 2016-07-29 2018-02-01 International Business Machines Corporation Medical recording system
US10650008B2 (en) 2016-08-26 2020-05-12 International Business Machines Corporation Parallel scoring of an ensemble model
US9582781B1 (en) 2016-09-01 2017-02-28 PagerDuty, Inc. Real-time adaptive operations performance management system using event clusters and trained models
WO2018085859A1 (en) 2016-11-07 2018-05-11 Ayasdi, Inc. Dimension grouping and reduction for model generation, testing, and documentation
US20180137415A1 (en) 2016-11-11 2018-05-17 Minitab, Inc. Predictive analytic methods and systems
CN106777891B (en) 2016-11-21 2019-06-07 中国科学院自动化研究所 A kind of selection of data characteristics and prediction technique and device
IL257381A (en) 2017-02-06 2018-03-29 Neural Algorithms Ltd System and method for automatic data modelling
US11416751B2 (en) 2017-03-31 2022-08-16 H2O.Ai Inc. Time-based ensemble machine learning model
US9864956B1 (en) 2017-05-01 2018-01-09 SparkCognition, Inc. Generation and use of trained file classifiers for malware detection
US11081215B2 (en) 2017-06-01 2021-08-03 International Business Machines Corporation Medical record problem list generation
US11106978B2 (en) 2017-09-08 2021-08-31 SparkCognition, Inc. Execution of a genetic algorithm with variable evolutionary weights of topological parameters for neural network generation and training
US10719521B2 (en) 2017-09-18 2020-07-21 Google Llc Evaluating models that rely on aggregate historical data
JP6859247B2 (en) 2017-10-26 2021-04-14 日本電信電話株式会社 Learning equipment, analysis systems, learning methods and learning programs
US10635978B2 (en) 2017-10-26 2020-04-28 SparkCognition, Inc. Ensembling of neural network models
US11663517B2 (en) 2017-11-03 2023-05-30 Salesforce, Inc. Automatic machine learning model generation
TWI651664B (en) 2017-11-15 2019-02-21 財團法人資訊工業策進會 Model generation server and its model generation method
US10405219B2 (en) 2017-11-21 2019-09-03 At&T Intellectual Property I, L.P. Network reconfiguration using genetic algorithm-based predictive models
US11599753B2 (en) 2017-12-18 2023-03-07 Oracle International Corporation Dynamic feature selection for model generation
CN107993723A (en) 2017-12-19 2018-05-04 苏州大学 A kind of warfarin dose prediction modeling method based on integrated evolutionary learning
CN108090570A (en) 2017-12-20 2018-05-29 第四范式(北京)技术有限公司 For selecting the method and system of the feature of machine learning sample
KR102038703B1 (en) 2017-12-27 2019-11-26 (주)가디엘 Method for estimation on online multivariate time series using ensemble dynamic transfer models and system thereof
CN108090516A (en) 2017-12-27 2018-05-29 第四范式(北京)技术有限公司 Automatically generate the method and system of the feature of machine learning sample
US11918333B2 (en) 2017-12-29 2024-03-05 Analytics For Life Inc. Method and system to assess disease using phase space tomography and machine learning
CN108335756B (en) 2018-01-18 2022-05-24 中山大学 Nasopharyngeal carcinoma database and comprehensive diagnosis and treatment decision method based on database
US20190219994A1 (en) 2018-01-18 2019-07-18 General Electric Company Feature extractions to model large-scale complex control systems
WO2019179836A1 (en) 2018-03-19 2019-09-26 Onera Technologies B.V. A method and a system for detecting a respiratory event of a subject and a method for forming a model for detecting a respiratory event
US11475372B2 (en) 2018-03-26 2022-10-18 H2O.Ai Inc. Evolved machine learning models
CN108960269B (en) 2018-04-02 2022-05-27 创新先进技术有限公司 Feature acquisition method and device for data set and computing equipment
CN108875815A (en) 2018-06-04 2018-11-23 深圳市研信小额贷款有限公司 Feature Engineering variable determines method and device
CN109117864B (en) 2018-07-13 2020-02-28 华南理工大学 Coronary heart disease risk prediction method, model and system based on heterogeneous feature fusion
CN109146076A (en) 2018-08-13 2019-01-04 东软集团股份有限公司 model generating method and device, data processing method and device
CN109217291A (en) 2018-08-28 2019-01-15 南京理工大学 Consider the electrical interconnection system Multipurpose Optimal Method of peak load shifting model
CN109242021A (en) 2018-09-07 2019-01-18 浙江财经大学 A kind of classification prediction technique based on multistage mixed model
CN109273094B (en) 2018-09-14 2021-11-12 道之精准医学科技(上海)有限公司 Construction method and construction system of Kawasaki disease risk assessment model based on Boosting algorithm
CN109635955A (en) 2018-11-28 2019-04-16 中诚信征信有限公司 A kind of feature combination method, device and equipment
CN109711558A (en) 2018-12-20 2019-05-03 四川新网银行股份有限公司 For the method and device of the machine learning model of feature construction containing latent instability
CN109828836B (en) 2019-01-20 2021-04-30 北京工业大学 Parameter dynamic configuration method for batch streaming computing system
CN109948668A (en) 2019-03-01 2019-06-28 成都新希望金融信息有限公司 A kind of multi-model fusion method
CN110175644A (en) 2019-05-27 2019-08-27 恒安嘉新(北京)科技股份公司 Feature selection approach, device, electronic equipment and storage medium

Also Published As

Publication number Publication date
US11062792B2 (en) 2021-07-13
WO2019016608A1 (en) 2019-01-24
US20190026431A1 (en) 2019-01-24
EP3655894A4 (en) 2021-05-26
CA3069833A1 (en) 2019-01-24
JP7337788B2 (en) 2023-09-04
JP2020530933A (en) 2020-10-29
EP3655894A1 (en) 2020-05-27
CN111095232A (en) 2020-05-01

Similar Documents

Publication Publication Date Title
US20220093216A1 (en) Discovering novel features to use in machine learning techniques, such as machine learning techniques for diagnosing medical conditions
US20220093215A1 (en) Discovering genomes to use in machine learning techniques
CN110996785B (en) Machine discrimination of anomalies in biological electromagnetic fields
Lim et al. Analysis of single-electrode EEG rhythms using MATLAB to elicit correlation with cognitive stress
RU2657384C2 (en) Method and system for noninvasive screening physiological parameters and pathology
Sowmya et al. Contemplate on ECG signals and classification of arrhythmia signals using CNN-LSTM deep learning model
JP7173482B2 (en) Health care data analysis system, health care data analysis method and health care data analysis program
Rouhani et al. A comparison of different feature extraction methods for diagnosis of valvular heart diseases using PCG signals
Lucena et al. The performance of short-term heart rate variability in the detection of congestive heart failure
Karimi Moridani An automated method for sleep apnoea detection using HRV
Agrawal et al. A machine learning approach for classifying healthy and infarcted patients using heart rate variabilities derived vector magnitude
Assadi et al. PVC arrhythmia classification based on fractional order system modeling
Sabouri et al. Effective features in the diagnosis of cardiovascular diseases through phonocardiogram
Raj An improved time-frequency method for efficient diagnosis of cardiac arrhythmias
Andersson Classification of heart sounds with deep learning
Bharathi Vidhya et al. Utilizing variable auto encoder-based TDO optimization algorithm for predicting loneliness from electrocardiogram signals
Lechuga et al. New Methods on Image Analysis, Two Applications: An Augmented Reality Application for Heartbeat Sound Analysis And A MRI Brain Injury Image Analysis
Maran MULTI-LAYERED ARCHITECTURE CONVOLUTION NEURAL NETWORKS FOR DIAGNOSING AND PREDICTING HEART DISEASES ON MULTI-MODAL
Baysal Assessment of Undergraduate Student Graduation Projects Focusing on Deep Learning in Biomedical Sciences
Gupta et al. Heart Disease Prediction using Machine learning algorithms
Tihak et al. Classification of Atrial Fibrillation ECG Signals Using 2D CNN
Deepa et al. Experimental evaluation of artificial intelligence assisted heart disease prediction using deep learning principle
Hameed An Investigation into using Deep Convolutional Neural Networks for ECG Analysis
Jahmunah Development of advanced artificial intelligence techniques for the detection of myocardial infarction ECG signals in clinical settings
Alves Sleep analysis through electroencephalogram cyclic alternating pattern a phase detection

Legal Events

Date Code Title Description
STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION