JP7496470B2 - Techniques for analyzing and detecting execution artifacts in microwell plates - Patents.com - Google Patents

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Patent Application No. 16/940,320, filed July 27, 2020, and also claims the benefit of U.S. Patent Application No. 16/940,325, filed July 27, 2020. The subject matter of these related applications is incorporated herein by reference.

Various embodiments relate generally to computer science and biochemical analysis, and more specifically to techniques for analyzing and detecting performance artifacts in microwell plates.

High-throughput screening is an automated process that allows researchers to perform tens or even hundreds of thousands of chemical, biological, genetic, and/or pharmacological tests per day. In a typical experiment, an integrated system automatically performs tests using a set of microwell plates, each plate containing a two-dimensional ("2D") grid of wells. The integrated system dispenses samples of various compounds to be tested into the various wells along with samples of the target. After an incubation period until any reaction between the compounds and the target occurs, various measurements are performed on the wells and the results are stored in an experimental dataset as a 2D array of measurements.

One challenge associated with high throughput screening is that the experimental data set may contain certain errors, called "run artifacts", that result from the execution of the experiment itself. For example, if a dispensing nozzle is partially clogged and cannot properly dispense samples of the target into the specific wells assigned to that nozzle, the measurements performed in those specific wells will not capture the actual or "complete" reaction between the target and the compound corresponding to the well. Because the measurements obtained from measurements performed in specific wells are inaccurate and do not reflect the actual or "complete" reaction, these measurements are considered run artifacts that reduce the overall quality of the experimental data set. In general, it is much more difficult to draw valid conclusions about the effectiveness of various compounds against the target while using an experimental data set of poor quality. Thus, various attempts have been made to identify and mitigate run artifacts in experimental data sets.

In one approach to identifying execution artifacts, a human reviewer analyzes "heat maps" (various arrays of measurements, or visual representations of various arrays of measurements) to attempt to detect anomalous patterns of measurements that may be indicative of execution artifacts. Upon identifying an anomalous pattern, the reviewer typically annotates the associated plate to indicate the type and severity of the suspected execution artifact(s). Based on the annotated information, measurements associated with the plate may be removed from the experimental dataset and/or reconsidered. Additionally, for some types of execution artifacts, an attempt is made to determine and correct the root cause of the artifact.

One drawback of the above approach is that manually analyzing heatmaps is time-consuming and error-prone. Often, reviewers are unable to adequately scrutinize all heatmaps associated with an experiment in the time available for the analysis process. To complete the analysis process in the available time, reviewers typically analyze only a limited number of heatmaps and/or perform a cursory analysis of the heatmaps. As a result, execution anomalies reflected in the heatmaps may be overlooked or misinterpreted.

Another drawback is that identifying execution artifacts is a subjective process. Thus, even heat maps that show similar visual patterns may vary in the number and/or type of execution artifacts identified by different reviewers. Furthermore, because manual review processes do not consistently identify execution artifacts, it is very difficult, if not impossible, to detect trends in execution artifacts over time. As a result, opportunities to improve experimental processes and/or equipment to reduce execution artifacts may be missed.

As the above indicates, what is needed in the art are more effective techniques for analyzing and detecting performance artifacts in experiments involving microwell plates.

One embodiment of the present invention provides a method for detecting execution artifacts in an experiment involving a microwell plate. The method includes calculating one or more sets of spatial features based on one or more heat maps associated with a first microwell plate, aggregating the one or more sets of spatial features to generate a first feature vector, and inputting the first feature vector into a trained classifier, which in response generates a first label indicating that the first microwell plate is associated with a first execution artifact.

At least one technical advantage of the disclosed technology over the prior art is that the disclosed technology can be used to more accurately and consistently analyze and detect execution artifacts in experiments involving microwell plates. In particular, the disclosed technology automatically classifies each microwell plate based on spatial patterns detected in a heat map generated for the microwell plate. Thus, the likelihood that execution anomalies reflected in the heat map will be overlooked or misinterpreted is reduced compared to prior art approaches. Furthermore, because microwell plates are classified in a consistent and objective manner with respect to execution artifacts, trends in execution artifacts over time can be effectively detected and used to improve experimental processes and/or equipment. These technical advantages provide one or more technical improvements over prior art approaches.

So that the features recited above in the various embodiments can be understood in detail, a more particular description of the inventive concept briefly summarized above can be made by reference to various embodiments, each of which is illustrated in the accompanying drawings. It should be noted, however, that the accompanying drawings illustrate only exemplary embodiments of the inventive concept, and thus should not be considered as limiting the scope in any way, as there are other equally effective embodiments.

FIG. 1 is a conceptual diagram of a system configured to implement one or more aspects of various embodiments. 2 is a more detailed diagram of the feature engine of FIG. 1 in accordance with various embodiments. 2 is a more detailed diagram of the output engine of FIG. 1 in accordance with various embodiments. 1 is a flow diagram of method steps for training a classifier to detect performance artifacts in experiments involving microwell plates, according to various embodiments. 1 is a flow diagram of method steps for detecting performance artifacts in experiments involving microwell plates using a trained classifier, in accordance with various embodiments.

In the following description, numerous specific details are set forth to provide a more thorough understanding of various embodiments. However, it will be apparent to one of ordinary skill in the art that the concepts of the present invention may be practiced without one or more of these specific details.

The disclosed techniques can be used to automatically detect run anomalies in experiments conducted using microwell plates. In a training phase, a training application generates a trained classifier based on a heatmap set, where each heatmap set specifies one or more heatmaps associated with a different microwell plate. For example, the heatmap set for a given microwell plate can include a cell count heatmap and any number of intensity heatmaps, where each intensity heatmap is associated with a different fluorescent dye. The cell count heatmap can specify the number of cells in each well of the microwell plate used during the experiment. The given intensity heatmap can specify the average intensity of each well used when excited via the associated fluorescent dye.

Importantly, performance artifacts associated with a microwell plate often manifest as low-frequency spatial patterns in one or more of the associated heatmaps. Thus, for each heatmap, the training application applies a wavelet transform to the heatmap to determine a set of low-frequency spatial patterns. For each microwell plate, the training application generates a feature vector based on the set of associated low-frequency spatial patterns.

The training application runs a clustering algorithm to divide the feature vectors into clusters. The feature vectors in each cluster are more similar to each other than to the feature vectors of other clusters. The training application generates a different label for each cluster that can be overridden via a graphical user interface ("GUI"). For example, the training application can automatically generate a label "L1" for a cluster associated with a microwell plate, where the fourth row from the bottom has fewer cells and lower intensity values compared to the other rows of the microwell plate. The label "L1" can be updated via the GUI to the label "row default". Based on the feature vectors and associated labels, the training application trains a classifier to map the feature vectors to a predicted label and associated label confidence.

Then, in the inference phase, the experiment analysis application uses the trained classifier to detect and evaluate anomalies in execution associated with experiments conducted through the set of micro-well plates based on the heatmap set for the set of micro-well plates. The experiment analysis application generates an average heatmap set representative of the entire experiment based on the heatmap set for the set of micro-well plates. The experiment application inputs each heatmap set associated with an experiment (including the average heatmap set) into the trained classifier to generate a predicted label and a label confidence.

For each micro-well plate in the set of micro-well plates, the experiment analysis application calculates an anomaly score indicating how far the micro-well plate deviates from the cluster associated with the predicted label. For the entire experiment, the experiment analysis application calculates a matching plate percentage, which indicates the percentage of micro-well plates associated with the experiment that have a predicted label equal to the predicted label of the entire experiment. The experiment analysis application then provides any number of predicted labels, label confidences, anomaly scores, and matching plate percentages, in any combination, to any number of software applications and/or displays via a GUI. In this way, the experiment analysis application classifies the entire experiment and each of the associated micro-well plates in a consistent and objective manner with respect to execution artifacts.

System Overview Figure 1 is a conceptual diagram of a system configured to implement one or more aspects of various embodiments. For purposes of illustration, multiple instances of similar objects are indicated with a reference number that identifies the object and, where appropriate, an alphanumeric character(s) in parentheses that identifies the instance. As shown, the system 100 includes, without limitation, computational instances 110(1) and 110(2), display devices 108(1) and 108(2), an unlabeled training data set 102, and an experimental data set 106.

In some embodiments, the system 100 may include, without limitation, any combination of any number of compute instances 110, any number of display devices 108, any number of unlabeled training data sets 102, and any number of experimental data sets 106. Components of the system 100 may be distributed across any number of shared geographic locations and/or any number of different geographic locations, and/or implemented in one or more cloud computing environments (i.e., encapsulating shared resources, software, data, etc.).

As shown, compute instance 110(1) includes, but is not limited to, processor 112(1) and memory 116(1), and compute instance 110(2) includes, but is not limited to, processor 112(2) and memory 116(2). Computation instances 110(1) and 110(2) are also referred to herein individually as "computation instance 110" and collectively as "computation instance 110". Processors 112(1) and 112(2) are also referred to herein individually as "processor 112" and collectively as "processor 112". Memory 116(1) and 116(2) are also referred to herein individually as "memory 116" and collectively as "memory 116". Each of compute instances 110 may be implemented in a cloud computing environment, as part of any other distributed computing environment, or in a stand-alone manner.

The processor 112 may be any instruction execution system, apparatus, or device capable of executing instructions. For example, the processor 112 may include a central processing unit, a graphics processing unit, a controller, a microcontroller, a state machine, or any combination thereof. The memory 116 of the compute instance 110 stores content such as software applications and data for use by the processor 112 of the compute instance 110. In some embodiments, each of any number of compute instances 110 may include any number of processors 112 and any number of memories 116 in any combination.

Each of the computational instances 110 may be implemented in a cloud computing environment, as part of any other distributed computing environment, or in a stand-alone manner. In particular, any number of computational instances 110 (including one) may provide a multiprocessing environment in any technically feasible manner.

Memory 116 can be one or more of readily available memory, such as random access memory, read-only memory, floppy disk, hard disk, or any other form of digital storage, local or remote. In some embodiments, storage (not shown) can supplement or replace memory 116. Storage can include any number and type of external memory accessible to processor 112. For example, without limitation, storage can include secure digital cards, external flash memory, portable compact disk read-only memory, optical storage, magnetic storage, or any suitable combination of the above.

In some embodiments, a computation instance 110 may be associated with any number (including zero) and/or types of input devices, output devices, and/or input/output devices, in any combination. An input device is any device capable of receiving input from a user. Some examples of input devices include, but are not limited to, a keyboard, a mouse, a trackpad, a microphone, a video camera, etc. An output device is any device capable of providing output to a user. Some examples of output devices include, but are not limited to, a display device 108, headphones, speakers, etc. An input/output device is any device capable of both receiving input from a user and providing output to a user, such as a touch screen.

As shown, in some embodiments, computation instance 110(1) is associated with display device 108(1) and computation instance 110(2) is associated with display device 108(2). Display devices 108(1) and 108(2) are also referred to herein individually as "display device 108" and collectively as "display device 108". Display device 108 can be any device capable of displaying images and/or other types of visual content. Some examples of display device 108 include, but are not limited to, a liquid crystal display, a light emitting diode display, a projection display, a plasma display panel, and the like. In some embodiments, display device 108 is a touch screen capable of displaying visual content and receiving input (e.g., from a user).

In some embodiments, the computing instance 110 can integrate any number and/or type of other devices (e.g., other computing instances 110, input devices, output devices, input/output devices, etc.) into a user device. Some examples of user devices include, but are not limited to, desktop computers, laptops, smartphones, smart televisions, game consoles, tablets, etc.

Generally, each of the compute instances 110 is configured to implement one or more applications. For purposes of explanation only, each application is described as residing in the memory 116 of a single compute instance 110 and executing on the processor 112 of a single compute instance 110. However, in some embodiments, the functionality of each application may be distributed across any number of other applications residing in the memory 116 of any number of compute instances 110 and executing in any combination on the processors 112 of any number of compute instances 110. Furthermore, the functionality of any number of applications may be integrated into a single application.

In some embodiments, the number of applications and/or portions of the applications are stored on one or more non-transitory computer readable media. The term "non-transitory" as used herein is a limitation of the medium itself (i.e., tangible, not signal) as opposed to a limitation on persistence of data storage (e.g., RAM vs. ROM). Non-transitory computer readable media are also referred to herein as "computer readable media." For example, in some embodiments, memory 116(1) is a computer readable medium and the number of applications and/or portions of the applications are stored in memory 116(1). In the same or other embodiments, memory 116(2) is a computer readable medium and the number of applications and/or portions of the applications are stored in memory 116(2).

In some embodiments, the number of applications and/or portions of the applications are stored on one or more computer readable media before being stored in memory 116(1) and/or memory 116(2). For example, in some embodiments, the number of applications and/or portions of the applications are stored on a machine (e.g., a server machine) and the number of applications and/or portions of the applications are downloaded from the machine to memory 116(1) and/or memory 116(2). In the same or other embodiments, the number of applications and/or portions of the applications are stored on some form of portable computer readable medium and the number of applications and/or portions of the applications are downloaded from the portable computer readable medium to memory 116(1) and/or memory 116(2). Examples of portions of portable computer readable media include, but are not limited to, digital video disks, memory disks, memory sticks, etc.

In some embodiments, aspects of the disclosure may take the form of a computer program product embodied in one or more computer readable media having computer readable program codecs embodied therein. Any combination of one or more computer readable media may be utilized. Each computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Further specific examples (non-exhaustive enumerations) of computer readable storage media would include the following: an electrical connection having one or more communication lines, a portable computer diskette, a hard disk, a random access memory, a read only memory, an erasable programmable ROM, or a flash memory), an optical fiber, a portable compact disk read only memory, an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium capable of containing or storing a program for use by or in connection with an instruction execution system, apparatus, or device.

The computational instance 110 is configured to detect and facilitate root cause analysis of execution artifacts in experiments involving microwell plates performed using high throughput screening. As previously described herein, in a typical experiment, an integrated system automatically performs tests using a set of microwell plates. As referred to herein, a microwell plate can be, without limitation, any plate that includes a 2D grid of wells, each capable of holding a limited volume. Microwell plates are also commonly referred to as microplates, multiwell plates, and multiwell culture plates.

Each experiment may include, but is not limited to, any number of chemical, biological, genetic, and/or pharmacological tests, with each test being performed in a different well. In a typical experiment, the assignment of tests to wells is randomized within each microwell plate. After a test is completed, various measurements are performed on the wells and the results are stored in the experiment's dataset as a 2D array of measurements. Because the assignment of tests is randomized, the distribution of measurements within each 2D array is ostensibly random.

One challenge associated with high throughput screening is that experimental data sets may contain certain errors known as "execution artifacts" that originate from the experiment itself. Some of the root causes of execution artifacts include, but are not limited to, instrument issues, calibration issues, and environmental changes. As one skilled in the art will recognize, these types of situations often manifest as low frequency spatial patterns in one or more of the 2D arrays of measurements. As a general problem, it is much more difficult to draw valid conclusions about an experiment when the quality of the associated experimental data set is significantly degraded by execution artifacts. Thus, various attempts have been made to identify and mitigate execution artifacts in experimental data sets.

In a review-based approach to identifying execution artifacts, human reviewers visually analyze heat maps to attempt to detect anomalous patterns in measurements that are indicative of execution artifacts. One drawback of such review-based approaches is that manually analyzing heat maps is time-consuming and error-prone. As a result, execution anomalies reflected in the heat maps may be overlooked or misinterpreted. Another drawback is that identifying execution artifacts is a subjective process. Thus, even for heat maps that show similar visual patterns, the type and/or number of execution artifacts identified may vary across reviewers.

Automatic Identification of Performance Artifacts Based on Spatial Information To address the above problems, computational instance 110(1) includes, but is not limited to, a training application 120. As described below, training application 120 generates a trained classifier 170 that automatically maps feature vectors 138 associated with micro-well plates to predicted labels 186 indicative of the type and/or severity of performance artifacts associated with the micro-well plates. In some embodiments, trained classifier 170 also generates a label confidence 188 that correlates to the likelihood that predicted label 186 is accurate.

For purposes of explanation only, "feature vector 138" refers to any instance of feature vector 138, regardless of whether the particular instance is shown in any figure. "Predicted label 186" refers to any instance of predicted label 186, regardless of whether the particular instance is shown in any figure. "Label confidence 188" refers to any instance of label confidence 188, regardless of whether the particular instance is shown in any figure.

The training application 120 resides in the memory 116(1) of the computational instance 110(1) and executes on the processor 112(1) of the computational instance 110(1). In some embodiments, the training application 120 generates a trained classifier 170 based on an unlabeled training data set 102. As shown, the unlabeled training data set 102 includes, but is not limited to, heatmap sets 104(1)-104(H), where H can be any positive integer. Each of the heatmap sets 104(1)-104(H) is associated with a different microwell plate, and the H microwell plates can be associated with any number of previously performed experiments. For purposes of explanation only, "heatmap set 104" refers to any instance of the heatmap set 104 (including each of the heatmap sets 104(1)-104(H)), regardless of whether a particular instance is shown in any figure.

Heatmap set 104 includes, without limitation, F heatmaps (not shown in FIG. 1 ), where F is an integer equal to or greater than 1. Each heatmap is a 2D array of measurements or a visual representation of a 2D array of measurements, each measurement corresponding to a different well contained in an associated microwell plate. As referred to herein, a “measurement” can be derived based on the number and/or type of measurements performed on any associated microwell plate. In general, the spatial location of measurements included in a heatmap correlates to the spatial location of the corresponding well within a 2D grid of wells contained in the associated microwell plate. In some embodiments, each heatmap can indicate the number and/or type of any values in any technically feasible manner.

In some embodiments, each heat map is replaced with a measurement array, and the heat map set 104 is replaced with a set of measurement arrays. Each measurement array may include, but is not limited to, any number of measurements. For each measurement included in a given measurement array, the measurement array indicates the corresponding well of the associated microwell plate in any technically feasible manner. In some other embodiments, the heat map set 104 is replaced with any number of measurements, and for each measurement, the type of measurement and the corresponding well of the associated microwell plate are indicated in any technically feasible manner. The techniques described herein may be modified as appropriate.

One well-known type of execution artifact is the "edge artifact" associated with wells at the periphery of a microwell plate. The results of any tests performed in wells located at the periphery of a microwell plate are typically corrupted by physical and environmental variations (such as evaporation). For this reason, in some embodiments, tests are not performed in wells at the periphery of a microwell plate. Thus, the size of the 2D grid of wells contained in the microwell plate is larger than the size of the associated heat map. For example, in some embodiments, each microwell plate includes, without limitation, a 32 x 48 grid of wells, and each of the heat maps is a measurement of a 30 x 46 array.

Each heatmap included in heatmap set 104 corresponds to a different type of measurement performed on the associated microwell plate. For example, in some embodiments, each heatmap set 104 includes, but is not limited to, a heatmap corresponding to cell count and six heatmaps corresponding to six different imaging channels. Each heatmap may be generated in any technically feasible manner. For example, in some embodiments, the heatmap corresponding to an imaging channel identifies the average intensity of each well when excited via a fluorescent dye associated with the imaging channel.

The training application 120 obtains the unlabeled training data set 102 in any technically feasible manner. For example and without limitation, the training application 120 may read the unlabeled training data set 102 from the memory 116, receive the unlabeled training data set 102 as input, etc. In some embodiments, the training application 120 performs any number (including zero) and/or type of preprocessing operations on the unlabeled training data set 102, without limitation. Some examples of types of preprocessing operations include, without limitation, interpolation of undefined measurements, clipping of extreme measurements, and normalization within and/or across the heatmap sets 104(1)-104(H).

As shown, the training application 120 includes, without limitation, feature engines 130(1)-130(H) (where H is the total number of heatmap sets 104 included in the unlabeled training dataset 102), a clustering engine 140, a labeling engine 150, and a training engine 160. The feature engines 130(1)-130(H) are different instances of a single feature engine 130 (not explicitly shown). For purposes of explanation only, "feature engine 130" as used herein refers to any instance of feature engine 130, regardless of whether a particular instance is shown in any figure.

As shown, the training application 120 inputs the heatmap sets 104(1)-104(H) to the feature engines 130(1)-130(H), respectively. In response, the feature engines 130(1)-130(H) output feature vectors 138(1)-138(H), respectively. In some embodiments, the training application 120 includes fewer than H instances of the feature engine 130, and the training application 120 inputs the heatmap sets 104(1)-104(H) to any number of instances of the feature engine 130 sequentially, simultaneously, or in any combination thereof. For example, in some embodiments, the training application 120 inputs the heatmap sets 104(1)-104(H) sequentially to a single instance of the feature engine 130. In response, the single instance of the feature engine 130 sequentially outputs feature vectors 138(1)-138(H).

Feature vectors 138(1)-138(H) may each, without limitation, represent any amount and/or type of information associated with a spatial pattern associated with heatmap set 104(1)-104(H). Thus, feature vectors 138(1)-138(H) may each, without limitation, represent a spatial pattern associated with a different microwell plate. In some embodiments, each of feature vectors 138(1)-138(H) may include, without limitation, any number of spatial features and/or any number of other types of features, in any combination.

The feature engine 130 may perform any number and/or types of operations on the heatmap set 104 to generate a feature vector 138 associated with the heatmap set 104. In some embodiments, as described in more detail below in connection with FIG. 2, the feature engine 130 applies a wavelet transform to each of the heatmaps included in the heatmap set 104 to generate a multi-level wavelet decomposition. The feature engine 130 then extracts features from the two lowest levels of the multi-level wavelet decomposition and concatenates the extracted features to generate the feature vector 138.

As one skilled in the art will recognize, the wavelet transform extracts local spatial information over a particular portion of the heat map, with the two lowest levels of the multi-level wavelet decomposition representing low frequency spatial patterns. Furthermore, as previously described herein, environmental changes associated with an experiment often manifest as low frequency spatial patterns in one or more heat maps. Thus, the feature vector 138 generated based on the wavelet transform correlates to the type and severity of execution artifacts associated with the corresponding micro-well plate.

As shown, the clustering engine 140 generates a cluster set 148 based on the feature vectors 138(1)-138(H). The cluster set 148 includes, without limitation, any number and/or type of clusters (not shown in FIG. 1). The feature vectors 138(1)-138(H) are distributed among the clusters based on the similarity between the feature vectors 138(1)-133(H). Each cluster defines, without limitation, one or more feature vectors 138 that are more similar to each other than the feature vectors 138 included in the other clusters. Each cluster is associated with the microwell plate from which the feature vectors 138 included in the cluster were derived. The clustering engine 140 may perform any number and/or type of clustering algorithms based on the feature vectors 138(1)-138(H) in any technically feasible manner to generate the cluster set 148.

In some embodiments, the clustering engine 140 performs an agglomerative clustering algorithm based on the feature vectors 138(1)-138(H) and an empirically determined total number of clusters. In some embodiments, the clustering engine 140 performs an agglomerative clustering algorithm based on the feature vectors 138(1)-138(H) and a distance threshold. In some other embodiments, the clustering engine 140 performs a centroid-based clustering algorithm (e.g., a k-means algorithm), a density-based clustering algorithm, or a distribution-based clustering algorithm.

The labeling engine 150 generates a label dataset 156 and a labeled training dataset 158 based on the cluster set 148 and the unlabeled training dataset 102. As described in more detail in connection with FIG. 3, in some embodiments, for each cluster included in the cluster set 148, the label dataset 156 includes, but is not limited to, a labeled cluster. Each labeled cluster includes, but is not limited to, a cluster, a cluster label, and optionally a set of average heatmaps. The cluster label specifies a label that uniquely identifies the cluster. The labeling engine 150 can determine the cluster label in any technically feasible manner. For example, in some embodiments, the labeling engine 150 sets the first cluster label to a default integer (e.g., 1) and then increments the integer for each subsequent cluster label.

As described in more detail in connection with FIG. 3, the average heatmap set of a cluster is a global representation of the cluster, including, but not limited to, F average heatmaps, where F is the number of heatmaps included in each of the heatmap sets 104(1)-104(H). The labeling engine 150 may calculate the average heatmap set of a cluster in any technically feasible manner. For example, in some embodiments, the labeling engine 150 sets each of the average heatmaps of a cluster equal to a trimmed average of the corresponding heatmaps included in the heatmap set 104 assigned to the cluster. In some embodiments, the average heatmaps are replaced with average measurement arrays.

In some embodiments, the labeling engine 150 generates a labeled training dataset 158 based on the label dataset 156. The labeled training dataset 158 includes, but is not limited to, feature vectors 138(1)-138(H) and a label (not shown) for each of the feature vectors 138(1)-138(H). The label for a given feature vector 138 is the cluster label of the cluster that includes the feature vector 138.

In some embodiments, as indicated by the dashed arrow, the labeling engine 150 displays the label GUI 152 via the display device 108(1). The labeling engine 150 may generate any type of display via the label GUI 152 for displaying any amount and/or type of data in any technically feasible manner. In some embodiments, the labeling engine 150 may display any portion (including some or all) of the label dataset 156, the labeled training dataset 158, and/or the unlabeled training dataset 102 in any combination at any time via the label GUI 152. For example, in some embodiments, the labeling engine 150 displays a visual representation of the cluster set 148, any number of heatmap sets 104(1)-104(H), and/or an average heatmap set for any number of clusters prior to generating the labeled training dataset 158.

The labeling engine 150 can generate any type of visual representation of the cluster set 148 in any technically feasible manner. In some embodiments, the labeling engine 150 runs a t-distributed stochastic neighborhood embedding ("t-SNE") algorithm based on the feature vectors 138(1)-138(H) to generate a transformed output. The labeling engine 150 then displays a scatter plot of the transformed output, where the points representing the feature vectors 138(1)-138(H) are colored based on the associated cluster label.

In some embodiments, the labeling engine 150 modifies the label dataset 156 based on input received via the label GUI 152. Some examples of modifications the labeling engine 150 can make to the label dataset 156 include, but are not limited to, merging clusters that represent the same type of execution artifact, redistributing the heatmap set 104 (and associated micro-well plates) among clusters, modifying cluster levels, etc. In particular, the cluster labels can be updated to identify the type of spatial pattern, the type of execution artifact, the severity of the execution artifact, etc. After the clustering engine 140 modifies the label dataset 156, the labeling engine 150 modifies and/or regenerates the labeled training dataset 158 to reflect the modifications to the label dataset 156.

In the same or other embodiments, the labeling engine 150 may cause the clustering engine 140 to iteratively revise the cluster set 148 based on any number and/or type of criteria in a technically feasible manner. For example, in some embodiments, the labeling engine 150 may cause the clustering engine 140 to revise parameters associated with the clustering algorithm (e.g., total number of clusters, distance threshold, etc.) based on user feedback. After the clustering engine 140 regenerates the cluster set 148, the labeling engine 150 revise and/or regenerates the label dataset 156 and/or the labeled training dataset 158.

In some embodiments, the label dataset 156 may include, without limitation, any amount of information related to the unlabeled training dataset 102, the cluster set 148, the labeled training dataset 158, and/or the execution artifacts identified via the trained classifier 170, instead of or in addition to the clusters, cluster labels, and average heatmap set. In the same or other embodiments, the labeling engine 150 displays, via the label GUI 152, any amount of information obtained from the unlabeled training dataset 102, the cluster set 148, and/or the labeled training dataset 158 in any technically feasible manner.

As shown, the training engine 160 generates a trained classifier 170 based on the labeled training dataset 158. More precisely, the training engine 160 trains the classifier to map different feature vectors 138 to different predicted labels 186, where each of the predicted labels 186 is equal to one of the cluster labels included in the label dataset 156. In some embodiments, the training engine 160 also trains the classifier to calculate a label confidence 188 for each of the predicted labels 186, where the label confidence 188 correlates with the likelihood that the predicted label 186 is correct.

As one skilled in the art will recognize, the process of training a classifier based on the results of a clustering algorithm is referred to as "inductive clustering," and any amount and type of associated operations are referred to herein as "inductive clustering operations." The training engine 160 may execute any number and/or type of supervised machine learning algorithms to generate any type of trained classifier 170 based on the labeled training dataset 158. In some embodiments, the trained classifier 170 is a trained random forest, a trained neural network, a trained decision tree, a trained support vector machine, or any other technically feasible trained machine learning model.

In some embodiments, the training application 120 may perform any number and/or type of unsupervised machine learning operations, supervised machine learning operations, semi-supervised machine learning operations, and/or reinforcement learning operations in any combination to generate any number and/or type of trained models that jointly map input feature vectors 138 to predicted labels 186. For example, in some embodiments, the unlabeled training data set 102 is replaced with a manually labeled training data set, and the clustering engine 140 and labeling engine 150 are omitted from the system 100.

Advantageously, each of the cluster labels included in the label dataset 156 is associated with any number (including zero) and/or type of execution anomalies, and each of the predicted labels 186 classifies the associated feature vector 138 with respect to the execution anomalies. Thus, the trained classifier 170 automatically classifies the micro-well plate associated with the input feature vector 138 with respect to any number and/or type of execution anomalies associated with the cluster set 148.

In some embodiments, the feature vector 138 associated with the heatmap set 104 can be replaced with any number and/or type of set of features, each representing any amount and type of spatial information associated with the heatmap set 104 relevant to identifying execution artifacts. The training engine 160 generates a trained classifier 170 based on a labeled training dataset 158 that includes, but is not limited to, a set of features in place of the feature vector 138.

In some embodiments, the training application 120 transmits the trained classifier 170, the label dataset 156, and optionally the unlabeled training dataset 102 to the experimental analysis application 180. In some embodiments, the training application 120 transmits any of the trained classifier 170, the label dataset 156, and the unlabeled training dataset 102, in any combination, to any number and/or type of software application instead of or in addition to the experimental analysis application 180. In the same or other embodiments, the training application 120 stores any of the trained classifier 170, the label dataset 156, and the unlabeled training dataset 102, in any combination, in any memory accessible by the experimental analysis application 180 and/or any number of other software applications.

In some embodiments, the training application 120 generates a reference guide based on the label dataset 156 and optionally presents the reference guide to the experiment analysis application 180 and/or any number and/or type of other software applications. For example, in some embodiments, the training application 120 generates a reference guide based on the average heatmap set and cluster labels included in the label dataset 156.

The experiment analysis application 180 uses the trained classifier 170 to detect execution artifacts included in the experiment data set 106 and facilitates analysis of the root cause of the detected execution artifacts. In some embodiments, the experiment analysis application 180 also uses the label data set 156 and, optionally, the unlabeled training data set 102 to facilitate analysis of the root cause of the detected execution artifacts. For purposes of illustration only, for each similar object included in both the training application 120 and the experiment analysis application 180, a prime is added to the alphanumeric character(s) in parentheses that identify the instance included in the experiment analysis application 180. Furthermore, the functionality of each object included in the experiment analysis application 180 that is also included in the training application 120 is the same as the functionality described for the object in the context of the training application 120.

The experimental data set 106 represents the results of a single experiment conducted over a set of microwell plates. The experimental data set 106 includes, but is not limited to, heatmap sets 104(1')-104(E'), where E can be any positive integer. Each of the heatmap sets 104(1')-104(E') represents a different microwell plate associated with the experiment. As indicated in italics, the heatmap sets 104(1')-104(E') are associated with microwell plates designated 1-E, respectively. As previously described herein, the heatmap set 104 includes, but is not limited to, F heatmaps (not shown in FIG. 1), where F is an integer equal to or greater than 1.

For purposes of illustration only, the experimental analysis application 180 is described herein in the context of a single experimental data set 106. In some embodiments, any number (including one) of instances of the experimental analysis application 180 detects execution artifacts contained in any number of experimental data sets 106, either sequentially, simultaneously, or any combination thereof.

The experiment analysis application 180 resides in memory 116(2) of the computational instance 110(2) and executes on processor 112(2) of the computational instance 110(2). In some embodiments, any number of instances of the experiment analysis application 180 may reside in memory 116 of any number of the computational instances 110 and execute on the processors of the computational instances 110. As shown, the experiment analysis application 180 includes, but is not limited to, an input engine 182, feature engines 130(0')-130(E'), trained classifiers 170(0')-170(E'), and an output engine 190.

The input engine 182 acquires the experimental data sets 106 in any technically feasible manner. In some embodiments, the input engine 182 performs pre-processing operations on any number (including zero) and/or type of experimental data sets 106, without limitation. Some examples of the types of pre-processing operations that the input engine 182 can perform may include, but are not limited to, interpolation of undefined measurements, clipping of extreme measurements, and normalization within and/or across heatmap sets 104(1')-104(E').

After obtaining and optionally preprocessing the experimental dataset 106, the input engine 182 generates a heatmap set 104(0') based on the heatmap sets 104(1')-104(E'). The heatmap set 104(0') represents a non-existent "average" micro-well plate associated with the experimental dataset 106. The input engine 182 may generate the heatmap set 104(0') in any technically feasible manner. For example, in some embodiments, the input engine 182 sets each of the F heatmaps included in the heatmap set 104(0') equal to the average of the corresponding heatmaps included in the heatmap sets 104(1')-104(E'). In some other embodiments, the input engine 182 sets each of the F heatmaps included in the heatmap set 104(0') equal to a trimmed average of the corresponding heatmaps included in the heatmap sets 104(1')-104(E'). In some embodiments, the input engine 182 does not calculate the heatmap set 104(0'), and the techniques described herein are modified accordingly.

As shown, the experiment analysis application 180 inputs the heat map sets 104(0')-104(E') to the feature engines 130(0')-130(E'), respectively. In response, the feature engines 130(0')-130(E) output feature vectors 138(0')-138(E'), respectively. In some embodiments, the experiment analysis application 180 includes fewer than E instances of the feature engine 130, and the experiment analysis application 180 inputs the heat map sets 104(0')-104(E') to any number of instances of the feature engine 130 sequentially, simultaneously, or in any combination thereof.

The experimental analysis application 180 inputs feature vectors 138(0')-138(E') to trained classifiers 170(0')-170(E'), respectively. In response, trained classifiers 170(0')-170(E') output predicted labels 186(0)-186(E), respectively. In some embodiments, trained classifiers 170(0')-170(E') also output label confidences 188(0)-188(E), respectively.

In some embodiments, the experiment analysis application 180 includes fewer than E instances of the trained classifier 170, and the experiment analysis application 180 inputs the heatmap sets 104(0')-104(E') to any number of instances of the feature engine 130 sequentially, simultaneously, or in any combination thereof. In the same or other embodiments, the experiment analysis application 180 can configure the trained classifier 170 to generate any number of predicted labels 186, and optionally any number of label confidences 188, based on any number of feature vectors 138 in any technically feasible manner.

In some embodiments, the training application 120 generates any type of trained machine learning model instead of the trained classifier 170. The training application 120 can generate the trained machine learning model based on the labeled training dataset 158 in any technically feasible manner. The experiment analysis application 180 then generates the predicted labels 186(0)-186(E) in any technically feasible manner based on the trained machine learning model. For example, in some embodiments, the experiment analysis application 180 inputs the feature vectors 138(0')-138(E') into any number of instances of the trained machine learning model. In response, the instance(s) of the trained machine learning model output the predicted labels 186(0)-186(E) and, optionally, the label confidences 188(0)-188(E).

Each of the predicted labels 186(0)-188(E) is one of the cluster labels and may be different from any number of the other predicted labels 186. The predicted label 186(0) is a putative classification of the entire experiment associated with the experimental dataset 106, and the label confidence 188(0) correlates to the likelihood that the predicted label 186(0) applies to the entire experiment. The predicted labels 186(1)-186(E) are putative classifications of the microwell plates 1-E, respectively. The label confidences 188(1)-188(E) correlate to the likelihood that the predicted labels 186(1)-186(E) apply to the microwell plates 1-E, respectively.

As described in more detail in connection with FIG. 3, the output engine 190 generates an experiment summary 196 and plate summaries 198(1)-198(E) based on the predicted labels 186(0)-186(E), the label confidences 188(0)-188(E), the label dataset 156, and (optionally) the unlabeled training dataset 102. The experiment summary 196 presents any amount of information about the experiment as a whole, related to performance artifacts or lack thereof. The plate summaries 198(1)-198(E) present any amount of information about microwell plates 1-E, respectively, related to performance artifacts or lack thereof.

In some embodiments, the experimental summary 196 includes, but is not limited to, a predicted label 186(0), a label confidence 188(0), and a percentage of matching plates (not shown in FIG. 1). The output engine 190 sets the percentage of matching plates equal to the percentage of predicted labels 186(1)-186(E) that are equal to predicted label 186(0). In some embodiments, the percentage of matching plates is replaced with a number of matching plates, which indicates how many of the micro-well plates associated with the experimental dataset 106 are also associated with predicted label 186(0).

In the same or other embodiments, the plate summary 198(x) for an integer x between 1 and E includes, but is not limited to, a predicted label 186(x), a label confidence 188(x), and an anomaly score for micro-well plate x. The anomaly score for micro-well plate x indicates how far away micro-well plate x is from the cluster associated with the predicted label 186(x). The output engine 190 may calculate the anomaly score in any technically feasible manner.

For example, in some embodiments, the output engine 190 uses the feature engine 130 to calculate a feature vector 138 associated with the average heatmap set of the cluster associated with the predicted label 186(x). The output engine 190 then calculates a dissimilarity between the feature vector 138(x') and the feature vector 138 associated with the average heatmap set. In some embodiments, the anomaly score is replaced with a similarity score for micro-well plate x, which indicates how similar micro-well plate x is with respect to the cluster associated with the predicted label 186(x).

In some embodiments, the output engine 190 displays an analysis GUI 192 via the display device 108(2), as indicated by the dashed arrow. The output engine 190 can generate any type of display via the analysis GUI 192 for displaying any amount and/or type of data in any technically feasible manner. In some embodiments, the output engine 190 can use, via the analysis GUI 192, the experiment summary 196, the plate summary 198(1)-198(E), the label dataset 156, and/or the unlabeled training dataset 102 in any combination at any time. For example, the output engine 190 can indicate, via the analysis GUI 192, that predicted labels 186(1)-186(E') are assigned to the heatmap sets 104(1')-104(E'), respectively. In some embodiments, the output engine 190 displays a visual representation of the cluster set 148, any number of heatmap sets 104(1)-104(H), and/or the average heatmap set for any number of clusters in any technically feasible manner.

In some embodiments, the output engine 190 causes the training application 120 to iteratively modify and/or regenerate the cluster set 148, the label dataset 156, the labeled training dataset 158, and/or the trained classifier 170 based on input received via the analysis GUI 192. Some examples of modifications that the output engine 190 may cause the training application 120 to perform based on input received via the analysis GUI 192 include, without limitation, merging clusters, redistributing heatmap sets (and associated microwell plates) among clusters, modifying cluster labels, etc.

Advantageously, because the trained classifier 170 automatically and objectively classifies the heatmap sets 104(1')-104(E'), the trained classifier 170 can be used to efficiently and accurately identify execution artifacts in the experiments associated with the experimental dataset 106. Furthermore, the anomaly scores provide insight into whether each micro-well plate belongs to a cluster without significant execution artifacts, belongs to a cluster of micro-well plates with known execution artifacts, or is associated with a new type of execution artifact or other anomaly. Thus, the anomaly scores facilitate root cause analysis.

Furthermore, because the experiment summary 196 and plate summaries 198(1)-198(E) present objective information regarding execution anomalies, the experiment summary 196 and plate summaries 198(1)-198(E) can be used to efficiently detect trends related to execution anomalies over time and across experiments. Based on the detected trends, a user can make modifications to the experimental process and/or equipment to reduce the number of execution anomalies included in future experimental data sets 106.

It should be noted that the techniques described herein are illustrative rather than limiting and may be modified without departing from the broader spirit and scope of the present invention. Many modifications and variations of the functionality provided by the training application 120, feature engine 130, clustering engine 140, labeling engine 150, trained classifier 170, experimental analysis application 180, input engine 182, and output engine 190 will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.

It will be understood that the system 100 depicted herein is illustrative and that variations and modifications are possible. For example, in some embodiments, the functionality provided by the labeling engine 150 described herein is integrated into the clustering engine 140. In the same or other embodiments, the functionality provided by the training application 120 and the functionality provided by the experiment analysis application 180 are integrated into a single application. Additionally, the connection topology between the various components of FIG. 1 can be modified as desired.

2 is a more detailed diagram of the feature engine 130 of FIG. 1, in accordance with various embodiments. As shown, the feature engine 130 generates a feature vector 138 based on the heatmap set 104. For illustrative purposes only, the heatmap set 104 reflects measurements associated with seven different types of measurements of a microwell plate having 30×46 used wells. As a result, the heatmap set 104 includes, without limitation, a total of 9660 measurements (not shown).

As shown, the heatmap set 104 includes, without limitation, heatmaps 210(0)-210(6). For illustrative purposes only, as shown in italics, heatmap 210(0) is a 2D array including, without limitation, 1380 measurements specifying cell counts for 1380 used wells. Each of heatmaps 210(1)-210(6) is a 2D array including, without limitation, 1380 measurements specifying intensities for 1380 used wells in different imaging channels. In other embodiments, the heatmap set 104 can include, without limitation, any number of heatmaps 210, and each of the heatmaps 210 can correspond to any type of measurement.

The feature engine 130 includes, but is not limited to, spatial information extractors 220(0)-220(6) and an aggregation engine 280. The spatial information extractors 220(0)-220(6) generate spatial feature sets 270(0)-270(6), respectively, based on the heat maps 210(0)-210(6). In some embodiments, the feature engine 130 can include any number of instances of the spatial information extractor 220, which can generate spatial feature sets 270(0)-270(6) simultaneously, sequentially, or in any combination thereof, based on the heat maps 210(0)-210(6), respectively. In the same or other embodiments, each of the spatial feature sets 270(0)-270(H) can include, but is not limited to, any number of spatial features and/or any number of other types of features, in any combination. Each of spatial feature sets 270(0)-270(6) is also referred to herein as a "spatial feature set."

As shown, spatial information extractor 220(0) includes, but is not limited to, a pre-processor 230(0), a wavelet transform 240(0), a multi-level wavelet decomposition 250(0), and a feature extractor 260(0). In general, spatial information extractor 220(y) includes, but is not limited to, a pre-processor 230(y), a wavelet transform 240(y), a multi-level wavelet decomposition 250(y), and a feature extractor 260(y), for integer y between 0 and 6. Pre-processor 230 is shown using a dashed box to indicate that pre-processor 230 is optional.

The pre-processor 230(y) performs any number and/or type of pre-processing operations on the heatmap 210(y). Some examples of types of pre-processing operations the pre-processor 230(y) can perform include, without limitation, interpolation of undefined measurements, clipping of extreme measurements, and normalization within the heatmap 210(y). In some embodiments, the pre-processor 230(y) is omitted from the spatial information extractor 220(y). In the same or other embodiments, the feature engine 130, the training application 120, and/or the input engine 182 can perform any number and/or type of processing operations on the heatmap set 104.

After the pre-processor 230(y) pre-processes the heat map 210(y), the spatial information extractor 220(y) applies a wavelet transform 240(y) to the heat map 210(y) to generate a multi-level wavelet decomposition 250(y). The spatial information extractor 220(y) can apply any type of wavelet transform 240(y) to the heat map 210(y) in any technically feasible manner. For example, in some embodiments, the spatial information extractor 220(y) performs a function call to a 2D discrete wavelet transform function included in a wavelet transform module. Through the function call, the spatial information extractor 220(y) configures the 2D discrete wavelet transform function to extrapolate the heat map 210(y) using reflect padding before computing the herwavelet transform using a cascaded filter bank algorithm. The function call returns a multi-level wavelet decomposition 250(y) that includes, but is not limited to, the approximation, horizontal detail, vertical detail, and diagonal detail coefficients.

The feature extractor 260(y) generates the spatial feature set 270(y) based on the multi-level wavelet decomposition 250(y). The feature extractor 260(y) may generate the spatial feature set 270(y) in any technically feasible manner. In some embodiments, the feature extractor 260(y) extracts any number and/or type of features from the multi-level wavelet decomposition 250 and then performs any number and/or type of post-processing operations on the extracted features to generate the spatial feature set 270(y).

For example, in some embodiments, feature extractor 260(y) extracts features from the two lowest levels of multilevel wavelet decomposition 250(y) to generate spatial feature set 270(y). In some embodiments, feature extractor 260(y) extracts features from the two lowest levels of multilevel wavelet decomposition 250(y) and then normalizes each of the extracted features to generate spatial feature set 270(y). In other embodiments, feature extractor 260(y) extracts a set of features from the three lowest levels of multilevel wavelet decomposition 250(y), compresses the set of features using principal component analysis, and sets spatial feature set 270(y) equal to the resulting compressed set of features. The set of features is also referred to herein as a "feature set."

As shown, aggregation engine 280 generates feature vector 138 based on spatial feature sets 270(0)-270(6). Aggregation engine 280 may generate feature vector 138 in any technically feasible manner. For example, in some embodiments, aggregation engine 280 concatenates spatial feature sets 270(0)-270(6) to generate feature vector 138. Feature vector 138 is also referred to herein as a "set of features."

Facilitating Root Cause Analysis of Execution Artifacts Figure 3 is a more detailed diagram of the output engine 190 of Figure 1, in accordance with various embodiments. As shown, the output engine 190 includes, without limitation, an experiment summary 196 and plate summaries 198(1)-198(E), where E can be any positive integer. The output engine 190 generates the experiment summary 196 and plate summaries 198(1)-198(E) based on the predicted labels 186(0)-186(E), the label confidences 188(0)-188(E), the label dataset 156, and (optionally) the unlabeled training dataset 102.

The label dataset 156 includes, but is not limited to, labeled clusters 340(1)-340(C), where C can be any positive integer. As shown, the labeled cluster 340(C) includes, but is not limited to, a cluster 330(C), a cluster label 342(C), and an average heatmap set 344(C). More generally, the labeled cluster 340(z), where z is an integer from 1 to C, includes, but is not limited to, a cluster 330(z), a cluster label 342(z), and an average heatmap set 344(z). The cluster 330(z) specifies the feature vectors 138 assigned to the cluster 330(z) in any technically feasible manner. As shown in italics, in some embodiments, the cluster 330(z) specifies the list of feature vectors 138 assigned to the cluster 330(z). For purposes of explanation only, clusters 330(1)-330(C) are also referred to herein individually as "cluster 330" and collectively as "cluster 330." Additionally, cluster labels 342(1)-342(C) are also referred to herein individually as "cluster label 342" and collectively as "cluster label 342."

1, predicted label 186(0) and label confidence 188(0) are associated with a non-existent average micro-well plate associated with an experiment conducted via micro-well plates 1-E. Predicted labels 186(1)-186(E) and label confidence 188(1)-188(E) are associated with micro-well plates 1-E, respectively. Each of predicted labels 186(0)-186(E) is equal to one of cluster labels 342(1)-342(C) and may differ from any number of other predicted labels 186.

As shown, in some embodiments, the experiment summary 196 includes, but is not limited to, a predicted label 186(0), a label confidence 188(0), and a percentage of matching plates 310. The output engine 190 may calculate the percentage of matching plates 310 in any technically feasible manner. For example, in some embodiments, the output engine 190 sets the percentage of matching plates equal to the percentage of predicted labels 186(1)-186(E) that are equal to the predicted label 186(0).

The plate summary 198(x) for an integer x between 1 and E includes, but is not limited to, a predicted label 186(x), a label confidence 188(x), and an anomaly score 320(x). The anomaly score 320(x) indicates how far away the micro-well plate x is from the cluster 330 associated with the predicted label 186(x). The output engine 190 may calculate the anomaly score 320(x) in any technically feasible manner.

For illustrative purposes only, in some embodiments, predicted label 186(1) is equal to cluster label 342(C) "row default" and output engine 190 sets anomaly score 320(x) equal to the dissimilarity between feature vector 138(x) and feature vector 138 associated with average heatmap set 344(C). Output engine 190 may calculate feature vector 138 associated with average heatmap set 344(C) and the dissimilarity between feature vector 138(x) and feature vector 138 associated with average heatmap set 344(C) in any technically feasible manner.

As shown, in some embodiments, the output engine 190 generates and displays an analysis GUI 192. For purposes of illustration only, the analysis GUI 192 is shown at a particular point in time based on an exemplary user's input. As shown, the analysis GUI 192 includes, without limitation, a plate slider 370 configured to select none of the micro-well plates, a cluster slider 380 configured to select the labeled cluster 340(C) associated with the row default of the cluster label 342(C), and a cluster display pane 390.

Because the cluster slider 380 is configured to select the labeled cluster 340(C), the output engine 190 populates the cluster display pane 390 with information related to the labeled cluster 340(C). As shown, the output engine 190 displays the average heat map set 344(C) to clearly indicate that a subset of the micro-well plates included in the experiment associated with the experimental dataset 106 are assigned to the cluster 330(C) having the row default cluster label 342(C). More specifically, the output engine 190 identifies that a subset of the micro-well plates including micro-well plates 1, 3, 45-52, and 58 are associated with the cluster label 342(C) and the average heat map set 344(C).

As shown in FIG. 3, average heatmap set 344(C) includes heatmaps 210(0)-210(6) that visually show the average cell counts and average intensities for imaging channels 1-6 of 148 microwell plates assigned to cluster 330(C) by training application 120. In each heatmap 210 shown in FIG. 3, the measurements in the fourth row from the bottom are anomalously low and are a performance artifact associated with row defaults.

Based on the cluster display pane 390, measurements associated with micro-well plates 1, 3, 45-52, and 58 can be excluded from the experimental data set 106. Further, a root cause analysis can conclude that a particular dispense nozzle is partially clogged.

Figure 4 is a flow diagram of method steps for training a classifier to identify execution artifacts in experiments involving microwell plates, according to various embodiments. The method steps are described with reference to the systems of Figures 1-3, but one of ordinary skill in the art will understand that any system configured to perform the method steps in any order is within the scope of the present invention.

As shown, the method 400 begins at step 402, where for each heatmap 210 included in the unlabeled training data set 102, the spatial information extractor 220 generates a distinct spatial feature set 270. At step 404, for each heatmap set 104(x) included in the unlabeled training data set 102, where x is an integer between 1 and H, the aggregation engine 280 generates an associated feature vector 138(x) based on the spatial feature set 270 associated with the heatmap set 104(x).

In step 406, the clustering engine 140 performs a clustering operation based on the feature vectors 138(1)-138(H) to generate a cluster set 148. In step 408, the labeling engine 150 generates a label dataset 156 based on the cluster set 148. In step 410, the labeling engine 150 generates a labeled training dataset 158 based on the label dataset 156, and optionally displays a label GUI 152.

At step 412, the labeling engine 150 determines whether the labeling engine 150 has received any input via the label GUI 152. If at step 412 the labeling engine 150 determines that the labeling engine 150 has not received any input via the label GUI 152, the method 400 proceeds directly to step 416.

However, if at step 412 the labeling engine 150 determines that the labeling engine 150 has not received input via the label GUI 152, then the method 400 proceeds directly to step 414. At step 414, the labeling engine 150 updates any number of the cluster sets 148, the label dataset 156, and/or the labeled training dataset 158 in any combination based on the input.

At step 416, the training engine 160 performs machine learning operations based on the labeled training dataset 158 to generate a trained classifier 170. As referred to herein, a machine learning operation may be any type of operation performed by and/or associated with software that can learn from experience and/or access and use data to learn. Some examples of machine learning operations include, without limitation, unsupervised machine learning operations, supervised machine learning operations, semi-supervised machine learning operations, and reinforcement learning operations.

At step 418, the training application 120 presents the trained classifier 170 and the label dataset 156 to the experimental analysis application 180 and/or any number of other software applications. In some embodiments, the training application 120 also presents the unlabeled training dataset 102 to the experimental analysis application 180 and/or any number of other software applications. The method 400 then ends.

Figure 5 is a flow diagram of method steps for detecting execution artifacts in experiments involving microwell plates using a trained classifier, according to various embodiments. The method steps are described with reference to the systems of Figures 1-3, but one of ordinary skill in the art will understand that any system configured to perform the method steps in any order is within the scope of the present invention.

As shown, the method 500 begins at step 502, where the input engine 182 generates a heatmap set 104(0') representing the entire experiment based on the heatmap sets 104(1')-104(E') included in the experimental dataset 106. At step 504, for each of the heatmaps 210 associated with the experimental dataset 106, the spatial information extractor 220 generates a different spatial feature set 270. At step 506, for each of the heatmap sets 104(x'), where x is an integer between 0 and E, the aggregation engine 280 generates an associated feature vector 138(x') based on the spatial feature set 270 associated with the heatmap set 104(x').

In step 508, the experiment analysis application 180 uses the trained classifier 170 to map each of the feature vectors 138(x'), where x is an integer between 0 and E, to a predicted label 186(x) and, optionally, to a label confidence 188(x). In step 510, the output engine 190 generates an experiment summary 196 and a plate summary 198(1)-198(E) based on the predicted labels 186(0)-186(E) and, optionally, the label confidences 188(0)-188(E), and/or the label dataset 156. In step 512, the experiment analysis application 180 presents any portion of any number of the experiment summaries 196, the plate summaries 198(1)-198(E), and/or the label dataset 156 to any number and/or type of software applications, in any combination. The method 500 then ends.

In summary, the disclosed techniques can be used to train a classifier to accurately and consistently detect anomalies in runs in experiments involving microwell plates. In some embodiments, the training application generates a trained classifier based on an unlabeled training dataset. The unlabeled training dataset includes, but is not limited to, any number of heatmap sets, each heatmap set representing measurements associated with a different microwell plate. The training application includes, but is not limited to, a feature engine, a clustering engine, a labeling engine, and a training engine. The training application configures the feature engine to generate a feature vector for each heatmap set included in the unlabeled training dataset.

To generate a feature vector for a particular heatmap set, the feature engine applies a wavelet transform to each heatmap in the heatmap set to generate a multilevel wavelet decomposition. The feature engine then extracts features from the two lowest levels of each multilevel wavelet decomposition to generate a spatial feature set. The feature engine aggregates the spatial feature sets to generate a feature vector associated with the heatmap set.

The clustering engine performs an agglomerative clustering algorithm based on the feature vectors to generate a cluster set including, but not limited to, clusters of any number of feature vectors. The labeling engine generates a label dataset including, but not limited to, labeled clusters for each cluster. Each labeled cluster includes, but is not limited to, an associated cluster, a cluster label, and a set of average heatmaps representing the cluster. The labeling engine optionally displays a label GUI. The labeling engine can update the label dataset based on input received via the label GUI.

The labeling engine generates a labeled training dataset based on the label dataset. The labeled training dataset includes, but is not limited to, a cluster label associated with each feature vector. Based on the labeled training dataset, the training engine trains a classifier to map the feature vector to a predicted label (i.e., one of the cluster labels) and an associated label confidence. The training application sends the trained classifier and, optionally, the label dataset to an experimental analysis application and/or any number of other software applications.

In some embodiments, the experimental analysis application uses the trained classifier and the label dataset to detect and evaluate performance anomalies in an experimental dataset. The experimental dataset is associated with an experiment performed using a micro-well plate and includes, but is not limited to, any number of heatmap sets. Each heatmap set represents measurements associated with a different micro-well plate. The experimental analysis application includes, but is not limited to, an input engine, a feature engine, a trained classifier, and an output engine.

The input engine generates an "average" heatmap set based on the experimental dataset and uses the feature engine to generate a feature vector for the average heatmap set. The input engine also uses the feature engine to generate a feature vector for each heatmap set in the experimental dataset. The experiment analysis engine inputs each feature vector into a trained classifier to generate a predicted label and a label confidence. The output engine then generates a plate summary and an experiment summary based on the predicted labels, label confidence, and label dataset.

Each plate summary is associated with a different microwell plate and includes, but is not limited to, a predicted label for the associated feature vector, a confidence for the associated label, and an anomaly score. The anomaly score indicates how far the microwell plate is from the cluster associated with the predicted label. The experiment summary represents the entire experiment and includes, but is not limited to, an "average" predicted label for the feature vector associated with the average heatmap set, a confidence for the associated label, and a percentage of matching plates. The percentage of matching plates indicates the percentage of microwell plates associated with the experimental dataset that have a predicted label equal to the average of the predicted labels. The experiment analysis application presents any number of software applications and/or displays any number of plate summaries, experiment summaries, and/or any portion of the label dataset via an analysis GUI.

At least one technical advantage of the disclosed technology over the prior art is that an experiment analysis application can use the trained classifier to more accurately and consistently analyze and detect execution artifacts in experiments involving microwell plates. Because the experiment analysis application uses the trained classifier to automatically classify microwell plates based on the spatial patterns of the associated heat maps, the likelihood that execution anomalies reflected in the heat maps will be overlooked or misinterpreted is reduced compared to prior approaches. Furthermore, computing anomaly scores and average heat map sets for each cluster facilitates both root cause analysis and identification of new types of execution artifacts. Unlike prior art approaches, the trained classifier objectively and consistently classifies microwell plates for execution anomalies over time and for different experiments. As a result, trends in execution anomalies can be efficiently detected and used to improve experimental processes and/or equipment. These technical advantages result in one or more technical improvements over prior art approaches.

1. In some embodiments, a computer-implemented method for detecting execution artifacts in an experiment involving a microwell plate includes calculating one or more sets of spatial features based on one or more heat maps associated with a first microwell plate, aggregating the one or more sets of spatial features to generate a first feature vector, and inputting the first feature vector into a trained classifier, which in response generates a first label indicating that the first microwell plate is associated with a first execution artifact.

2. The computer-implemented method of claim 1, further comprising calculating an anomaly score based on the first feature vector and a first cluster of feature vectors associated with the first label.

3. The computer-implemented method of claim 1 or 2, further comprising: calculating a second feature vector associated with an experiment based on a plurality of heat maps, the plurality of heat maps including the one or more heat maps associated with the first microwell plate; and inputting the second feature vector into the trained classifier to generate a second label that classifies the experiment with respect to a plurality of labels including the first label accordingly.

4. The computer-implemented method of any one of clauses 1 to 3, further comprising: calculating a second feature vector associated with an experiment based on a plurality of microwell plates including the first microwell plate; inputting the second feature vector into the trained classifier, which generates the first label accordingly; and calculating a percentage indicating how many of the microwell plates in the plurality of microwell plates are associated with the first label.

5. The computer-implemented method of any one of clauses 1-4, further comprising: determining that a subset of microwell plates associated with an experiment is also associated with the first label, the subset of microwell plates including the first microwell plate; and displaying, via a graphical user interface ("GUI"), an average heat map associated with the first label; and generating an indication via the GUI that the subset of microwell plates is associated with the average heat map.

6. The computer-implemented method of any one of clauses 1 to 5, further comprising generating an indication via a GUI that the first label is associated with the one or more heatmaps.

7. The computer-implemented method of any one of clauses 1 to 6, wherein the trained classifier further generates a label confidence indicating the likelihood that the first label applies to the first microwell plate.

8. The computer-implemented method of any one of clauses 1 to 7, wherein computing the one or more sets of spatial features includes applying a wavelet transform to a first heat map included in the one or more heat maps to generate a multi-level wavelet decomposition, and extracting a first set of spatial features from at least a lowest level of the multi-level wavelet decomposition.

9. The computer-implemented method of any one of clauses 1 to 8, wherein a first heat map in the one or more heat maps exhibits a plurality of intensities, and each intensity in the plurality of intensities is associated with a different well in the first microwell plate.

10. The computer-implemented method of any one of clauses 1 to 9, wherein the trained classifier comprises a random forest, a neural network, a decision tree, or a support vector machine trained by one or more guided clustering operations.

11. In some embodiments, one or more non-transitory computer-readable media include instructions that, when executed by one or more processors, cause the one or more processors to detect an execution artifact in an experiment involving a micro-well plate by performing the steps of: calculating one or more sets of spatial features based on one or more measurements associated with a first micro-well plate; generating a first feature vector based on the one or more sets of spatial features; and inputting the first feature vector into a trained machine learning model, which in response generates a first label indicating that the first micro-well plate is associated with a first execution artifact.

12. The one or more non-transitory computer-readable media of clause 11, further comprising: calculating an anomaly score based on the first feature vector and a first cluster of feature vectors associated with the first label.

13. The one or more non-transitory computer-readable media of clause 11 or 12, further comprising: calculating a second feature vector associated with an experiment based on a plurality of measurement sequence, the plurality of measurement sequence including the one or more measurement sequence associated with the first microwell plate; and inputting the second feature vector into the trained machine learning model to generate a second label that classifies the experiment with respect to a plurality of labels including the first label accordingly.

14. One or more non-transitory computer-readable media according to any one of clauses 11 to 13, further comprising: calculating a second feature vector associated with an experiment based on a plurality of microwell plates including the first microwell plate; inputting the second feature vector into the trained machine learning model, which generates the first label accordingly; and calculating a percentage indicating how many of the microwell plates in the plurality of microwell plates are associated with the first label.

15. One or more non-transitory computer-readable media according to any of clauses 11-14, further comprising: determining that a subset of microwell plates associated with an experiment is also associated with the first label, the subset of microwell plates including the first microwell plate; and displaying, via a graphical user interface ("GUI"), an average measurement sequence associated with the first label; and generating an indication via the GUI that the subset of microwell plates is associated with the average measurement sequence.

16. One or more non-transitory computer-readable media according to any one of clauses 11 to 15, wherein computing the one or more sets of spatial features to generate the multi-level wavelet decomposition comprises applying a wavelet transform to a first measurement sequence included in the one or more measurement sequence, and extracting a first set of spatial features from at least a lowest level of the multi-level wavelet decomposition.

17. One or more non-transitory computer-readable media according to any one of clauses 11 to 16, wherein calculating the one or more sets of spatial features comprises calculating first spatial information based on a first measurement sequence included in the one or more measurement sequence, and extracting a first set of low frequency spatial features from the first spatial information.

18. One or more non-transitory computer-readable media according to any one of clauses 11 to 17, wherein a first measurement sequence in the one or more measurement sequence specifies a number of cells, each number of cells in the number of cells being associated with a different well in the first microwell plate.

19. The one or more non-transitory computer-readable media of any of clauses 11-18, further comprising: inputting the second feature vector into the trained machine learning model, the first microwell plate being associated with a first experiment and the second microwell plate being associated with a second experiment and a second feature vector, and in response, generating a second label indicating that the second microwell plate is associated with a second execution artifact.

20. In some embodiments, a system includes one or more memories storing instructions; and one or more processors coupled to the one or more memories that, when executing the instructions, perform the steps of: calculating a plurality of spatial features based on a plurality of measurements associated with a first micro-well plate; generating a set of features based on the plurality of spatial features; and calculating a first label based on the set of features and a trained machine learning model, the first label indicating that the first micro-well plate is associated with a first execution artifact.

Any and all combinations of any of the claim elements recited in any claim and/or any of the elements recited in this application, in any way, are within the intended scope of the embodiments and protection.

The descriptions of various embodiments are presented for purposes of illustration, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments.

Aspects of the present embodiments may be embodied as a system, method, or computer program product. Thus, aspects of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, existing software, microcode, etc.), or an embodiment combining software and hardware aspects, which may all be generally referred to as a "module" or "system" or "computer." Additionally, any hardware and/or software techniques, processes, functions, components, engines, modules, or systems described in this disclosure may be implemented as a circuit or set of circuits.

As previously described herein, aspects of the disclosure may take the form of a computer program product embodied in one or more computer readable media having computer readable program codecs embodied therein. Any combination of one or more computer readable media may be utilized. Each computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of computer readable storage media would include an electrical connection having one or more communication lines, a portable computer diskette, a hard disk, a random access memory, a read only memory, an erasable programmable ROM, or a flash memory), an optical fiber, a compact disk read only memory, an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium capable of containing or storing a program for use by or in connection with an instruction execution system, apparatus, or device.

Aspects of the present disclosure are described above with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions can be presented to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to generate a machine. The instructions, when executed via a processor of a computer or other programmable data processing apparatus, enable implementation of the functions/operations specified in the flowchart and/or block diagram block(s). Such a processor can be, but is not limited to, a general purpose processor, a special purpose processor, an application specific processor, or a field programmable gate array.

The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, apparatus, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of code that includes one or more executable instructions for implementing a specified logical function(s). It should also be noted that in some implementations, the functions noted in the blocks may occur in a different order than the order noted in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order depending on the functionality involved. It should also be noted that each block of the block diagrams and/or flowchart diagrams, and combinations of blocks in the block diagrams and/or flowchart diagrams, may be implemented by a special-purpose hardware-based system that performs the specified functions or operations, or a combination of special-purpose hardware and computer instructions.

While the forgoing is directed to embodiments of the present disclosure, other and further embodiments of the present disclosure may be devised without departing from the basic scope thereof, which scope is determined by the following claims.
Preferred embodiments of the present invention will be described below in detail.
EMBODIMENT 1
1. A computer-implemented method for detecting performance artifacts in an experiment involving a microwell plate, comprising:
calculating one or more sets of spatial features based on the one or more heat maps associated with the first microwell plate;
aggregating the one or more sets of spatial features to generate a first feature vector; and
inputting the first feature vector into a trained classifier, which in response generates a first label indicating that the first micro-well plate is associated with a first execution artifact;
The method comprising:
EMBODIMENT 2
2. The computer-implemented method of embodiment 1, further comprising: calculating an anomaly score based on the first feature vector and a first cluster of feature vectors associated with the first label.
EMBODIMENT 3
calculating a second feature vector associated with an experiment based on a plurality of microwell plates including the first microwell plate;
inputting the second feature vector into the trained classifier, whereby generating the first label in response thereto; and
calculating a percentage indicating how many of the micro-well plates in the plurality of micro-well plates are associated with the first label;
2. The computer-implemented method of embodiment 1, further comprising:
EMBODIMENT 4
2. The computer-implemented method of embodiment 1, wherein the trained classifier further generates a label confidence indicating the likelihood that the first label is applied to the first micro-well plate.
EMBODIMENT 5
2. The computer-implemented method of embodiment 1, wherein a first heat map in the one or more heat maps exhibits a plurality of intensities, each intensity in the plurality of intensities being associated with a different well in the first micro-well plate.
EMBODIMENT 6
The trained classifier may be
calculating first spatial information based on a first heat map associated with the first microwell plate;
calculating a first set of features based on the first spatial information; and
performing one or more machine learning operations based on the first set of features to generate the trained classifier, the trained classifier classifying sets of features associated with different micro-well plates with respect to a plurality of labels associated with a plurality of execution artifacts;
2. The computer-implemented method of embodiment 1, wherein the computer-implemented method is trained by:
EMBODIMENT 7
7. The computer-implemented method of embodiment 6, wherein the trained classifier classifies a given set of features of a particular micro-well plate by estimating a label confidence of a label included in the plurality of labels, the label confidence indicating the likelihood that the label applies to the particular micro-well plate.
EMBODIMENT 8
The first spatial information includes a multi-level wavelet decomposition, and computing the first set of features includes:
extracting a first plurality of spatial features from at least a lowest level of the multilevel wavelet decomposition; and
aggregating the first plurality of spatial features with a second plurality of spatial features to generate the first set of features, the second plurality of spatial features being derived from a second heat map also associated with the first micro-well plate;
7. The computer-implemented method of embodiment 6, comprising:
EMBODIMENT 9
7. The computer-implemented method of embodiment 6, wherein the first heat map identifies a plurality of cell counts, each cell count within the plurality of cell counts being associated with a different well within the first microwell plate.
EMBODIMENT 10
7. The computer-implemented method of claim 6, wherein a first machine learning operation included in the one or more machine learning operations includes a supervised machine learning operation, an unsupervised machine learning operation, a semi-supervised machine learning operation, or a reinforcement learning operation.
EMBODIMENT 11
When executed by one or more processors, the one or more processors are
calculating one or more sets of spatial features based on one or more measurements associated with the first micro-well plate;
generating a first feature vector based on the one or more sets of spatial features; and
inputting the first feature vector into a trained machine learning model, which in response generates a first label indicating that the first micro-well plate is associated with a first execution artifact;
One or more non-transitory computer-readable media comprising instructions for detecting execution artifacts in an experiment involving a microwell plate by performing the steps of:
EMBODIMENT 12
calculating a second feature vector associated with an experiment based on a plurality of measurement arrays, the plurality of measurement arrays including the one or more measurement arrays associated with the first micro-well plate; and
inputting the second feature vector into the trained machine learning model, whereby a second label is generated in response that classifies the experiment with respect to a plurality of labels including the first label;
12. The one or more non-transitory computer-readable media of embodiment 11, further comprising:
EMBODIMENT 13
determining that a subset of microwell plates associated with an experiment are also associated with the first label, the subset of microwell plates including the first microwell plate; and
displaying, via a graphical user interface ("GUI"), an average measurement array associated with the first label; and
generating an indication via the GUI that the subset of micro-well plates is associated with the average measurement array;
12. The one or more non-transitory computer-readable media of embodiment 11, further comprising:
EMBODIMENT 14
Calculating the one or more sets of spatial features comprises:
applying a wavelet transform to a first measurement array in the one or more measurement arrays to generate a multi-level wavelet decomposition; and
extracting a first set of spatial features from at least a lowest level of the multilevel wavelet decomposition;
12. One or more non-transitory computer-readable media as recited in embodiment 11, comprising:
EMBODIMENT 15
Calculating the one or more sets of spatial features comprises:
calculating first spatial information based on a first measurement array included in the one or more measurement arrays; and
extracting a first set of low frequency spatial features from the first spatial information;
12. One or more non-transitory computer-readable media as recited in embodiment 11, comprising:
EMBODIMENT 16
The trained classifier comprises:
determining one or more spatial patterns based on a first array of measurements associated with the first microwell plate;
calculating a first set of features based on the one or more spatial patterns; and
performing one or more machine learning operations based on the first set of features to generate a trained classifier, the trained classifier classifying sets of features associated with different micro-well plates with respect to a plurality of labels associated with a plurality of execution artifacts;
12. The one or more non-transitory computer-readable media of embodiment 11, wherein the one or more non-transitory computer-readable media are trained by:
EMBODIMENT 17
17. The one or more non-transitory computer-readable media of embodiment 16, further comprising: prior to performing the one or more machine learning operations, running a clustering algorithm on a set of features to generate the plurality of labels.
EMBODIMENT 18
17. The one or more non-transitory computer-readable media of embodiment 16, wherein determining the one or more spatial patterns comprises applying a wavelet transform to the first array of measurements.
EMBODIMENT 19
17. The one or more non-transitory computer-readable media of embodiment 16, wherein a first machine learning operation included in the one or more machine learning operations includes a supervised machine learning operation, an unsupervised machine learning operation, a semi-supervised machine learning operation, or a reinforcement learning operation.
EMBODIMENT 20
1. A system comprising:
one or more memories for storing instructions;
One or more processors, which when executing the instructions,
calculating a plurality of spatial features based on a plurality of measurements associated with the first microwell plate;
generating a set of features based on the plurality of spatial features; and
calculating a first label based on the set of features and a trained machine learning model, the first label indicating that the first micro-well plate is associated with a first execution artifact;
and the one or more processors coupled to the one or more memories for performing the

Claims (12)

1. A computer-implemented method for detecting performance artifacts in an experiment involving a microwell plate, comprising:
calculating one or more sets of spatial features based on one or more heat maps associated with a first micro-well plate, the first heat map having a plurality of values associated with a plurality of wells of the first micro-well plate ;
generating a first feature vector based on the one or more sets of spatial features; and inputting the first feature vector into a trained classifier, which in response generates a first label indicating that the first micro-well plate is associated with a first execution artifact.
The method comprising: The computer-implemented method of claim 1, further comprising: calculating an anomaly score based on the first feature vector and a first cluster of feature vectors associated with the first label. calculating a second feature vector associated with an experiment based on a plurality of microwell plates including the first microwell plate;
inputting the second feature vector into the trained classifier, whereby generating the first label in response thereto; and calculating a percentage indicating how many of the micro-well plates in the plurality of micro-well plates are associated with the first label.
The computer-implemented method of claim 1 , further comprising: The computer-implemented method of claim 1, wherein the trained classifier further generates a label confidence indicating the likelihood that the first label applies to the first microwell plate. 2. The computer-implemented method of claim 1, wherein the first heat map comprises a plurality of intensity values , each intensity value within the plurality of intensity values being associated with a different well contained within the first micro-well plate. The trained classifier may be
calculating first spatial information based on the first heat map associated with the first micro-well plate;
calculating a first set of features based on the first spatial information; and performing one or more machine learning operations based on the first set of features to generate the trained classifier, the trained classifier classifying sets of features associated with different micro-well plates with respect to a plurality of labels associated with a plurality of execution artifacts.
The computer-implemented method of claim 1 , wherein the training is performed by: The computer-implemented method of claim 6, wherein the trained classifier classifies a given set of features of a particular micro-well plate by estimating a label confidence of a label included in the plurality of labels, the label confidence indicating the likelihood that the label applies to the particular micro-well plate. The first spatial information includes a multi-level wavelet decomposition, and computing the first set of features includes:
extracting a first plurality of spatial features from at least a lowest level of the multi-level wavelet decomposition; and aggregating the first plurality of spatial features with a second plurality of spatial features to generate the first feature set, the second plurality of spatial features being derived from a second heat map also associated with the first micro-well plate.
The computer-implemented method of claim 6 , comprising: The computer-implemented method of claim 6, wherein the first heat map identifies a plurality of cell counts, each cell count in the plurality of cell counts being associated with a different well in the first microwell plate. The computer-implemented method of claim 6, wherein a first machine learning operation included in the one or more machine learning operations includes a supervised machine learning operation, an unsupervised machine learning operation, a semi-supervised machine learning operation, or a reinforcement learning operation. When executed by one or more processors, the one or more processors are
calculating one or more sets of spatial features based on one or more measurements associated with a first micro-well plate, the one or more measurements including a first measurement having a plurality of values associated with a plurality of wells of the first micro-well plate ;
generating a first feature vector based on the one or more sets of spatial features; and inputting the first feature vector into a trained machine learning model, which in response generates a first label indicating that the first micro-well plate is associated with a first execution artifact.
One or more non-transitory computer-readable media comprising instructions for detecting execution artifacts in an experiment involving a microwell plate by performing the steps of: 1. A system comprising:
one or more memories for storing instructions;
One or more processors, which when executing the instructions,
calculating a plurality of spatial features based on a plurality of measurements associated with a first micro-well plate, the measurements being associated with a plurality of wells of the first micro-well plate ;
generating a set of features based on the plurality of spatial features; and calculating a first label based on the set of features and a trained machine learning model, the first label indicating that the first micro-well plate is associated with a first execution artifact.
and the one or more processors coupled to the one or more memories for performing the

Images