WO2025226509A1

WO2025226509A1 - Methods and apparatus for training sample container detection and classification systems

Info

Publication number: WO2025226509A1
Application number: PCT/US2025/025135
Authority: WO
Inventors: Laszlo Szakacs; Yao-Jen Chang; Benjamin S. Pollack; Ankur KAPOOR
Original assignee: Siemens Healthcare Diagnostics Inc
Current assignee: Siemens Healthcare Diagnostics Inc
Priority date: 2024-04-24
Filing date: 2025-04-17
Publication date: 2025-10-30
Anticipated expiration: 2026-10-24

Abstract

A method of creating datasets for training a machine-learning-based sample container detector and classifier for a diagnostic laboratory system includes receiving a raw dataset of training data including a plurality of images of sample containers having different properties; receiving a clean annotation subset of the raw dataset; training an annotation cleaning model using the clean annotation subset, wherein once trained, the annotation cleaning model is configured to annotate images of sample containers; running the raw dataset through the trained annotation cleaning model to generate a clean annotation dataset; applying a clustering algorithm to the clean annotation dataset to identify errors in labels in the clean annotation dataset; correcting identified label errors in the clean annotation dataset so as to generate a clean annotation, clean label dataset. The method may further include training a sample container detector and classifier using the clean annotation, clean label dataset. Numerous other embodiments are provided.

Description

METHODS AND APPARATUS FOR TRAINING SAMPLE CONTAINER DETECTION AND CLASSIFICATION SYSTEMS

CROSS-REFERENCES TO RELATED APPLICATIONS

[0001] This application claims benefit under 35 USC § 119(e) of U.S. Provisional Patent Application No. 63/638,070, filed on April 24, 2024, and European Patent Application 24464007.4, filed on April 24, 2024, the disclosures of which are hereby incorporated by reference herein in their entirety.

FIELD

[0002] The present application relates to sample imaging, and more particularly to methods and apparatus for training sample container detection and classification systems.

BACKGROUND

[0003] Machine learning algorithm performance depends on the quality of the data used during algorithm training. Poor training data generally leads to poor machine learning algorithm performance, while performance improves with training data quality. For this reason, training data curation has become a crucial step in developing useable machine learning algorithms.

[0004] Collecting sufficient data to train machine learning algorithms is costly, time consuming, and prone to error. For example, for image-based object detection and classification machine learning algorithms, thousands to tens of thousands (or more) of images may be required to train across multiple classes of images. Each training image may include one or more annotations that identify objects of interest within the image as well as one or more labels which classify the identified objects within the image. Such a large amount of training data is generally prohibitively time consuming and costly to manually create. As such, computers may be employed to automatically generate training data. However, generated training data may be error prone, including inaccuracies in the location of objects in an image (e.g., noisy annotations) and/or classification of objects within the image (e.g., noisy labels). As such, a need exists for methods and apparatus for training machine-learning based detection and classification systems using training data with noisy annotations and/or noisy labels.

SUMMARY

[0005] In some embodiments, a method of creating datasets for training a machine- learning-based sample container detector and classifier used in a diagnostic laboratory system is provided. The method includes receiving a raw dataset of training data, the raw dataset including a plurality of images of sample containers having different properties; receiving a clean annotation subset of the raw dataset; training, via a processor, an annotation cleaning model using the clean annotation subset, wherein once trained, the annotation cleaning model is configured to annotate images of sample containers; running the raw dataset through the trained annotation cleaning model to generate a clean annotation dataset; applying a clustering algorithm to the clean annotation dataset to identify errors in labels in the clean annotation dataset; and correcting identified errors in labels in the clean annotation dataset so as to generate a clean annotation, clean label dataset.

[0006] In some embodiments, a method of training a sample container detector and classifier for use in a diagnostic laboratory system comprises receiving a raw dataset of training data, the raw dataset including a plurality of images of sample containers having different properties; generating a clean annotation subset of the raw dataset by examining a subset of the raw dataset and correcting erroneous annotations in the subset or adding annotations to unannotated images in the subset; using the clean annotation subset to train a sample container detector to annotate images of sample containers; running, via a processor, the raw dataset through the trained sample container detector to generate a clean annotation dataset; applying, via the processor, clustering to generate a plurality of clusters of images of the clean annotation dataset; identifying images with erroneous labels within one or more clusters; employing neighbor instances of the identified images to correct the erroneous labels of the identified images so as to generate a clean annotation, clean label dataset; and training a sample container detector and classifier using the clean annotation, clean label dataset.

[0007] A system of one or more computers may be configured to perform particular operations or actions by virtue of having software, firmware, hardware, or a combination of them installed on the system that in operation causes or cause the system to perform the actions. In some embodiments, one or more computers may include one or more graphics processing units (GPUs). One or more computer programs may be configured to perform particular operations or actions by virtue of including instructions that, when executed by data processing apparatus, cause the apparatus to perform the actions.

[0008] Other features and aspects of the present invention will become more fully apparent from the following detailed description, the appended claims, and the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

[0009] FIG. 1A illustrates a flow diagram of an example method of creating a dataset for training a machine-learning-based sample container detector and classifier in accordance with one or more embodiments provided herein.

[0010] FIG. 1 B illustrates an example computer in which the method of FIG. 1A may be implemented in accordance with one or more embodiments provided herein. [0011] FIG. 10 illustrates an example method of creating a training dataset for a machine- learning-based sample container detector and classifier in accordance with embodiments provided herein.

[0012] FIG. 2A illustrates an example raw dataset that includes a plurality of annotated images and unannotated images in accordance with one or more embodiments provided herein.

[0013] FIG. 2B illustrates a clean annotation dataset based on the raw dataset of FIG. 2A in accordance with one or more embodiments provided herein.

[0014] FIG. 3 illustrates an example clean annotation dataset after application of a clustering algorithm to the dataset in accordance with embodiments provided herein.

[0015] FIG. 4A illustrates an example cluster of images resulting from application of a clustering algorithm to a clean annotation dataset in accordance with embodiments provided herein.

[0016] FIG. 4B illustrates the cluster of images of FIG. 4A after erroneous labels have been corrected in accordance with embodiments provided herein.

[0017] FIG. 5A illustrates another example cluster of images resulting from application of a clustering algorithm to a clean annotation dataset in accordance with embodiments provided herein.

[0018] FIG. 5B illustrates the cluster of images of FIG. 5A after erroneous labels have been corrected in accordance with embodiments provided herein.

[0019] FIG. 6 illustrates an example of a clean annotation, clean label dataset that includes a plurality of images of sample containers having different sample container properties in accordance with embodiments provided herein.

[0020] FIG. 7 illustrates a diagnostic laboratory system employing a trained sample container detector and classifier in accordance with embodiments provided herein.

[0021] FIG. 8 illustrates a flowchart of a method of creating datasets for training a machine-learning-based sample container detector and classifier used in a diagnostic laboratory system in accordance with embodiments provided herein.

[0022] FIG. 9 illustrates a flowchart of a method of training a sample container detector and classifier for use in a diagnostic laboratory system in accordance with embodiments provided herein.

DETAILED DESCRIPTION

[0023] Independent of the grammatical term usage, individuals with male, female or other gender identities are included within the term.

[0024] As described above, manually creating a sufficient amount of training data for machine-learning-based detection and classification systems is generally prohibitively time consuming and costly. However, computer generated training data may contain annotation and/or label inaccuracies (e.g., noisy annotations and/or noisy labels). Embodiments provided herein allow use of training data that includes noisy annotations and/or noisy labels fortraining machine-learning-based detection and classification systems (e.g. such as for use during sample container localization and classification within a diagnostic laboratory system). Specifically, embodiments provided herein allow use of training data having (1) noisy annotations; (2) noisy labels; and (3) noisy annotations and noisy labels.

[0025] FIG. 1A illustrates a flow diagram 100 of an example method of creating a dataset for training a machine-learning-based sample container detector and classifier (e.g., for use in a diagnostic laboratory system) in accordance with one or more embodiments provided herein. With reference to flow diagram 100, a raw dataset 102 of training data is obtained. For example, the raw dataset 102 may include a plurality of images of sample containers having different properties such as capped or uncapped, different cap colors, different cap shapes, foil sealed, unannotated or annotated (e.g., with bounding boxes), labeled (e.g., with one or more identified properties), or the like. In one or more embodiments, the raw dataset 102 may be computer generated and include one or more of noisy annotations and noisy labels.

[0026] After the raw dataset 102 of training data is obtained, a “clean annotation” subset 104 of the raw dataset 102 having correct or “clean” annotations is generated. For example, a subset of the raw dataset images may be inspected and cleaned so as to ensure that the images contain accurate annotations (e.g., properly sized and/or located bounding boxes around objects of interest within the images such as the tops of sample containers, caps, foil seals, etc.). This clean annotation subset 104 of data is then employed to train an annotation cleaning model 106 configured to annotate images of sample containers. Because the annotation cleaning model 106 is trained on a subset of training data known to contain accurate annotations, it generally produces clean annotations for images.

[0027] Once trained, the annotation cleaning model 106 is employed to annotate the full training dataset of images (raw dataset 102), resulting in a clean annotation dataset 108. That is, the raw dataset 102 may be run through the trained annotation cleaning model 106 to generate the clean annotation dataset 108.

[0028] After the clean annotation dataset 108 is obtained, a clustering algorithm 110 is applied to the clean annotation dataset 108 to identify errors in labels in the clean annotation dataset 108. For example, a clustering algorithm may be employed to cluster similar images in a feature space (e.g., images with similar properties will be close to each other in the same cluster in feature space). Each cluster may be examined to determine if images in the cluster have the same labels. An image within a cluster that has a different label than a majority of its neighbors (e.g., capped versus uncapped, different cap shapes, etc.) may be flagged. As used herein, neighbor instances refer to images of a sample container (e.g., portions of a sample container within a bounding box) that belong to the same cluster and that are close in feature space based on the clustering algorithm.

[0029] Following clustering, identified errors in labels in the clean annotation dataset 108 may be corrected so as to generate a clean annotation, clean label dataset 112. Thereafter, a sample container detector and classifier 114 may be trained using the clean annotation, clean label dataset and deployed (e.g., for use in a diagnostic laboratory system). These and other embodiments are described further below with reference to FIGS. 1 B-9.

[0030] The methods described herein allow use of datasets that include noisy annotations and/or noisy labels for training sample container detector and classifier systems. The methods may include a first portion that accommodates noisy annotations and a second portion that accommodates noisy labels. This allows the methods to be applied to classification systems (e.g., on both image and numerical data) as well as to object detection systems. Additionally, the disclosed methods are architecture independent and do not rely on a selected detection, classification, clustering or similarity measure method. Because there are no such constraints, state-of-the-art architectures and/or detection, classification, clustering or similarity measure methods may be employed.

[0031] FIG. 1 B illustrates an example computer 120 in which the method of FIG. 1A may be implemented in accordance with one or more embodiments. With reference to FIG. 1 B, computer 120 includes a processor 122 coupled to a memory 124. Memory 124 may include raw dataset 102, clean annotation subset 104, annotation cleaning model 106, clean annotation dataset 108, clustering algorithm 110, clean annotation, clean label dataset 112, and, in some embodiments, trained sample container detector and classifier 114. In some embodiments, external storage 125, such as a hard drive, may be employed to store one or more of raw dataset 102, clean annotation dataset 108, and clean annotation, clean label dataset 112. During processing of a dataset, a subset of the dataset may be transferred from external storage 125 to memory 124 for processing and then replaced with another portion of the dataset for processing. This may be repeated numerous times to process an entire dataset. Memory 124 may also include one or more programs 126 for carrying out the methods described herein when executed by processor 122, such as creating training data from a raw dataset having noisy annotations and/or noisy labels, training machine-learning models, running data through trained models, applying clustering algorithms, and/or the like. In some embodiments, processor 122 executing one or more of programs 126 may train annotation cleaning model 106 (e.g., using any machine-learning algorithms employed to create annotation cleaning model 106 as described therein below). Further, in some embodiments, processor 122 executing one or more programs 126 may train sample container detector and classifier 114 (e.g., train any machine-learning algorithms employed to create sample container detector and classifier 114 as described further below). Memory 124 may include multiple memory units and/or types of memory. In some embodiments, all or a portion of memory 124 may be external to and/or remote from computer 120. Additionally, in some embodiments, multiple processors may be employed.

[0032] FIG. 10 illustrates an example method 150 of creating a training dataset for a sample container detector and classifier in accordance with embodiments provided herein. With reference to FIG. 1C, method 150 includes a first portion 151a relating to producing a dataset with clean annotations and a second portion 151b relating to producing a dataset with clean labels. Specifically, the first portion 151a (relating to producing a clean annotation dataset) includes blocks 152, 154, 156, and 158 and the second portion 151 b (relating to producing a clean label dataset) includes blocks 160, 162, and 164. The final dataset (having clean annotations and clean labels) may be used to train a sample container detector and classifier (block 166). The trained sample container detector and classifier may then be used to detect and classify sample containers (block 168).

[0033] With reference to FIG. 1C, in block 152, a raw dataset is obtained that includes a plurality of images of sample containers having different properties such as capped or uncapped, different cap colors, different cap shapes, different diameters or heights, different container types such as tubes or cups, foil sealed, unannotated or annotated (e.g., with bounding boxes), labeled (e.g., with one or more identified properties), or the like. FIG. 2A illustrates an example raw dataset 200 that includes a plurality of annotated images and unannotated images of sample containers 201 in accordance with one or more embodiments. Annotated images are depicted with a thick black border or bounding box (e.g., annotated images 202 with annotations 203). Annotated images that are annotated incorrectly (e.g., their bounding boxes are not properly positioned) are shown surrounded by a dashed box (e.g., incorrectly annotated images 204 with annotations 205). Unannotated images 206 are depicted without a border. In one or more embodiments, the raw dataset may be computer generated and include one or more of noisy annotations (e.g., incorrectly annotated images 204) and noisy labels (e.g., described below with reference to FIG. 4A-5B). In some embodiments, the raw dataset may include hundreds, thousands, or tens of thousands of images. Other (e.g., larger) numbers of images may be employed.

[0034] Referring again to FIG. 1C, after the raw dataset of training data is obtained, in block 154, annotations are added to or corrected for a subset of the raw dataset images to produce a clean annotation subset of training data. For example, a subset of the raw dataset images may be inspected and cleaned (e.g., manually or otherwise corrected), and annotations may be added to any unannotated images in the subset, so as to ensure that the images of the subset contain accurate annotations (e.g., properly sized and/or located bounding boxes around objects of interest within the images such as the tops of sample containers, caps, foil seals, etc.). An example clean annotation subset 210 is shown in FIG. 2B wherein all images have proper annotations (e.g., annotated images 202 that include properly positioned bounding boxes).

[0035] The size of the clean annotation subset will depend upon the complexity of the object(s) to be detected and the number of variations present. In some embodiments, a small sample of each variation may be sufficient (e.g., about 5-10 images for each sample container variation such as each container type, capped or uncapped, cap color, cap type, foil covered, etc.). Other numbers of images may be used for the image subset.

[0036] In block 156 (FIG. 1 C), the clean annotation subset of data is employed to train an annotation cleaning model, such as annotation cleaning model 106 of FIG. 1A, configured to annotate images of sample containers. Any suitable machine-learning algorithm may be employed for the annotation cleaning model. In some embodiments, the annotation cleaning model may include a sample container detector. Suitable algorithms for the annotation cleaning model may include a You Only Look Once (YOLO) algorithm, a convolutional neural network (CNN), a region-based CNN (R-CNN), a fully convolutional neural network (FCN), a region-based FCN (R-FCN), a histography of oriented gradients (HOG) algorithm, a singleshot detection (SSD) algorithm, or the like. Because the annotation cleaning model is trained on a subset of training data known to contain accurate annotations, it generally produces clean annotations for images.

[0037] Once trained, the annotation cleaning model is employed (in block 158 of FIG. 1C) to annotate the full training dataset of images (e.g., raw dataset 102 of FIG. 1A), resulting in a clean annotation dataset (e.g., clean annotation dataset 108 in FIG. 1A). That is, the raw dataset may be run through the trained annotation cleaning model to generate a clean annotation dataset.

[0038] After the clean annotation dataset is obtained, in block 160, a clustering algorithm (e.g., clustering algorithm 110 in FIG. 1A) is applied to the clean annotation dataset to separate the images into a plurality of clusters in feature space. For example, a clustering algorithm such as k-means clustering or another dimensionality reduction and/or grouping algorithm may be employed to cluster similar images. Other example clustering or grouping algorithms that may be employed include t-distributed stochastic neighbor embedding, isomap embedding, locally linear embedding, spectral embedding, multidimensional scaling, latent Dirichlet allocation, or the like.

[0039] FIG. 3 illustrates an example clean annotation dataset 300 after application of a clustering algorithm to the dataset in accordance with embodiments provided herein. As shown in FIG. 3, application of a clustering algorithm results in a plurality of spatially separated clusters. (More generally, the clusters are separated in feature space.) Ideally, each cluster includes images that are all labeled the same (e.g., a first cluster 302a of images of uncapped sample containers all labelled as uncapped sample containers, a second cluster 302b of images of sample containers of a first cap type all labelled as having the first cap type, a third cluster 302c of images of foil sealed containers all labelled as having foil seals, a fourth cluster 302d of images of sample containers of a second cap type all labelled as having the second cap type, and the like). In practice, some clusters may include dissimilarly labelled images. For example, an image of a capped sample container that is erroneously labelled as having no cap may appear in a cluster with an image of a capped sample container that is correctly labelled as having a cap, and vice versa, as described below with reference to FIGS. 4A-5B. [0040] FIG. 4A illustrates an example cluster 400 of images resulting from application of a clustering algorithm (e.g., clustering algorithm 110 of FIG. 1A) to a clean annotation dataset (e.g., clean annotation dataset 108 of FIG. 1A) in accordance with embodiments provided herein. In FIG. 4A, labels (a few identified by reference numeral 401) are depicted by the patterns of the borders used on images (e.g., see legend in FIG. 4A). For example, images labelled as having no cap contain a border with no pattern (e.g., images 402a-g) while images labelled as having a cap have a crosshatched or solid pattern (e.g., images 404a-g).

[0041] While most images within the cluster 400 include correct or “clean” labels that identify the images as being of capped sample containers, some of the images contain erroneous or noisy labels that indicate that they are images of uncapped sample containers (despite being images of capped sample containers). By clustering images in this manner, erroneously labelled images may be easily identified (e.g., based on border pattern, color, or another indicator).

[0042] FIG. 5A illustrates an example cluster 500 that includes images of uncapped sample containers, some of which are erroneously labelled as being capped (as indicated by the dark border/bounding boxes on some images). Specifically, FIG. 5A includes images labelled incorrectly as capped (as depicted by the dark borders/bounding boxes in images 502a-i, for example), as well as images labelled correctly as uncapped (as depicted by the borders/bounding boxes with no patterns in images 504a-i, for example).

[0043] Referring again to FIG. 10, in block 162, each cluster may be examined to determine if images proximate in feature space (e.g., such as images 402a and 404e in FIG. 4A and images 504b and 502c in FIG. 5A) are labelled the same. Images that have different labels than a majority of their neighbors (e.g., capped versus uncapped, different cap shapes, different cap colors, etc.) may be flagged. In some embodiments, nearest neighbors, next nearest neighbors, and/or more distant neighbors within a cluster may be examined. Identification of incorrectly labelled images within each cluster may be performed manually (e.g., via operator inspection) or automatically (e.g., by using an algorithm to examine the cluster and identify the correct label for the cluster, such as by determining which label occurs most frequently in the cluster, and to identify any erroneous labels that differ from the correct label).

[0044] Following clustering (block 160) and identification of erroneous labels within clusters (block 162), in block 164, identified errors in labels in the clean annotation dataset may be corrected so as to generate a clean annotation, clean label dataset (e.g., clean annotation, clean label dataset 112 of FIG. 1A). For example, FIG. 4B illustrates the images and labels of cluster 400 of FIG. 4A (referred to as cluster 400’ in FIG. 4B) after erroneous labels have been corrected (e.g., cleaned). Likewise, FIG. 5B illustrates the images and labels of cluster 500 of FIG. 5A (referred to as cluster 500’ in FIG. 5B) after erroneous labels have been corrected.

[0045] Erroneous labels identified within clusters produced by the clustering algorithm may be corrected either manually or automatically. For example, in some embodiments, an operator may visually inspect the clusters (e.g., clusters 302a-d in FIG. 3) and relabel any erroneous labels. In other embodiments, a computer, such as computer 120 of FIG. 1 B, may employ a majority voting or similar approach to select labels for images. For example, an erroneous label of an image may be replaced with the label employed by a majority of the neighbors of the erroneously labelled image. Other methods of identifying and/or relabeling erroneously labelled images maybe employed.

[0046] FIG. 6 illustrates an example of a clean annotation, clean label dataset 600 that includes a plurality of images of sample containers having different sample container properties in accordance with embodiments provided herein (e.g., capped and uncapped, foil sealed, different cap properties such as different cap shapes, etc.). As shown, each image contains the correct annotation (e.g., bounding box position) and label (e.g., depicted by the border shading of each image).

[0047] Referring again to FIG. 1C, following generation of the clean annotation, clean label dataset, a sample container detector and classifier may be trained using the clean annotation, clean label dataset (block 166), deployed, and used to detect and classify sample containers (block 168), such as in a diagnostic laboratory system. Example machine-learning algorithms that may be used to form the sample container detector and classifier include YOLO-based algorithms, CNNs, R-CNNs, FCNs, R-FCNs, Fast R-CNNs, Faster R-CNNs, SSD algorithms, HOG algorithms, residual neural networks, dense convolutional networks, etc., although other models and/or algorithms may be employed. In some embodiments, the sample container detector and classifier may be formed from a single model, while in other embodiments the sample container detector and classifier may be formed from separate models. In one or more embodiments, all or a portion of blocks 152-166 may be implemented in a computer as program instructions executable by a processor, such as computer 120 and processor 122 of FIG. 1 B.

[0048] FIG. 7 illustrates a diagnostic laboratory system 700 employing a trained sample container detector and classifier in accordance with embodiments provided herein. With reference to FIG. 7, the diagnostic laboratory system 700 may be configured to perform a plurality of analyses or tests on a plurality of different biological samples. For example, the tests may determine levels of constituents or chemicals present in biological samples such as blood, urine, cerebral fluid, and the like. The diagnostic laboratory system 700 may include a plurality of diagnostic instruments 702a-c (e.g., analyzers) that are configured to perform the same or different tests on the biological samples. Fewer or more diagnostic instruments may be employed. In some embodiments, the diagnostic instruments 702a-c may be interconnected by a transport system (e.g., transport system 704). The transport system may be configured to transport the biological samples between the diagnostic instruments 702a-c and/or other devices in the diagnostic laboratory system 700, such as an input module 706 for loading sample carriers into or out of the diagnostic laboratory system 700, a sample prep module 708 (e.g., centrifuges, decappers, etc.), a sample quality check module 710, etc. The diagnostic laboratory system 700 may include other configurations.

[0049] The diagnostic laboratory system 700 may be coupled to a computer 712 that may be located within the diagnostic laboratory system 700 or external to the diagnostic laboratory system 700. In some embodiments, portions of the computer 712 may be located within the diagnostic laboratory system 700 and other portions of the computer 712 may be located external to the diagnostic laboratory system 700. The computer 712 may include a processor 714 and a memory 716, wherein the memory 716 stores one or more programs 718 configured to be executed or run on the processor 714. The memory 716 may also include a trained sample container detector and classifier 720 (e.g., trained with clean annotation, clean label data from workflow 100 of FIG. 1A, method 150 of FIG. 1C, and/or any of the other methods described herein). In some embodiments, the memory 716 and/or the program(s) 718 may be located external to the computer 712. For example, the computer 712 may be connected to the Internet or an intranet to access external data and the like. In some embodiments, the program(s) 718 may operate the diagnostic instruments 702a-c and process data generated by the diagnostic instruments 702a-c.

[0050] In operation, one or more of input module 706, sample prep module 708, sample quality check module 710, and diagnostic instruments 702a-c may include an imaging device (e.g., a camera or similar imaging device, such as imaging device 722 shown within sample quality check module 710) for imaging sample containers (e.g., sample containers 724a and 724b) within the diagnostic laboratory system 700. Thereafter, trained sample container detector and classifier 720 may be employed to identify sample containers in such images and classify the sample containers (e.g., capped or uncapped, cap type, cap color, etc.). Based on such information, computer 712 may determine how sample containers are to be routed through and/or processed within diagnostic laboratory system 700 (e.g., whether a sample container needs to be decapped, whether a foil top needs to be removed, how a sample container should be oriented and/or how fast the sample container should be moved due to it being capped or uncapped, etc.).

[0051] As described above, the methods provided herein allow use of datasets that include noisy annotations and/or noisy labels and may be applied to classification systems (e.g., on both image and numerical data) as well as to object detection systems. Additionally, the disclosed methods are architecture independent and do not rely on a selected detection, classification, clustering or similarity measure method so that state-of-the-art architectures and/or detection, classification, clustering or similarity measure methods may be employed.

[0052] FIG. 8 illustrates a flowchart of a method 800 of creating datasets for training a machine-learning-based sample container detector and classifier used in a diagnostic laboratory system in accordance with embodiments provided herein. With reference to FIG. 8, method 800 begins in block 802 with receiving a raw dataset of training data (e.g., raw dataset 102 of FIG. 1A), the raw dataset including a plurality of images of sample containers having different properties. As stated, in some embodiments, the raw dataset may include noisy annotations and/or noisy labels, as well as some images that are not annotated. Thereafter, in block 804, method 800 includes generating a clean annotation subset (e.g., clean annotation subset 104 of FIG. 1A) of the raw dataset. For example, a subset of the raw dataset may be selected and properly annotated (e.g., each image may be checked for proper annotations, annotations may be added or corrected, or the like). In some embodiments, generating the clean annotation subset may include one or more of manually correcting annotations of an annotated subset of the raw dataset and manually adding annotations to an unannotated subset of the raw dataset.

[0053] After the clean annotation subset is generated, method 800 includes, in block 806, training, via a processor (e.g., processor 122 of FIG. 1 B), an annotation cleaning model (e.g., annotation cleaning model 106 of FIG. 1A) using the clean annotation subset, wherein once trained, the annotation cleaning model is configured to annotate images of sample containers. In some embodiments, the annotation cleaning model may use a YOLO algorithm and/or one or more CNNs, R-FCNs, or R-CNNs. Other object detection algorithms may be used. Once the annotation cleaning model is trained, method 800 includes, in block 808, running the raw dataset through the trained annotation cleaning model to generate a clean annotation dataset (e.g., clean annotation dataset 108 of FIG. 1A). Thereafter, method 800 includes, in block 810, applying a clustering algorithm (e.g., clustering algorithm 110 of FIG. 1A) to the clean annotation dataset to identify errors in labels in the clean annotation dataset, and in block 812, correcting identified errors in labels in the clean annotation dataset so as to generate a clean annotation, clean label dataset (e.g., clean annotation, clean label dataset 112 of FIG. 1A).

[0054] After the clean annotation, clean label dataset is created, method 800 includes, in block 814, training a sample container detector and classifier (e.g., resulting in trained sample container detector and classifier 114 of FIG. 1A) using the clean annotation, clean label dataset. For example, a sample container detector and classifier may be implemented using one or more YOLO algorithms, CNNs, R-CNNs, Fast R-CNNs, Faster R-CNNs, etc., and trained using the clean annotation, clean label dataset. Other models and/or algorithms may be employed. Once the sample container detector and classifier has been trained, it may be put into use (e.g., deployed to detect and classify sample containers). For example, the trained sample container detector and classifier may be employed in diagnostic laboratory system 700 of FIG. 7. Thereafter, the trained sample container detector and classifier may be employed to detect and classify sample containers within the diagnostic laboratory system. In some embodiments, an imaging device (e.g., imaging device 722 or another imaging device) in the diagnostic laboratory system may image a sample container and detect and classify the sample container using the trained sample container detector and classifier. For example, a sample container image may be passed through the trained sample container detector and classifier to determine if the sample container includes a cap or a foil seal, whether the sample container is a quality-control tube, pediatric tube, or cup, one or more properties of a cap of the sample container such as cap material, cap shape, and cap color, etc.

[0055] FIG. 9 illustrates a flowchart of a method 900 of training a sample container detector and classifier for use in a diagnostic laboratory system in accordance with embodiments provided herein. With reference to FIG. 9, method 900 begins in block 902 with receiving a raw dataset of training data (e.g., raw dataset 102 of FIG. 1A), the raw dataset including a plurality of images of sample containers having different properties. Method 900 includes, in block 904, generating a clean annotation subset (e.g., clean annotations subset 104 of FIG. 1A) of the raw dataset by examining a subset of the raw dataset and correcting erroneous annotations in the subset or adding annotations to unannotated images in the subset. Method 900 also includes, in block 906, using the clean annotation subset to train a sample container detector to annotate images of sample containers. Thereafter, method 900 includes, in block 908, running, via a processor (e.g., processor 122 of FIG. 1 B), the raw dataset through the trained sample container detector to generate a clean annotation dataset (e.g., clean annotation dataset 108 of FIG. 1A). In some embodiments, this may include predicting (and/or placing) bounding boxes for sample containers in images of the raw dataset (e.g., to improve bounding box placement within the raw dataset).

[0056] Method 900 then includes, in block 910, applying, via the processor, clustering (e.g., via clustering algorithm 110 of FIG. 1A) to generate a plurality of clusters of images of the clean annotation dataset. Following clustering, method 900 includes, in block 912, identifying images with erroneous labels within one or more clusters, and in block 914, employing neighbor instances of the identified images to correct the erroneous labels of the identified images so as to generate a clean annotation, clean label dataset (e.g., clean annotation, clean label dataset 112 of FIG. 1 A). In some embodiments, correcting images with erroneous labels may include, for each image with an erroneous label, (a) identifying a cluster that includes the image with the erroneous label; (b) identifying neighbors of the image with the erroneous label within the identified cluster; (c) determining an updated label to replace the erroneous label based on a common label of a majority of the identified neighbors of the image with the erroneous label; and (d) replacing the erroneous label of the image with the updated label. Neighbors may include nearest neighbors (e.g., images closest to the image in question) or neighbor images further away but within the same cluster (e.g., images one, two, or more images removed from the image in question).

[0057] Thereafter, method 900 includes, in block 916, training a sample container detector and classifier using the clean annotation, clean label dataset.

NON-LIMITING ILLUSTRATIVE EMBODIMENTS

[0058] The following is a list of non-limiting illustrative embodiments disclosed herein.

[0059] Illustrative embodiment 1. A method of creating datasets for training a machine- learning-based sample container detector and classifier used in a diagnostic laboratory system, comprising: receiving a raw dataset of training data, the raw dataset including a plurality of images of sample containers having different properties; generating a clean annotation subset of the raw dataset; training, via a processor, an annotation cleaning model using the clean annotation subset, wherein once trained, the annotation cleaning model is configured to annotate images of sample containers; running the raw dataset through the trained annotation cleaning model to generate a clean annotation dataset; applying a clustering algorithm to the clean annotation dataset to identify errors in labels in the clean annotation dataset; and correcting identified errors in labels in the clean annotation dataset so as to generate a clean annotation, clean label dataset.

[0060] Illustrative embodiment 2. The method according to the preceding embodiment, further comprising training a sample container detector and classifier using the clean annotation, clean label dataset. [0061] Illustrative embodiment 3. The method according to one of the preceding embodiments, further comprising deploying the trained sample container detector and classifier in a diagnostic laboratory system.

[0062] Illustrative embodiment 4. The method according to one of the preceding embodiments, further comprising employing the trained sample container detector and classifier to detect and classify sample containers within the diagnostic laboratory system.

[0063] Illustrative embodiment 5. The method according to one of the preceding embodiments, wherein employing the trained sample container detector and classifier comprises: imaging a sample container; and detecting and classifying the sample container using the sample container detector and classifier.

[0064] Illustrative embodiment 6. The method according to one of the preceding embodiments, further comprising determining if the sample container includes a cap or a foil seal.

[0065] Illustrative embodiment 7. The method according to one of the preceding embodiments, further comprising determining if the sample container is a quality-control tube, pediatric tube, or cup.

[0066] Illustrative embodiment 8. The method according to one of the preceding embodiments, further comprising determining one or more properties of a cap of the sample container.

[0067] Illustrative embodiment 9. The method according to one of the preceding embodiments, wherein the one or more properties include cap material, cap shape, and cap color.

[0068] Illustrative embodiment 10. The method according to one of the preceding embodiments, wherein the sample container detector and classifier comprises one or more neural networks.

[0069] Illustrative embodiment 11. The method according to one of the preceding embodiments, wherein the sample container detector and classifier includes a sample container detector model and a sample container classifier model.

[0070] Illustrative embodiment 12. The method according to one of the preceding embodiments, wherein the sample container detector and classifier includes a single model for both detection and classification.

[0071] Illustrative embodiment 13. The method according to one of the preceding embodiments, wherein generating the clean annotation subset comprises at least one of: manually correcting annotations of an annotated subset of the raw dataset; and manually adding annotations to an unannotated subset of the raw dataset. [0072] Illustrative embodiment 14. The method according to one of the preceding embodiments, wherein the annotation cleaning model comprises a sample container detector that predicts bounding boxes for sample containers in images.

[0073] Illustrative embodiment 15. The method according to one of the preceding embodiments, wherein applying a clustering algorithm comprises applying one or more of k- means clustering, t-distributed stochastic neighbor embedding, isomap embedding, locally linear embedding, spectral embedding, and multidimensional scaling.

[0074] Illustrative embodiment 16. The method according to one of the preceding embodiments, wherein correcting identified errors in labels comprises, for each image with an erroneous label: identifying a cluster that includes the image with the erroneous label; identifying neighbors of the image with the erroneous label within the identified cluster; determining an updated label to replace the erroneous label based on a common label of a majority of the identified neighbors of the image with the erroneous label; and replacing the erroneous label of the image with the updated label.

[0075] Illustrative embodiment 17. The method according to one of the preceding embodiments, wherein correcting identified errors in labels comprises manually correcting each erroneous label.

[0076] Illustrative embodiment 18. A method of training a sample container detector and classifier for use in a diagnostic laboratory system, comprising: receiving a raw dataset of training data, the raw dataset including a plurality of images of sample containers having different properties; generating a clean annotation subset of the raw dataset by examining a subset of the raw dataset and correcting erroneous annotations in the subset or adding annotations to unannotated images in the subset; using the clean annotation subset to train a sample container detector to annotate images of sample containers; running, via a processor, the raw dataset through the trained sample container detector to generate a clean annotation dataset; applying, via the processor, clustering to generate a plurality of clusters of images of the clean annotation dataset; identifying images with erroneous labels within one or more clusters; employing neighbor instances of the identified images to correct the erroneous labels of the identified images so as to generate a clean annotation, clean label dataset; and training a sample container detector and classifier using the clean annotation, clean label dataset.

[0077] Illustrative embodiment 19. The method according to the preceding embodiment, wherein the trained sample container detector predicts bounding boxes for sample containers in images.

[0078] Illustrative embodiment 20. The method according to one of the preceding embodiments, wherein running the raw dataset through the trained sample container detector improves bounding box placement within the raw dataset. [0079] Illustrative embodiment 21. The method according to one of the preceding embodiments, wherein applying clustering to identify neighbor instances within the clean annotation dataset comprises applying one or more of k-means clustering, t-distributed stochastic neighbor embedding, isomap embedding, locally linear embedding, spectral embedding, and multidimensional scaling.

[0080] Illustrative embodiment 22. The method according to one of the preceding embodiments, wherein employing neighbor instances of the identified images to correct the erroneous labels of the identified images comprises, for each image with an erroneous label: identifying a cluster that includes the image with the erroneous label; identifying neighbors of the image with the erroneous label within the identified cluster; determining an updated label to replace the erroneous label based on a common label of a majority of the identified neighbors of the image with the erroneous label; and replacing the erroneous label of the image with the updated label.

[0081] Illustrative embodiment 23. The method according to one of the preceding embodiments, further comprising deploying the trained sample container detector and classifier to detect and classify sample containers.

[0082] Illustrative embodiment 24. The method according to one of the preceding embodiments, wherein deploying the trained sample container detector and classifier comprises deploying the trained sample container detector and classifier in a diagnostic laboratory system.

[0083] Illustrative embodiment 25. The method according to one of the preceding embodiments, further comprising employing the trained sample container detector and classifier to detect and classify sample containers within the diagnostic laboratory system.

[0084] The foregoing description discloses only example embodiments of the invention; modifications of the above disclosed apparatus and methods which fall within the scope of the invention will be readily apparent to those of ordinary skill in the art. Accordingly, while the present invention has been disclosed in connection with the example embodiments thereof, it should be understood that other embodiments may fall within the spirit and scope of the invention, as defined by the following claims.

Claims

CLAIMS What is claimed is:

1. A method of creating datasets for training a machine-learning-based sample container detector and classifier used in a diagnostic laboratory system, comprising: receiving a raw dataset of training data, the raw dataset including a plurality of images of sample containers having different properties; generating a clean annotation subset of the raw dataset; training, via a processor, an annotation cleaning model using the clean annotation subset, wherein once trained, the annotation cleaning model is configured to annotate images of sample containers; running the raw dataset through the trained annotation cleaning model to generate a clean annotation dataset; applying a clustering algorithm to the clean annotation dataset to identify errors in labels in the clean annotation dataset; and correcting identified errors in labels in the clean annotation dataset so as to generate a clean annotation, clean label dataset.

2. The method of claim 1 further comprising training a sample container detector and classifier using the clean annotation, clean label dataset.

3. The method of claim 2 further comprising deploying the trained sample container detector and classifier in a diagnostic laboratory system.

4. The method of claim 3 further comprising employing the trained sample container detector and classifier to detect and classify sample containers within the diagnostic laboratory system.

5. The method of claim 4 wherein employing the trained sample container detector and classifier comprises: imaging a sample container; and detecting and classifying the sample container using the sample container detector and classifier.

6. The method of claim 5 further comprising determining if the sample container includes a cap or a foil seal.

7. The method of claim 5 further comprising determining if the sample container is a quality-control tube, pediatric tube, or cup.

8. The method of claim 5 further comprising determining one or more properties of a cap of the sample container.

9. The method of claim 8 wherein the one or more properties include cap material, cap shape, and cap color.

10. The method of claim 2 wherein the sample container detector and classifier comprises one or more neural networks.

11. The method of claim 2 wherein the sample container detector and classifier includes a sample container detector model and a sample container classifier model.

12. The method of claim 2 wherein the sample container detector and classifier includes a single model for both detection and classification.

13. The method of claim 1 wherein generating the clean annotation subset comprises at least one of: manually correcting annotations of an annotated subset of the raw dataset; and manually adding annotations to an unannotated subset of the raw dataset.

14. The method of claim 1 wherein the annotation cleaning model comprises a sample container detector that predicts bounding boxes for sample containers in images.

15. The method of claim 1 wherein applying a clustering algorithm comprises applying one or more of k-means clustering, t-distributed stochastic neighbor embedding, isomap embedding, locally linear embedding, spectral embedding, and multidimensional scaling.

16. The method of claim 1 wherein correcting identified errors in labels comprises, for each image with an erroneous label: identifying a cluster that includes the image with the erroneous label; identifying neighbors of the image with the erroneous label within the identified cluster; determining an updated label to replace the erroneous label based on a common label of a majority of the identified neighbors of the image with the erroneous label; and replacing the erroneous label of the image with the updated label.

17. The method of claim 1 wherein correcting identified errors in labels comprises manually correcting each erroneous label.

18. A method of training a sample container detector and classifier for use in a diagnostic laboratory system, comprising: receiving a raw dataset of training data, the raw dataset including a plurality of images of sample containers having different properties; generating a clean annotation subset of the raw dataset by examining a subset of the raw dataset and correcting erroneous annotations in the subset or adding annotations to unannotated images in the subset; using the clean annotation subset to train a sample container detector to annotate images of sample containers; running, via a processor, the raw dataset through the trained sample container detector to generate a clean annotation dataset; applying, via the processor, clustering to generate a plurality of clusters of images of the clean annotation dataset; identifying images with erroneous labels within one or more clusters; employing neighbor instances of the identified images to correct the erroneous labels of the identified images so as to generate a clean annotation, clean label dataset; and training a sample container detector and classifier using the clean annotation, clean label dataset.

19. The method of claim 18 wherein the trained sample container detector predicts bounding boxes for sample containers in images.

20. The method of claim 19 wherein running the raw dataset through the trained sample container detector improves bounding box placement within the raw dataset.

21. The method of claim 18 wherein applying clustering to identify neighbor instances within the clean annotation dataset comprises applying one or more of k-means clustering, t- distributed stochastic neighbor embedding, isomap embedding, locally linear embedding, spectral embedding, and multidimensional scaling.

22. The method of claim 18 wherein employing neighbor instances of the identified images to correct the erroneous labels of the identified images comprises, for each image with an erroneous label: identifying a cluster that includes the image with the erroneous label; identifying neighbors of the image with the erroneous label within the identified cluster; determining an updated label to replace the erroneous label based on a common label of a majority of the identified neighbors of the image with the erroneous label; and replacing the erroneous label of the image with the updated label.

23. The method of claim 18 further comprising deploying the trained sample container detector and classifier to detect and classify sample containers.

24. The method of claim 23 wherein deploying the trained sample container detector and classifier comprises deploying the trained sample container detector and classifier in a diagnostic laboratory system.

25. The method of claim 24 further comprising employing the trained sample container detector and classifier to detect and classify sample containers within the diagnostic laboratory system.