TW202345104A - A system and method for quality check of labelled images - Google Patents

Abstract

Method (200) and systems (100) for identifying mislabeled images from a set of labelled images, for a deep neural network, are described. A sequence of plurality of input labelled images (102) is provided as an input to a segmentation network (116) for generating predictions for each image from said set of labelled images (102). An scoring module (118) is configured to compute two or more scoring functions for each image form the set of images (102) using the predictions generated by the segmentation network (116). A quality check module (120) is configured to configured to identify mislabeled images from the set of labelled images (102) by visualizing said computed two or more scoring functions in multi-dimensional graphical representation.

Description

Systems and methods for quality inspection of tagged images

The present subject matter generally relates to a system and method for a unified interactive framework for identifying marking errors to improve marking procedures, particularly for autonomous driving applications.

Few machine learning techniques use machine learning models that can learn from unlabeled data without any human intervention. This type of machine learning technique is called unsupervised learning. For example, a deep learning model can segment data into groups (or clusters) based on patterns the model finds in the data. These groups can then be used as labels so that the data can be used to train supervised learning models.

State-of-the-art deep learning techniques for supervised tasks in computer vision require labeled training data. Manual labeling is expensive and grows exponentially with the size of the data set, thus bringing significant labeling costs to the industry. Automated semantic segmentation, which involves assigning class labels to each pixel of an image, is an example of a task where obtaining labeled training data is particularly costly. The problem becomes even more acute when the area of concern lies in the realm of autonomous driving. In most areas of developing such autonomous driving capabilities, large amounts of video/imagery sequences are captured by vehicles equipped with different reference sensors covering hundreds of thousands of miles. However, in such solutions as the manual curation of data in the dataset described above, there can be problems when it comes to limited budgets for annotating the data.

Training supervised deep neural networks requires large amounts of clean, labeled data. In order to train such networks, large datasets such as MSCOCO, Mapillary Vistas, Youtube-8M are needed. Conventional methods for processing large amounts of data have focused on unsupervised methods (requiring no labels), weak/semi-supervised methods (requiring partial labels), and (semi-)automatic labeling of data. Such methods can use segment prediction as an example of polygons by annotating a set number of pixels in each iteration. For this annotation, manual intervention is still required. The main focus of this work is to significantly reduce the annotation time of human annotators. While most of these methods succeed in reducing annotation time, they require at least a few iterations of the algorithm to provide human-comparable results.

Current work in the literature focuses on reducing the amount of annotated data required to train a model on the annotated data and use the model to make predictions. After manual detection from humans, the predicted images are further added to the dataset. However, this procedure has flaws. First, the data set can be poorly labeled and thus can lead to poor model performance. Second, each predicted image to be added (in the dataset) needs to be manually verified and becomes a resource-intensive operation. Therefore, solutions are needed to reduce this resource-intensive process by limiting the images to be verified by humans.

The prior art WO2019137196A1 discloses an image annotation information processing method and device, a server and a system. It can provide supervision and judgment processing logic for multiple nodes with different processing results. When there is an error in the image annotation information, the results can be automatically returned so that the operator can review, modify, etc. The operator's professional capabilities can be improved through continuous audit feedback interaction, gradually improving the image annotation efficiency, and greatly improving the training set image annotation accuracy. According to specific examples, annotation quality can be effectively ensured, timely and effective information feedback can be provided in the workflow, and sample image annotation information operation efficiency can be improved.

Another prior art CN105404896A discloses an annotation data processing method and an annotation data processing system. The annotation data processing method includes the following steps: Step S110: Calculate the similarity of multiple annotation results on the annotation task; Step S120: Compare the similarity with the similarity threshold, if the similarity is greater than or equal to the similarity threshold value, the program goes to step S130, and if the similarity is less than the similarity threshold value, the program goes to step S140; Step S130: Determine whether multiple annotation results pass quality detection; and Step S140: Determine multiple annotations. The result is that the quality test fails. According to the annotation data processing method and annotation data processing system, the quality of annotation results is automatically detected by utilizing similarities, so that annotators can obtain the quality of annotation results in a timely manner, and then correct annotation errors in a timely manner, and therefore It can effectively enhance annotation accuracy.

The present invention provides a computing system (100), which includes: a memory (110); and a processor (112) coupled to the memory (110) configured to convert a plurality of labeled images ( A set of 102) is provided to a segmentation network (116), wherein the segmentation network (116) is configured to generate predictions for each image from the set of the plurality of labeled images (102); Characterized by: a scoring module (118) configured to use the predictions generated by the segmentation network (116) to calculate two scores for each image from the plurality of labeled images (102). one or more scoring functions; and a quality inspection module (120) configured to visualize images from the set of labeled images (102) obtained from a multi-dimensional graphical representation patch to identify mislabeled images from the set of labeled images (102), wherein the multi-dimensional graphical representation is obtained from the two or More than two different values of the scoring function are obtained.

The present invention provides a computer-implemented method (200) for identifying mislabeled images from a set of labeled images for a deep neural network, the method (200) comprising the steps of: The segmentation network (116) receives (201) a set of labeled images (102) and generates predictions for each image from the set of labeled images (102); by a score A module (118) that computes (202) two or more scoring functions by using the predictions generated for each of the plurality of labeled images (102); by using a The quality inspection module (120) performs the processing of the plurality of labeled images (102) by visualizing image patches from the set of labeled images (102) obtained from a multi-dimensional graphical representation. Mislabeled images are identified (203) in the collection, wherein the multidimensional graphical representation is obtained from different values of the two or more scoring functions for each image from the plurality of labeled images (102).

1 illustrates a system environment for identifying mislabeled images from a collection of labeled images for a deep neural network, in accordance with an example implementation of the present subject matter. The present subject matter describes various methods for obtaining a correct set of labeled images from a larger set of input labeled images and sending incorrectly labeled images for further labeling. In one example, the set of labeled input images 102 may contain images of autonomous driving scenes with varying semantic layout and content. In one example, the set of labeled input images 102 may be images from a driving scene including traffic signals, vehicles, pedestrians, and the like.

The system environment may include the computing system 100 and the neural network architecture. Computing system 100 may be communicatively coupled to a neural network architecture. In one example, computing system 100 may be coupled to a neural network architecture directly or remotely. Examples of computing systems 100 may include, but are not limited to, laptop computers, notebook computers, desktop computers, and the like.

Computing system 100 may include memory 110 . Memory 110 may include any non-transitory computer-readable medium, including, for example, volatile memory such as static random access memory (SRAM) and dynamic random access memory (DRAM), and/or non-volatile memory , such as read-only memory (ROM), erasable programmable ROM, flash memory, hard disk, optical disk and tape.

In one example, computing system 100 may also include processor 112 coupled to memory 110 . Processor 112 may include a microprocessor, microcomputer, microcontroller, digital signal processor, central processing unit, state machine, logic circuitry, and/or any other device that manipulates signals and data based on computer-readable instructions. Additionally, specialized hardware, as well as hardware capable of executing computer-readable instructions, may be used to provide the functionality of the various elements illustrated in the Figures, including any functional block labeled "processor." Additionally, computing system 100 may include interface 114 . Interface 114 may include various interfaces, such as interfaces for users. Interface 114 may include a data output device. In one example, interface 114 may provide an interactive platform for receiving input images from a user.

In an exemplary implementation of the subject matter of the present invention, a method and system for a unified and interactive framework for identifying markup errors are proposed. The computing system 100 includes a segmentation network 116, a scoring module 118, and a quality inspection module 120. In one specific example, segmentation network 116 is communicatively coupled to deep neural network 300.

Segmentation network 116 is configured to generate predictions for each image from the set of labeled images 102 . In one specific example, segmentation network 116 generates segmentation predictions based on a plurality of classifiers and a plurality of training data from deep neural network 300 . In this specific example, segmentation network 116 is trained to identify pixels belonging to different categories. To generate predictions, each labeled sample of the image is treated as an oracle, but the predictions and labels may be different. This difference can be used to calculate a scoring function between the label and the prediction to be defined for each image segment, which scoring function measures the dissimilarity/similarity between the label and the prediction.

For labeled images, each pixel in the image belongs to a single class derived from the classification structure description. The attribution of pixels to categories is deterministic. For labeled images, a label is provided to each pixel using this class ideology, and therefore the label has no probability value associated with it. This results in a clear/specific identification of the label of the tagged image 102 .

Scoring module 118 is configured to use predictions generated by segmentation network 116 to compute two or more scoring functions for each image from the set of images 102 . In one example, two or more scoring functions may include function types such as metric of performance (IoU) scores, probability scores, uncertainty scores, and/or combinations thereof. In one specific example, performance metric scores may be used to determine the accuracy of segmented network 116 . In one example, the performance metric score may include Intersection over Union (IoU), such as semantic segmentation used to measure similarity scores. In one specific example, scoring module 118 is further configured to calculate a confidence score using probability values from segmentation network 116 regarding the class of a pixel for each image of labeled image 102 set.

In one specific example, to calculate the performance metric (IoU), the scoring module 118 compares the manual labels to the predicted output generated by the segmentation network 116 . In a similar manner, to calculate the uncertainty score (entropy), the scoring module 118 interprets the neural network output into a probability score.

The quality inspection module 120 is configured to identify mislabeled images from the set of labeled images 102 by visualizing image patches from the set of labeled images (102) obtained from the multidimensional graphical representation. Herein, a multidimensional graphical representation is obtained from different values of two or more scoring functions for each image from the labeled image (102). In one specific example, the quality inspection module 120 is further configured to generate a two-dimensional graphical representation using the calculated scoring function to allow faster selection of mislabeled images from the set of labeled images 102 .

The scoring module 118 enables a scoring mechanism that treats labels (from the original annotator) as oracles. The quality check process individually approves the oracle status of each tagged image from the image 102 collection. Therefore, it may be wrong to think of labels as oracles for each instance. This insight suggests another scoring function that ensures a certainty/confidence score for predictions and/or labels generated by the segmentation network 116 .

In one example, segmentation network 116 provides a probability value for each pixel regarding the class, thus enabling the use of uncertainty/confidence scores. The confidence score is used to define the (category-specific) uncertainty score for each prediction produced by the segmentation network 116 . In this example, entropy calculates the uncertainty of the prediction, that is, a high entropy value indicates that the network is uncertain about its prediction, while a low entropy value indicates strong trust. In addition to entropy, other measurement techniques that are variants of the basic entropy definition are also conceivable.

In this particular example, the identified mislabeled images are further sent for relabeling. Herein, the set of mislabeled images is a subset of the set of labeled images 102 .

2 illustrates a flowchart of a method 200 for identifying mislabeled images from a set of labeled images for a deep neural network, implemented in accordance with an example implementation of the present subject matter. The method 200 may be implemented by the computing system 100 of FIG. 1 including the memory 110, the processor 112 and the interface 114. Additionally, computing system 100 may be communicatively coupled with a neural network architecture, as described in FIG. 1 . Although method 200 is described in the context of a system similar to computing system 100 of FIG. 1, other suitable devices or systems may be used to perform method 200.

Referring to FIG. 2 , at block 201 , method 200 may include receiving a plurality of sets of labeled images 102 via segmentation network 116 and generating predictions for each image from the set of labeled images 102 . In one example, the input labeled images 102 may be images from a driving scene including traffic signals, vehicles, pedestrians, etc.

At block 202 , the method 200 may include calculating, by the scoring module 118 , two or more scoring functions using the generated predictions for each of the labeled images 102 .

At block 203 , the method 200 may include identifying mislabeled images from the set of labeled images 102 by using the quality inspection module 120 to visualize image patches from the set of labeled images 102 obtained from the multidimensional graphical representation. . Herein, a multidimensional graphical representation is obtained from different values of the two or more scoring functions for each image from the labeled image 102. In one specific example, method 200 further includes step 204 for generating a multidimensional graphical representation using the calculated scoring function to allow faster selection of incorrectly labeled images from the set of labeled images 102 . In this particular example, the identified mislabeled images provide a compact subset of the sequence of the set of labeled images 102 for further labeling.

In one specific example, at block 205, the method 200 further includes the step of the QC administrator directly selecting regions of the multi-dimensional graphical representation that are sufficiently error-labeled. Here, the QC administrator may select sufficiently mislabeled regions following step 203 of identifying mislabeled images from the set of labeled images 102 . In this particular example, a scatter application displaying a multi-dimensional graphical representation may be presented to the QC administrator for use in selecting areas of sufficient error marking.

Additionally in this particular example, at block 206 , the method 200 further includes the QC staff receiving the regions obtained by the QC administrator that have sufficient error flags in the multidimensional graphical representation and running across each image in the assigned grid. Iteratively and with comments to be sent to indicate that sufficiently mismarked areas need to be remarked for further marking. All images assigned for relabeling are then passed to the labeler for relabeling.

3 is an exemplary graphical representation of the use of a calculated scoring function to allow faster selection of incorrectly labeled images produced by the quality inspection module 120 from a collection of labeled images. In this paper, in Figure 3a, the scoring function is considered as a performance metric score (e.g., IoU) and the confidence score (i.e., uncertainty) can be used to place each image in a two-dimensional scatter plot. This scatter plot is defined by the metric score/IoU on the y-axis and the confidence score/uncertainty on the x-axis. In Figure 3a, since the image is not a point object, the center of the image is aligned with the corresponding IoU and uncertainty score.

Rather than providing a simple image on the scatter plot shown in Figures 3a and 3b, the present invention uses different (color) views of the image, where visualizing errors can be difficult. To enable quickly labeling instances of mislabeling, the scatter plot has regions of interest as shown in Figure 3b, where it is easy to find images with mislabeling and/or incorrect labeling. This stems from the fact that when the network's predictions are inconsistent with the labels (low IoU) but the network maintains trust in its predictions (low uncertainty), the chance of finding a wrong label increases. Similarly, images with high IoU and low uncertainty point to no or small errors (and therefore no QC use case). As shown in Figure 3b, the scatter plot is divided into 4 regions, as shown in Table I, of which region 2 is focused for quality inspection due to incorrectly labeled images. In situations where deep learning networks reflect high confidence scores in predictions, these region boundaries can be severely distorted. But as shown in Figure 3a, such regions are still easily separated from human annotators with minimal effort. Figure 3b shows an exemplary scatter plot for an image patch containing objects (such as vehicles) on the BDD dataset. Here we observe that the region (1, 2, 3, 4) is distorted, but despite this, the structure can be seen in the scatter plot. area uncertainty Low high IOU high Upper left (area 1 ) Upper right (area 4 ) Low Lower left (area 2 ) Lower right (area 3 ) Table I

The present invention uses scatter plots to build interactive user applications. The application provides a region selector tool that can be used to select any desired region. This selector can then be effectively used to select data points with error flags. This selection tool combined with the area of interest in the scatter plot allows faster selection of incorrectly labeled patches. Figure 3 shows the main window of the distributed QC application.

In one embodiment of the present invention, a two-step procedure for identifying mislabeled images for relabeling may be employed. In the first step of this procedure, the quality control (QC) administrator directly uses the scatter QC application and selects sufficiently error-marked regions from the scatter plot shown in Figure 3b. This area of interest is automatically further divided into smaller grids, each of which is passed to another QC worker. This process will be iterated over each image in the assigned grid and marked for re-marking with comments from QC staff. All images assigned for relabeling are then passed to the labeler for relabeling. The Scatter QC application allows bounding box-based region selection on scatter plots. Human-assisted input utilizing this bounding box selection can effectively focus on regions with sufficient error labeling. QC administrators and QC staff further segment and select images for relabeling when focusing on incorrectly labeled areas. In the final phase of Labeling as a Service (LaaS), human annotators relabel patches.

At the core of the invention is a unified system and interactive framework for identifying markup errors. In alternative embodiments of the present invention, application areas of computing system 100 may include semantic segmentation, object detection, classification, and the like.

The present invention focuses on finding erroneous tags from specific categories, rather than finding potential erroneous tags for all categories in the structure description at once. In addition, the present invention provides methods for calculating evaluation metrics (average IoU) and uncertainty scores (average entropy) of sample labeled images. While most literature and patents focus on (semi-)automated labeling and time and click reduction (for the annotator), the present invention focuses on quality inspection procedures. Generally speaking, quality inspection procedures are tedious and repetitive procedures in which errors are assumed to be fewer (compared to corrections to be made in automated labeling), and therefore poor quality labels are likely to pass quality inspection. In contrast, the present invention does not rely entirely on human intervention to perform quality checks on large numbers of labeled images.

Although aspects of the present disclosure have been described in language specific to structural features and/or methods, it should be understood that the scope of the appended claims is not limited to the specific features or methods described herein. Rather, the specific features and methods are disclosed as examples of this disclosure.

100:Computing system 102: Labeled input image 110:Memory 112: Processor 114:Interface 116: Segmented network 118:Scoring module 120:Quality inspection module 200:Method 201:Block 202:Block 203: Block/Step 204:Step 205:Block 206:Block 300:Deep Neural Network

Embodiments are provided with reference to the accompanying drawings, in which:

[Fig. 1] illustrates a system environment for identifying mislabeled images from a labeled image set for a deep neural network according to an example implementation of the inventive subject matter;

[Fig. 2] illustrates a flowchart of a method for identifying mislabeled images from a labeled image set for a deep neural network according to an example implementation of the inventive subject matter;

[Fig. 3a] and [Fig. 3b] illustrate graphical representations of scoring functions for each image in a two-dimensional plane according to example implementations of the inventive subject matter.

100:Computing system

102: Labeled input image

110:Memory

112: Processor

114:Interface

116: Segmented network

118:Scoring module

120:Quality inspection module

300:Deep Neural Network

Claims (10)

A computing system (100) comprising: a memory (110); and A processor (112) coupled to the memory (110) configured to provide a set of tagged images (102) to a segmented network (116), wherein the segmented network Path (116) configured to generate predictions for each image from the set of labeled images (102); Its characteristics are: A scoring module (118) is configured to use the predictions generated by the segmentation network (116) to calculate two or more images for each of the plurality of labeled images (102). two scoring functions; and A quality inspection module (120) is configured to generate data from the plurality of labeled images (102) by visualizing image patches from the set of labeled images (102) obtained from a multi-dimensional graphical representation. Identifying mislabeled images in the set of 102), wherein the multidimensional graphical representation is obtained from different values of the two or more scoring functions for each image from the plurality of labeled images (102) . The computing system (100) of claim 1, wherein the quality inspection module (120) is configured to generate a two-dimensional graphical representation using the two or more calculated scoring functions to allow from the The incorrectly labeled image is selected more quickly from the set of labeled images (102). The computing system (100) of claim 1, wherein the segmented network (116) is communicatively coupled to a deep neural network (300). The computing system (100) of claim 3, wherein the segmentation network (116) is configured to generate segmentation predictions based on a plurality of classifiers and a plurality of training data from the deep neural network (300) . The computing system (100) of claim 1, wherein the segmented network (116) is a trained network on the plurality of labeled images. Such as the computing system (100) of claim 1, wherein the two or more scoring functions include functions of performance measures (IoU), probability scores, uncertainty scores and/or combinations thereof. A computer-implemented method (200) for identifying mislabeled images from a set of labeled images for a deep neural network, the computer-implemented method (200) including steps for: receiving (201) a set of a plurality of labeled images (102) via a segmented network (116) and generating a prediction for each image from the set of the plurality of labeled images (102); Computing (202) two or more scoring functions by a scoring module (118) using the predictions generated for each of the plurality of labeled images (102) ; By using a quality inspection module (120), by visualizing image patches from the set of labeled images (102) obtained from a multi-dimensional graphical representation, from the plurality of labeled images (102) Identifying (203) mislabeled images in the set of 102), wherein the multidimensional graphical representation is derived from the two or more scoring functions for each image from the plurality of labeled images (102) Different values are obtained. The computer-implemented method (200) of claim 7, wherein the method (200) further includes using the calculated two or more scoring functions to generate a multi-dimensional graphical representation for allowing data from the plurality of A step of faster selecting (204) the mislabeled image from the set of labeled images (102). The computer-implemented method (200) of claim 7, wherein the method (200), after the step (203) of identifying the mislabeled image from the set of labeled images (102), further comprises a QC A step in which the administrator directly selects sufficiently error-labeled regions in the multidimensional graphical representation (205). The computer-implemented method (200) of claim 9, wherein after step (205) is performed by the QC administrator, the method (200) further includes a QC staff member receiving the sufficient error flags in the multi-dimensional graphical representation. Regions are iterated across each image in the assigned grid, with annotations to be sent indicating that sufficient mislabeled regions need to be re-labeled for further labeling (206).

Images