CN112419263B - A multi-category non-maximum suppression method and system based on inter-class coverage ratio - Google Patents

Abstract

The invention discloses a multi-class non-maximum inhibition method and a system based on an inter-class coverage ratio, which comprises the steps of setting a class confidence coefficient threshold value, and deleting all prediction frames with class confidence coefficients smaller than the threshold value; screening the prediction frames by calculating the overlapping degree between the prediction frames; judging the inclusion relation between the reference frame and the comparison frame by calculating the proportion of the overlapping area of the reference frame and the comparison frame in the reference frame; if the proportion is smaller than the proportion threshold value, judging that the reference frame does not contain the reference frame, removing the reference frame from the prediction frame set, and adding the reference frame to the empty set; otherwise, judging that the comparison frame comprises a reference frame, selecting a frame with the minimum area in the comparison frame, and screening the reference frame and the minimum frame by utilizing an inter-class preferred selection strategy; if the prediction frame set is not empty, continuously screening out the rest prediction frames; otherwise, an empty set is output. The invention can accurately measure the overlapping degree of prediction frames with different areas and is suitable for detecting various defects.

Description

A multi-category non-maximum suppression method and system based on inter-class coverage ratio

technical field

The present invention relates to the technical field of machine vision, in particular to a method and system for non-maximum suppression of multi-classes based on inter-class coverage ratio.

Background technique

Defect detection uses machine vision equipment to acquire images to judge whether there are defects in the collected images, and at the same time realize the output of the location and category of detected defects. Traditional surface defect detection methods based on machine vision often use conventional image processing algorithms or manual design features plus classification In recent years, with the rapid development of deep learning technology, the defect detection algorithm has also shifted from the traditional algorithm based on artificial features to the detection technology based on deep neural network, with Convolutional Neural Networks (CNN) as the The representative deep learning model has also been successfully applied in the defect detection of many products.

The CNN-based defect detection method has many similarities with the current mainstream CNN-based target detection method. Taking the Faster R-CNN network as an example for target detection or defect detection, feature maps are extracted through convolutional networks, and the feature maps are sent to Import the Region Proposal Network (RPN) to generate a large number of prediction frames that may contain targets or defects to be checked. After classification and regression, about 2000 prediction frames are retained, and then redundant prediction frames are removed by some algorithms.

However, CNN-based target detection methods cannot be directly applied to product defect detection, because in target detection, a large-sized target may contain one or several small-sized targets, but in defect detection, the interior of a large-sized defect Defects of other small sizes are not included, that is, the defect categories are incompatible; in addition, when several defects of the same category are gathered together, they should be identified as a whole.

Contents of the invention

The purpose of this section is to outline some aspects of embodiments of the invention and briefly describe some preferred embodiments. Some simplifications or omissions may be made in this section, as well as in the abstract and titles of this application, to avoid obscuring the purpose of this section, the abstract and titles, and such simplifications or omissions should not be used to limit the scope of the invention.

In view of the above existing problems, the present invention is proposed.

Therefore, the present invention provides a multi-category non-maximum suppression method based on the inter-class coverage ratio, which can accurately reflect the degree of overlap between prediction frames with large area differences, and can screen out different types of prediction frames during the detection process. And merging prediction frames of the same category solves the problems of inaccurate screening of prediction frames and limited scope of application in the prior art.

In order to solve the above-mentioned technical problems, the present invention provides the following technical solutions: including, setting a category confidence threshold, and deleting all prediction boxes in the prediction box set whose category confidence is less than the threshold; by calculating the overlap between the prediction boxes degree screening prediction frame; by calculating the ratio of the overlapping area of the reference frame and the comparison frame in the reference frame to determine the containment relationship between the reference frame and the comparison frame; if the ratio is less than the set ratio threshold, then Determining that the reference frame does not contain the reference frame, removing the reference frame from the predicted frame set, adding the reference frame to an empty set; otherwise, determining that the reference frame contains the reference frame, And select the frame with the smallest area in the comparison frame, and use the inter-class selection strategy to screen the reference frame and the smallest frame; judge whether the set of predicted frames is empty, if the set of predicted frames is not empty, continue Filter out the remaining predicted boxes; otherwise, output the empty set.

As a preferred solution of the multi-category non-maximum suppression method based on the inter-class coverage ratio in the present invention, wherein: the calculation of the degree of overlap includes, the predicted boxes in the predicted box set B according to the category confidence Sort in descending order; the degree of overlap is as follows:

Wherein, COP _BoM is the proportion of the overlapping area of the reference frame M and the reference frame Bo in the reference frame, S _MBo is the overlapping area of the reference frame M and the reference frame Bo, S _Bo is the area of the reference frame, and the reference frame M is the prediction frame with the highest confidence of the category, and the control frame Bo is the remaining prediction frames.

As a preferred solution of the multi-category non-maximum suppression method based on the inter-class coverage ratio in the present invention, wherein: the screening prediction frame includes, if the COP _BoM is greater than the overlapping threshold, then the reference frame M and the comparison frame Bo are removed from the prediction frame set, and the reference frame M is added to the empty set D; otherwise, the reference frame M is removed from the prediction frame set B, and the The reference frame M is added to the empty set D.

As a preferred solution of the multi-category non-maximum suppression method based on the inter-class coverage ratio in the present invention, wherein: the screening prediction frame also includes, judging whether the prediction frame set B is empty, if not empty , then continue to screen the prediction boxes; otherwise, output the prediction boxes in the empty set D.

As a preferred solution of the multi-category non-maximum suppression method based on the inter-class coverage ratio in the present invention, wherein: the overlapping threshold includes, and the overlapping threshold is equal to 0.8.

As a preferred solution of the multi-category non-maximum suppression method based on the inter-class coverage ratio in the present invention, wherein: the calculation ratio includes sorting the prediction frames in the prediction frame set B in ascending order according to the area of the prediction frames; Calculate the ratio COP _MBo of the overlapping area of the reference frame and the comparison frame in the reference frame according to the following formula:

Among them, S _M is the area of the reference frame, and the prediction frame with the smallest area is marked as the reference frame M1; the remaining prediction frames are marked as the comparison frame Bo1;

As a preferred solution of the multi-category non-maximum suppression method based on the inter-class coverage ratio in the present invention, wherein: the inter-class selection strategy includes judging whether the categories of the reference frame M1 and the minimum frame N are the same , if the categories are different, keep the prediction box with high category confidence, and remove the prediction box with low category confidence; if the categories are the same, calculate the difference between the category confidence of the two prediction boxes.

As a preferred solution of the multi-category non-maximum suppression method based on the inter-class coverage ratio in the present invention, wherein: the category confidence difference includes, if the difference is greater than the set difference threshold, keep The prediction box with a higher category confidence is removed, and the other prediction box is removed; otherwise, the prediction box with a larger area and the prediction box with a higher confidence value for the category are retained.

As a preferred solution of the multi-category non-maximum suppression system based on the inter-class coverage ratio of the present invention, it includes: an input module, which is used to input a prediction frame set and an empty set to the system; a rough selection module, and the The input module is connected, and it is used to screen out redundant frames with very low class confidence; the Score-NMS module is connected to the rough selection module, and it is used to screen out the prediction frames with a high degree of overlap; the Area-NMS module , connected to the Score-NMS module, which is used to further filter out the prediction frames left by the Score-NMS module.

Beneficial effects of the present invention: the present invention proposes COP parameters, which can accurately measure the degree of overlap between prediction frames of different areas; the accuracy of defect location is improved by using the optimal selection strategy between classes; in addition, the present invention is applicable to various categories The detection of defects is also applicable to the detection of appearance defects of various objects such as semiconductors, high-speed rail line fasteners, transmission tower insulators, textured surfaces, and metal surfaces.

Description of drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the accompanying drawings that need to be used in the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For Those of ordinary skill in the art can also obtain other drawings based on these drawings without any creative effort. in:

Fig. 1 is a schematic flow chart of a multi-category non-maximum suppression method based on the inter-class coverage ratio described in the first embodiment of the present invention;

2 is a schematic diagram of the COP _BoM calculation method of a multi-category non-maximum suppression method based on the inter-class coverage ratio described in the first embodiment of the present invention;

3 is a schematic diagram of the COP _MBo calculation method of a multi-category non-maximum suppression method based on the inter-class coverage ratio described in the first embodiment of the present invention;

Fig. 4 is a schematic diagram of the prediction frame inclusion relationship of a multi-category non-maximum suppression method based on the inter-class coverage ratio described in the first embodiment of the present invention;

Fig. 5 is a schematic flow chart of a method for class selection based on a multi-class non-maximum suppression method based on the inter-class coverage ratio described in the first embodiment of the present invention;

Fig. 6 is a schematic diagram of defect detection before implementation of inter-class preference in a multi-class non-maximum suppression method based on inter-class coverage ratio described in the first embodiment of the present invention;

Fig. 7 is a schematic diagram of defect detection after the first inter-class optimal selection of a multi-class non-maximum suppression method based on the inter-class coverage ratio described in the first embodiment of the present invention;

Fig. 8 is a schematic diagram of defect detection after the second inter-class selection of a multi-class non-maximum suppression method based on the inter-class coverage ratio described in the first embodiment of the present invention;

9 is a schematic diagram of defect detection after the third inter-class selection of a multi-class non-maximum suppression method based on the inter-class coverage ratio described in the first embodiment of the present invention;

Fig. 10 is a schematic diagram of samples containing Foreign_M defects of a multi-category non-maximum suppression method based on the inter-class coverage ratio described in the first embodiment of the present invention;

Fig. 11 is a schematic diagram of samples containing Gold_P defects of a multi-category non-maximum suppression method based on the inter-class coverage ratio described in the first embodiment of the present invention;

Fig. 12 is a schematic diagram of samples containing Raw_M defects of a multi-category non-maximum suppression method based on the inter-class coverage ratio described in the first embodiment of the present invention;

13 is a schematic diagram of the detection results of Foreign_M defect samples by the current multi-class NMS method based on the inter-class coverage ratio multi-class non-maximum suppression method described in the first embodiment of the present invention;

14 is a schematic diagram of the detection results of Gold_P defect samples by the current multi-category NMS method based on the inter-class coverage ratio multi-category non-maximum suppression method described in the first embodiment of the present invention;

15 is a schematic diagram of the detection results of Raw_M defect samples by the current multi-class NMS method based on the inter-class coverage ratio multi-class non-maximum suppression method described in the first embodiment of the present invention;

Fig. 16 is a schematic diagram of the detection results of Foreign_M defect samples by this method of a multi-category non-maximum suppression method based on the inter-class coverage ratio described in the first embodiment of the present invention;

Fig. 17 is a schematic diagram of the detection results of Gold_P defect samples by this method of a multi-category non-maximum suppression method based on the inter-class coverage ratio described in the first embodiment of the present invention;

Fig. 18 is a schematic diagram of the detection results of Raw_M defect samples of a multi-category non-maximum suppression method based on the inter-class coverage ratio described in the first embodiment of the present invention;

19 is a schematic diagram of an IoU calculation method of a multi-category non-maximum suppression method based on the inter-class coverage ratio described in the first embodiment of the present invention;

Fig. 20 is a schematic diagram of the detection result of Gold_P using IoU to measure the overlapping degree of a multi-category non-maximum suppression method based on the inter-class coverage ratio described in the first embodiment of the present invention;

Fig. 21 is a schematic diagram of the detection result of Gold_P measured by COP in a multi-category non-maximum suppression method based on the inter-class coverage ratio described in the first embodiment of the present invention;

Fig. 22 is a schematic diagram of the detection results of Incomplete_B by the current multi-class NMS method of a multi-class non-maximum suppression method based on the inter-class coverage ratio described in the first embodiment of the present invention;

Fig. 23 is a schematic diagram of the detection result of Incomplete_B by this method of a multi-category non-maximum suppression method based on the inter-class coverage ratio described in the first embodiment of the present invention;

Fig. 24 is a schematic flowchart of a method for calculating the degree of overlap COP _BoM according to the second embodiment of the present invention;

Fig. 25 is a schematic diagram of a COP _BoM calculation method of a method for calculating the overlap degree COP _BoM described in the second embodiment of the present invention;

26 is a schematic diagram of the module structure distribution of a multi-class non-maximum suppression system based on the inter-class coverage ratio described in the third embodiment of the present invention;

FIG. 27 is a schematic diagram of the network topology of a multi-category non-maximum suppression method based on the inter-class coverage ratio described in the third embodiment of the present invention;

FIG. 28 is a schematic flowchart of the Score-NMS module 300 algorithm of a multi-category non-maximum suppression method based on the inter-class coverage ratio described in the third embodiment of the present invention;

FIG. 29 is a schematic flowchart of the Area-NMS module 400 algorithm of a multi-category non-maximum suppression method based on the inter-class coverage ratio described in the third embodiment of the present invention;

Fig. 30 is a schematic diagram of the input prediction frame of a multi-category non-maximum suppression method based on the inter-class coverage ratio described in the third embodiment of the present invention;

FIG. 31 is a schematic diagram of the result after the rough selection module 200 of a multi-category non-maximum suppression method based on the inter-class coverage ratio described in the third embodiment of the present invention completes the screening of prediction frames;

FIG. 32 is a schematic diagram of the results of the Score-NMS module 300 of a multi-category non-maximum suppression method based on the inter-class coverage ratio described in the third embodiment of the present invention after the prediction box is screened;

FIG. 33 is a schematic diagram of the results of the Area-NMS module 400 of a multi-category non-maximum suppression method based on the inter-class coverage ratio described in the third embodiment of the present invention after the prediction box is screened;

Fig. 34 is a schematic diagram of the module structure distribution of a multi-category non-maximum suppression system described in the fourth embodiment of the present invention;

Fig. 35 is a schematic diagram of the network topology of a multi-category non-maximum suppression system according to the fourth embodiment of the present invention.

Detailed ways

In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation modes of the present invention will be described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Example. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative efforts shall fall within the protection scope of the present invention.

In the following description, a lot of specific details are set forth in order to fully understand the present invention, but the present invention can also be implemented in other ways different from those described here, and those skilled in the art can do it without departing from the meaning of the present invention. By analogy, the present invention is therefore not limited to the specific examples disclosed below.

Second, "one embodiment" or "an embodiment" referred to herein refers to a specific feature, structure or characteristic that may be included in at least one implementation of the present invention. "In one embodiment" appearing in different places in this specification does not all refer to the same embodiment, nor is it a separate or selective embodiment that is mutually exclusive with other embodiments.

The present invention is described in detail in conjunction with schematic diagrams. When describing the embodiments of the present invention in detail, for the convenience of explanation, the cross-sectional view showing the device structure will not be partially enlarged according to the general scale, and the schematic diagram is only an example, which should not limit the present invention. scope of protection. In addition, the three-dimensional space dimensions of length, width and depth should be included in actual production.

At the same time, in the description of the present invention, it should be noted that the orientation or positional relationship indicated by "upper, lower, inner and outer" in the terms is based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing the present invention. The invention and the simplified description do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and thus should not be construed as limiting the present invention. In addition, the terms "first, second or third" are used for descriptive purposes only, and should not be construed as indicating or implying relative importance.

Unless otherwise specified and limited in the present invention, the term "installation, connection, connection" should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection or an integrated connection; it can also be a mechanical connection, an electrical connection or a direct connection. A connection can also be an indirect connection through an intermediary, or it can be an internal communication between two elements. Those of ordinary skill in the art can understand the specific meanings of the above terms in the present invention in specific situations.

Example 1

With reference to Fig. 1～Fig. 23, be the first embodiment of the present invention, this embodiment provides a kind of multi-category non-maximum suppression method based on inter-class coverage ratio, including:

S1: Set the category confidence threshold, and delete all prediction boxes in the prediction box set whose category confidence is less than the threshold.

S2: Filter the prediction boxes by calculating the overlap between the prediction boxes.

Specifically, the steps of screening the prediction box are as follows:

(1) Assume that the input prediction box set is B, and the empty set is D;

(2) Sort the prediction frames in the set B in descending order according to the category confidence;

(3) Mark the prediction frame with the highest category confidence as the reference frame, denoted as M; the remaining prediction frames are marked as the control frame, denoted as Bo; define the proportion of the overlapping area between the reference frame and the reference frame in the reference frame as COP _BoM , that is, degree of overlap, as shown in Figure 2;

Specifically, the overlap degree COP _BoM is calculated as follows:

Among them, S _MBo is the overlapping area of the reference frame M and the control frame Bo, S _Bo is the area of the control frame, and COP (COverPercent) is the proportion of the overlapping area of the two prediction frames in the specified prediction frame;

(4) Set the overlapping threshold to 0.8. According to the relationship between the COP _BoM and the set threshold, judge whether there is an overlapping relationship between the reference frame M and the comparison frame Bo; if the COP _BoM is greater than the threshold, it indicates that the reference frame M and the comparison frame Bo overlap each other , remove both the reference frame M and the control frame Bo from the set B and add M to the set D; if the COP _BoM is less than the threshold, it indicates that there is no overlapping relationship, remove the reference frame M from the set B and add M to the set D;

(5) Determine whether the set B is empty; if it is not empty, the prediction box in the set B will be cyclically screened according to steps (2) to (4), until the set B is empty and stop the loop; if it is empty, then output Predicted boxes in set D.

S3: Determine the containment relationship between the reference frame and the reference frame by calculating the proportion of the overlapping area of the reference frame and the comparison frame in the reference frame.

Specifically, the steps for judging the containment relationship between the reference frame and the comparison frame are as follows:

(1) Sort the prediction frames in the set B in ascending order according to the area of the prediction frames;

(2) the prediction frame with the minimum area is marked as reference frame M1; All the other prediction frames are marked as reference frame Bo1; the ratio of the overlapping area defining reference frame and reference frame in reference frame is COP _MBo ;

Specifically, the COP _MBo is calculated according to the following formula:

Among them, S _M is the area of the reference frame, and S _MBo is the overlapping area of the reference frame M1 and the control frame Bo1;

(3) Set the ratio threshold, and judge whether there is an inclusion relationship between the reference frame M1 and the comparison frame Bo1 according to the relationship between the COP _MBo and the threshold; if the COP _MBo of the reference frame and the comparison frame is greater than the set threshold, it means that the overlapping area is within If the proportion of the frame is larger, it is determined that the reference frame contains the reference frame, as shown in Figure 4; if the COP _MBo of the reference frame and the reference frame is less than the set threshold, it is determined that the reference frame does not contain the reference frame.

S4: If the reference frame does not contain the reference frame, remove the reference frame M1 from the prediction frame set B, and add the reference frame M1 to the empty set D.

S5: If the comparison frame Bo1 contains the reference frame M1, select the smallest area frame N in the comparison frame Bo1, and use the inter-class selection strategy to screen the reference frame M1 and the smallest frame N.

It should be noted that the optimal selection strategy between classes is divided into the optimal selection between prediction boxes of different categories and the optimal selection between prediction boxes of the same category.

Specifically, determine whether the categories of the reference frame M1 and the smallest frame N are the same, refer to FIG. 5 ;

(1) If the categories are different, keep the prediction frame with high confidence in the category and remove the prediction frame with low confidence in the category;

(2) If the categories are the same, calculate the category confidence difference between the two prediction boxes.

① If the difference is greater than the set difference threshold, keep the prediction frame with a larger category confidence and remove the other prediction frame;

② If the difference is less than the set threshold, the two prediction frames are merged, that is, the prediction frame with a larger area is retained while a larger category confidence value is retained.

It should be noted that, in this embodiment, a total of three inter-category selections have been completed. Figure 6 is a defect detection diagram before the implementation of inter-category selection. In the figure, there are four prediction boxes (No. is 1-4), the prediction frame and its category confidence are as follows: 1:0.94, 2:0.60, 3:0.58, 4:0.44.

In the first inter-class selection, the predicted frame 1 with the smallest marked area is the reference frame, and the marked predicted frame 2 is the reference frame. Through calculation, it can be determined that the area of the reference frame is larger than that of the reference frame, and the confidence of the reference frame is greater than that of the reference frame. At the same time The category confidence difference between the reference frame and the control frame is less than the set threshold; therefore, the two prediction frames are merged, that is, the prediction frame 2 is retained, and the category confidence value of prediction frame 1 is assigned to prediction frame 2, and the prediction frame is removed. 1.

As shown in Figure 7, after the first inter-class selection is completed, the reserved prediction frame and its category confidence are as follows: 2:0.94, 3:0.58, 4:0.44;

In the second inter-class selection, the category confidence difference between prediction boxes 2 and 3 is also smaller than the set threshold, so the prediction box 3 is retained while the category confidence value is 0.94, and the prediction box 2 is removed, as shown in Figure 8;

In the third inter-class selection, the category confidence difference between the prediction boxes 3 and 4 is greater than the set threshold, so the prediction box 3 is directly retained, and the prediction box 2 is removed, as shown in Figure 9;

Preferably, the accuracy of defect location is improved through class selection.

S6: Determine whether the prediction frame set B is empty, if the prediction frame set is not empty, continue to screen out the remaining prediction frames; otherwise, output an empty set.

If the prediction frame set B is not empty, then the prediction frames in the set B are cycled according to steps S3-S5, and the cycle is stopped when the prediction frame set B is empty; if it is empty, the prediction frames in the set D are output.

In order to verify and explain the technical effect adopted in this method, this embodiment chooses the current multi-category NMS method and adopts this method to conduct a comparative test, and compares the test results by means of scientific demonstration to verify the real effect of this method.

The current multi-category NMS method is not sufficient for redundancy removal, and the defect location is not accurate enough;

In order to verify that this method has a higher degree of overlap measurement accuracy and defect location accuracy than the current multi-category NMS (Non-Maximum Suppression, non-maximum suppression) method, the present embodiment adopts the current multi-category NMS method and This method performs training and testing comparisons in actual defect samples.

The data set contains a total of 2200 defect samples and 8 categories of defects, of which the training set contains 2000 samples, and the test set contains 200 samples. Three of the defect samples are shown in Figure 10, Figure 11, and Figure 12, respectively. It is sample 1, which contains Foreign_M defects; Figure 11 is sample 2, which contains Gold_P defects; Figure 12 shows sample 3, which contains Raw_M defects; the detection results of the current multi-category NMS method in the three defect samples are shown in Figure 13 and Figure 14 respectively , as shown in FIG. 15 , and the detection results of this method in three defect samples are shown in FIG. 16 , FIG. 17 , and FIG. 18 .

(1) Referring to Figure 13, it can be seen that the current multi-category NMS algorithm retains 4 prediction frames after defect detection, and the prediction frames overlap each other, the redundancy removal is not sufficient, and the defect location is not accurate enough; after the detection of this method, as shown in Fig. As shown in Figure 16, a prediction box is retained, indicating that the confidence of the defect category is high, and the defect location is more accurate, and other samples have this effect.

(2) In order to verify that this method can accurately measure the degree of overlap between prediction frames of different areas, taking the current multi-category NMS method and the detection results of this method in the "Gold_P" defect sample as examples, the previous multi-category NMS The method's IoU (Intersection over Union, the ratio of the intersection area of two prediction frames to the union area) parameter calculation method and the COP parameter calculation method of this method are used to measure the degree of overlap of the prediction frames; Figure 20 shows the current multi-category NMS method The detection results of , where the coordinates of predicted frames 1 and 2 are:

After calculation, the respective areas and intersection areas of prediction frames 1 and 2 can be obtained:

① The IoU of prediction frames 1 and 2 can be obtained through the calculation method of IoU parameters in the current multi-category NMS method (as shown in Figure 19):

It can be found that the IoU of the current prediction frames 1 and 2 is less than the set threshold (usually around 0.7), and the prediction frames 1 and 2 actually overlap each other at this time, but the IoU cannot accurately measure the degree of overlap, resulting in incomplete removal of redundant frames .

② The COP of prediction frames 1 and 2 can be obtained by calculating the COP parameter in this method:

It can be found that the COPs of prediction frames 1 and 2 obtained by the COP parameter calculation method in this method can accurately measure the degree of overlap, and the result after processing by this method is shown in Figure 21.

(3) Take the detection results of the current multi-category NMS algorithm and the technical solution of the present invention in the "Incomplete_B" defect sample as an example; as shown in Figure 22, after using the current multi-category NMS method to detect defect samples, two prediction boxes of different categories, that is, Incomplete_B: 0.96 and Gold_P: 0.11, however, this does not meet the need for defect category incompatibility in defect detection.

However, the class-incompatible non-maximum suppression scheme constructed by the present invention can keep only one class prediction frame after detection, that is, Incomplete_B: 0.96, as shown in FIG. 23 .

Example 2

Referring to Fig. 24 to Fig. 25, it is the second embodiment of the present invention. The difference from embodiment 1 is that this embodiment proposes a method for calculating the degree of overlap COP _BoM , including:

S1: The predicted frame with the highest category confidence is used as the reference frame, and the other predicted frames except the predicted frame with the highest category confidence are used as control frames.

S2: Calculate the proportion of the overlapping area between the reference frame and the reference frame in the reference frame, which is the COP _BoM .

Calculated as follows:

Example 3

Referring to Fig. 26 to Fig. 33, it is the third embodiment of the present invention. Different from Embodiments 1 and 2, this embodiment proposes a multi-category non-maximum suppression system based on the inter-class coverage ratio, including:

The input module 100 is used to input the prediction box set and the empty set to the system.

The rough selection module 200, connected with the input module 100, is used to filter out redundant boxes with very low category confidence.

The Score-NMS module 300 is connected to the rough selection module 200, which is used to receive the prediction frame set and the empty set from the rough selection module 200 and to filter out the prediction frames with a high degree of overlap; specifically, refer to 28, Score -The NMS module 300 is based on the non-maximum suppression of the category confidence (Score), using the category confidence as a standard, by calculating the COP to measure the degree of overlap between the prediction frames, and performing the prediction on the prediction frame with a high degree of overlap (not less than 0.8) Screen out.

The Area-NMS module 400 is connected to the Score-NMS module 300, which is used to further screen out the prediction frames left by the Score-NMS module 300 screening; specifically, referring to FIG. 29, the Area-NMS module 400 is based on the area ( area) non-maximum suppression, with the area of the prediction frame as the standard, by calculating the COP to determine whether there is an inclusion relationship between the prediction frames, preferably, the mutually included prediction frames are selected by the inter-class selection method, so that The preserved prediction boxes are better localized and have higher class confidence.

In order to verify and explain the technical effects adopted in this system, this embodiment tests the rough selection module 200, the Score-NMS module 300 and the Area-NMS module 400 respectively, and compares the test results by means of scientific demonstration to verify the system. have real effects.

As shown in Figure 30, a total of 1000 prediction boxes are input, and their position coordinates and category confidence are as follows.

①The position coordinates of 1000 prediction frames are:

②The category confidence of 1000 prediction boxes is:

(1) The rough selection module 200 sets the category confidence threshold, such as being set to 0.05 in this embodiment, to screen out prediction frames whose category confidence is lower than 0.05, and retain 87 prediction boxes after screening, as shown in Figures 31 and 87 The position coordinates and category confidence of each prediction box are as follows.

① The position coordinates of the 87 prediction frames are:

②The category confidence of the 87 prediction boxes is:

(2) Use the Score-NMS module 300 to screen out prediction frames with a large degree of overlap, and set the COP _BoM threshold, such as 0.8 in this embodiment, to iteratively calculate the COP of the prediction frame with the largest category confidence and the remaining prediction frames _BoM , remove the corresponding prediction boxes whose COP _BoM value is greater than 0.8, and finally keep 4 prediction boxes, as shown in Figure 32.

(3) Merge the predicted frames with containment relationship through the inter-class selection method in the Area-NMS module, as shown in Figure 33, the last predicted frame has a higher category confidence (0.94) and better defect location.

Example 4

Referring to Figures 34 to 35, it is the fourth embodiment of the present invention, which is different from Embodiment 3. This embodiment proposes a multi-category non-maximum suppression system, including:

The input module 100 is used to input the prediction box set and the empty set to the system.

The rough selection module 200, connected with the input module 100, is used to filter out redundant boxes with very low category confidence.

The Area-NMS module 300 is connected with the rough selection module 200, and it is used to receive the prediction frame set and the empty set from the rough selection module 200 and is used to screen out the prediction frames that contain a relationship; specifically, the Area-NMS module 400 Based on area (area) non-maximum suppression, using the area of the prediction frame as the standard, by calculating the COP to judge whether there is an inclusion relationship between the prediction frames, it is better to select the best among the prediction frames that contain each other through the inter-class selection method Select, so that the reserved prediction box is better positioned and the category confidence is higher.

The Score-NMS module 400 is connected to the Area-NMS module 300, which is used to receive the prediction frame set and the empty set from the Area-NMS module 300 and to screen out the prediction frames with a high degree of overlap; specifically, the Score-NMS module 300 The NMS module 400 is based on the non-maximum suppression of the category confidence (Score), takes the category confidence as the standard, measures the degree of overlap between the prediction frames by calculating the COP, and screens the prediction frames with a high degree of overlap (not less than 0.8) remove.

It should be appreciated that embodiments of the invention may be realized or implemented by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer readable memory. The methods can be implemented in a computer program using standard programming techniques - including a non-transitory computer-readable storage medium configured with a computer program, where the storage medium so configured causes the computer to operate in a specific and predefined manner - according to the specific Methods and Figures described in the Examples. Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with the computer system. However, the programs can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language. Furthermore, the program can be run on an application specific integrated circuit programmed for this purpose.

In addition, operations of processes described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes described herein (or variations and/or combinations thereof) can be performed under the control of one or more computer systems configured with executable instructions, and as code that collectively executes on one or more processors (e.g. , executable instructions, one or more computer programs or one or more applications), hardware or a combination thereof. The computer program comprises a plurality of instructions executable by one or more processors.

Further, the method can be implemented in any type of computing platform operably connected to a suitable one, including but not limited to personal computer, minicomputer, main frame, workstation, network or distributed computing environment, stand-alone or integrated computer platform, or communicate with charged particle tools or other imaging devices, etc. Aspects of the invention can be implemented as machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optically read and/or written storage medium, RAM, ROM, etc., such that they are readable by a programmable computer, when the storage medium or device is read by the computer, can be used to configure and operate the computer to perform the processes described herein. Additionally, the machine-readable code, or portions thereof, may be transmitted over a wired or wireless network. The invention described herein includes these and other various types of non-transitory computer-readable storage media when such media include instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor. The invention also includes the computer itself when programmed according to the methods and techniques described herein. Computer programs can be applied to input data to perform the functions described herein, thereby transforming the input data to generate output data stored to non-volatile memory. Output information may also be applied to one or more output devices such as a display. In a preferred embodiment of the invention, the transformed data represents physical and tangible objects, including specific visual depictions of physical and tangible objects produced on a display.

As used in this application, the terms "component," "module," "system" and the like are intended to refer to a computer-related entity, which may be hardware, firmware, a combination of hardware and software, software, or an operating system. software. For example, a component may be, but is not limited to being, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and/or a computer. As an example, both an application running on a computing device and the computing device can be components. One or more components can reside within a process and/or thread of execution and a component can be localized on one computer and/or distributed between two or more computers. In addition, these components can execute from various computer readable media having various data structures thereon. These components can be communicated through, for example, according to having one or more packets of data (e.g., data from a component that interacts with another component in a local system, a distributed system, and/or in the form of network to interact with other systems) to communicate with local and/or remote processes.

It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention without limitation, although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be carried out Modifications or equivalent replacements without departing from the spirit and scope of the technical solution of the present invention shall be covered by the claims of the present invention.

Claims (7)

1. A multi-category non-maximum suppression method based on inter-class coverage ratio, characterized in that: comprising, Set a category confidence threshold, and delete all prediction boxes in the prediction box set whose category confidence is less than the threshold; Filtering prediction frames by calculating the degree of overlap between the prediction frames; Determine the containment relationship between the reference frame and the reference frame by calculating the ratio of the overlapping area of the reference frame and the comparison frame in the reference frame; if the ratio is less than the set ratio threshold, then determine the comparison frame The reference frame is not included, and the reference frame is removed from the prediction frame set, and the reference frame is added to an empty set; Otherwise, it is determined that the comparison frame contains the reference frame, and a frame with the smallest area is selected in the comparison frame, and the reference frame and the smallest frame are screened using the optimal selection strategy between classes. The optimal selection strategy between classes is: Judging whether the categories of the reference frame M1 and the minimum frame N are the same, if the categories are different, then retain the prediction frame with high confidence in the category, and remove the prediction box with low confidence in the category; if the categories are the same, calculate two The category confidence difference of the prediction frame; if the difference is greater than the set difference threshold, the prediction frame with a higher category confidence is retained and the other prediction frame is removed; otherwise, the prediction frame with a larger area and the other prediction frame are retained. The prediction frame with higher confidence value of the above category; Judging whether the prediction frame set is empty, if the prediction frame set is not empty, continue to filter out the remaining prediction frames; otherwise, output the empty set. 2. the multi-category non-maximum suppression method based on inter-class coverage ratio as claimed in claim 1, is characterized in that: said calculation overlap comprises, Sort the prediction frames in the prediction frame set B in descending order according to the category confidence; The degree of overlap is as follows:

Wherein, COP _BoM is the proportion of the overlapping area of the reference frame M and the reference frame Bo in the reference frame, S _MBo is the overlapping area of the reference frame M and the reference frame Bo, S _Bo is the area of the reference frame, and the reference frame M is the prediction frame with the highest confidence of the category, and the control frame Bo is the remaining prediction frames. 3. The multi-category non-maximum suppression method based on inter-class coverage ratio as claimed in claim 2, wherein: the screening prediction frame comprises, If the COP _BoM is greater than the overlapping threshold, the reference frame M and the reference frame Bo are removed from the prediction frame set, and the reference frame M is added to the empty set D; Otherwise, the reference frame M is removed from the prediction frame set B, and the reference frame M is added to the empty set D at the same time. 4. the multi-category non-maximum suppression method based on inter-class coverage ratio as claimed in claim 3, is characterized in that: the screening prediction frame also includes, Judging whether the prediction frame set B is empty, if not, continue to screen the prediction frames; otherwise, output the prediction frames in the empty set D. 5. The multi-category non-maximum suppression method based on inter-class coverage ratio as claimed in claim 3 or 4, wherein: the overlapping threshold comprises, The overlap threshold is equal to 0.8. 6. as any one of claim 1, 2, 3 multi-category non-maximum suppression method based on inter-class coverage ratio, it is characterized in that: described calculation ratio comprises, Sort the prediction frames in the prediction frame set B in ascending order according to the area of the prediction frames; Calculate the ratio COP _MBo of the overlapping area of the reference frame and the comparison frame in the reference frame according to the following formula:

Among them, S _M is the area of the reference frame, and the predicted frame with the smallest area is marked as the reference frame M1; the remaining predicted frames are marked as the control frame Bo1. 7. A system applied to the multi-category non-maximum suppression method based on the inter-class coverage ratio of claim 1, characterized in that: comprising, Input module (100), is used for inputting prediction frame set and empty set to system; A rough selection module (200), connected to the input module (100), which is used to filter out redundant frames with very low category confidence; Score-NMS module (300), is connected with described rough selection module (200), and it is used for screening out the prediction frame with higher degree of overlap; The Area-NMS module (400), connected to the Score-NMS module (300), is used to further screen out the prediction frames left by the Score-NMS module (300).

Images