CN112419263A - Multi-class non-maximum inhibition method and system based on inter-class coverage ratio - Google Patents

Multi-class non-maximum inhibition method and system based on inter-class coverage ratio Download PDF

Info

Publication number
CN112419263A
CN112419263A CN202011312543.9A CN202011312543A CN112419263A CN 112419263 A CN112419263 A CN 112419263A CN 202011312543 A CN202011312543 A CN 202011312543A CN 112419263 A CN112419263 A CN 112419263A
Authority
CN
China
Prior art keywords
frame
prediction
class
reference frame
inter
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202011312543.9A
Other languages
Chinese (zh)
Other versions
CN112419263B (en
Inventor
蒋三新
王新宇
腾繁
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Electric Power University
Original Assignee
Shanghai Electric Power University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Electric Power University filed Critical Shanghai Electric Power University
Priority to CN202011312543.9A priority Critical patent/CN112419263B/en
Publication of CN112419263A publication Critical patent/CN112419263A/en
Application granted granted Critical
Publication of CN112419263B publication Critical patent/CN112419263B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0004—Industrial image inspection
    • G06T7/0006—Industrial image inspection using a design-rule based approach
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/60—Analysis of geometric attributes
    • G06T7/62—Analysis of geometric attributes of area, perimeter, diameter or volume
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image

Abstract

The invention discloses a multi-class non-maximum inhibition method and a system based on an inter-class coverage ratio, which comprises the steps of setting a class confidence coefficient threshold value, and deleting all prediction frames with class confidence coefficients smaller than the threshold value; screening the prediction frames by calculating the overlapping degree between the prediction frames; judging the inclusion relation between the reference frame and the comparison frame by calculating the proportion of the overlapping area of the reference frame and the comparison frame in the reference frame; if the proportion is smaller than the proportion threshold value, judging that the reference frame does not contain the reference frame, removing the reference frame from the prediction frame set, and adding the reference frame to the empty set; otherwise, judging that the comparison frame comprises a reference frame, selecting a frame with the smallest area in the comparison frame, and screening the reference frame and the smallest frame by utilizing an inter-class preferred selection strategy; if the prediction frame set is not empty, continuously screening out the rest prediction frames; otherwise, an empty set is output. The invention can accurately measure the overlapping degree of prediction frames with different areas and is suitable for detecting various defects.

Description

Multi-class non-maximum inhibition method and system based on inter-class coverage ratio
Technical Field
The invention relates to the technical field of machine vision, in particular to a multi-class non-maximum inhibition method and system based on an inter-class coverage ratio.
Background
The defect detection utilizes machine vision equipment to acquire images to judge whether defects exist in the acquired images and simultaneously realizes the output of the positions and the types of the detected defects; in recent years, with the rapid development of deep learning technology, a defect detection algorithm is also shifted from a traditional algorithm based on artificial features to a detection technology based on a deep Neural network, and a deep learning model represented by a Convolutional Neural Network (CNN) is also successfully applied to defect detection of various products.
The CNN-based defect detection method has many similarities with the current mainstream CNN-based target detection method, taking the fast R-CNN Network as target detection or defect detection as an example, extracting a feature map through a convolution Network, sending the feature map into an area extraction Network (RPN) to generate a large number of prediction boxes possibly containing targets to be detected or defects, classifying and regressing to reserve about 2000 prediction boxes, and removing redundant prediction boxes through some algorithms.
However, the CNN-based target detection method cannot be directly applied to defect detection of products, because in the target detection, a large-sized target may include one or several small-sized targets, but in the defect detection, the inside of a large-sized defect does not include other small-sized defects, i.e., the defect types are incompatible; in addition, several homogeneous defects should be identified as a whole when they are grouped together.
Disclosure of Invention
This section is for the purpose of summarizing some aspects of embodiments of the invention and to briefly introduce some preferred embodiments. In this section, as well as in the abstract and the title of the invention of this application, simplifications or omissions may be made to avoid obscuring the purpose of the section, the abstract and the title, and such simplifications or omissions are not intended to limit the scope of the invention.
The present invention has been made in view of the above-mentioned conventional problems.
Therefore, the invention provides a multi-class non-maximum inhibition method based on inter-class coverage ratio, which can accurately reflect the overlapping degree between prediction frames with large area difference, can screen out different types of prediction frames and combine the same type of prediction frames in the detection process, and solves the problems of inaccurate screening of the prediction frames and limited application range in the prior art.
In order to solve the technical problems, the invention provides the following technical scheme: setting a category confidence threshold value, and deleting all the prediction frames with the category confidence values smaller than the threshold value in the prediction frame set; screening the prediction frames by calculating the overlapping degree between the prediction frames; judging the inclusion relation between the reference frame and the comparison frame by calculating the proportion of the overlapping area of the reference frame and the comparison frame in the reference frame; if the proportion is smaller than a set proportion threshold value, judging that the reference frame is not contained in the comparison frame, removing the reference frame from the prediction frame set, and adding the reference frame to an empty set; otherwise, judging that the comparison frame comprises the reference frame, selecting a frame with the smallest area in the comparison frame, and screening the reference frame and the smallest frame by utilizing an inter-class preferred selection strategy; judging whether the prediction frame set is empty or not, and if not, continuously screening out the rest prediction frames; otherwise, outputting the empty set.
As a preferred embodiment of the multi-class non-maximum suppression method based on the inter-class coverage ratio, in the present invention: the calculating the overlapping degree comprises sorting the prediction frames in the prediction frame set B in a descending order according to the category confidence degrees; the degree of overlap is as follows:
Figure BDA0002790257540000021
wherein the COPBoMThe ratio S of the overlapping area of the reference frame M and the comparison frame Bo in the comparison frameMBoIs the area of the overlapping region of the reference frame M and the reference frame Bo, SBoAs a control frame area, the baseThe quasi-frame M is the prediction frame with the maximum class confidence, and the comparison frame Bo is the rest prediction frames.
As a preferred embodiment of the multi-class non-maximum suppression method based on the inter-class coverage ratio, in the present invention: the screening prediction box includes, if the COPBoMIf the reference frame M and the comparison frame Bo are larger than the overlap threshold value, removing the reference frame M and the comparison frame Bo from the prediction frame set, and adding the reference frame M to the empty set D; otherwise, the reference frame M is removed from the prediction frame set B, while the reference frame M is added to the empty set D.
As a preferred embodiment of the multi-class non-maximum suppression method based on the inter-class coverage ratio, in the present invention: the screening of the prediction box further comprises the steps of judging whether the prediction box set B is empty, and if not, continuing to screen the prediction box; otherwise, outputting the prediction frame in the empty set D.
As a preferred embodiment of the multi-class non-maximum suppression method based on the inter-class coverage ratio, in the present invention: the overlap threshold comprises the overlap threshold being equal to 0.8.
As a preferred embodiment of the multi-class non-maximum suppression method based on the inter-class coverage ratio, in the present invention: the calculation of the proportion comprises the steps of sorting the prediction frames in the prediction frame set B in an ascending order according to the areas of the prediction frames; calculating the proportion COP of the overlapped area of the reference frame and the comparison frame in the reference frame according to the following formulaMBo：
Figure BDA0002790257540000031
Wherein S isMThe area of the reference frame is marked as a reference frame M1; the remaining prediction boxes are labeled as control boxes Bo 1;
as a preferred embodiment of the multi-class non-maximum suppression method based on the inter-class coverage ratio, in the present invention: the inter-class preferred selection strategy comprises the steps of judging whether the classes of the reference frame M1 and the minimum frame N are the same or not, if the classes are different, reserving a prediction frame with high class confidence coefficient, and removing the prediction frame with low class confidence coefficient; and if the categories are the same, calculating a category confidence difference value of the two prediction frames.
As a preferred embodiment of the multi-class non-maximum suppression method based on the inter-class coverage ratio, in the present invention: the category confidence difference comprises that if the difference is larger than a set difference threshold, a prediction frame with higher category confidence is reserved, and another prediction frame is removed; otherwise, reserving the prediction box with larger area and the prediction box with higher class confidence value.
As a preferable solution of the multi-class non-maximum suppression system based on the inter-class coverage ratio according to the present invention, wherein: the system comprises an input module, a prediction module and a prediction module, wherein the input module is used for inputting a prediction box set and an empty set into the system; the rough selection module is connected with the input module and is used for screening out redundant boxes with very low category confidence; the Score-NMS module is connected with the rough selection module and is used for screening out the prediction frames with higher overlapping degree; and the Area-NMS module is connected with the Score-NMS module and is used for further screening the prediction frames screened by the Score-NMS module.
The invention has the beneficial effects that: the COP parameter is provided, so that the overlapping degree of prediction frames with different areas can be accurately measured; the defect positioning accuracy is improved by applying an inter-class preferred selection strategy; in addition, the invention is suitable for detecting various defects, and is also suitable for detecting appearance defects of various objects such as semiconductors, high-speed rail line fasteners, power transmission tower insulators, texture surfaces, metal surfaces and the like.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise. Wherein:
fig. 1 is a schematic flowchart of a multi-class non-maximum suppression method based on inter-class coverage ratio according to a first embodiment of the present invention;
FIG. 2 is a diagram illustrating COP of a multi-class non-maximum suppression method based on inter-class coverage ratio according to a first embodiment of the present inventionBoMA schematic diagram of a calculation method;
FIG. 3 is a diagram illustrating COP of a multi-class non-maximum suppression method based on inter-class coverage ratio according to a first embodiment of the present inventionMBoA schematic diagram of a calculation method;
FIG. 4 is a diagram illustrating the predicted box containment relationship of a multi-class non-maximum suppression method based on inter-class coverage ratio according to a first embodiment of the present invention;
fig. 5 is a schematic flowchart of an inter-class preferred selection method of a multi-class non-maximum suppression method based on an inter-class coverage ratio according to a first embodiment of the present invention;
FIG. 6 is a schematic diagram illustrating defect detection before inter-class selection is performed by a multi-class non-maximum suppression method based on inter-class coverage ratio according to a first embodiment of the present invention;
fig. 7 is a schematic diagram illustrating defect detection after completion of first inter-class selection of a multi-class non-maximum suppression method based on inter-class coverage ratio according to a first embodiment of the present invention;
FIG. 8 is a schematic diagram illustrating defect detection after completion of second inter-class preferential selection of a multi-class non-maximum suppression method based on inter-class coverage ratio according to a first embodiment of the present invention;
fig. 9 is a schematic diagram illustrating defect detection after completion of third inter-class preferential selection of a multi-class non-maximum suppression method based on inter-class coverage ratio according to a first embodiment of the present invention;
FIG. 10 is a sample diagram of a Foreign _ M-containing defect based on inter-class coverage ratio multi-class non-maximum suppression method according to a first embodiment of the present invention;
FIG. 11 is a diagram illustrating a sample containing Gold _ P defects of a multi-class non-maximum suppression method based on an inter-class coverage ratio according to a first embodiment of the present invention;
FIG. 12 is a sample diagram of a multi-class non-maximum suppression method based on inter-class coverage ratio according to a first embodiment of the present invention, including a Raw _ M defect;
FIG. 13 is a diagram illustrating the detection result of the Foreign _ M defect sample by the current multi-class NMS method based on the inter-class coverage ratio multi-class non-maximum suppression method according to the first embodiment of the present invention;
fig. 14 is a schematic diagram illustrating a detection result of a Gold _ P defect sample by a current multi-class NMS method of a multi-class non-maximum suppression method based on an inter-class coverage ratio according to a first embodiment of the present invention;
fig. 15 is a schematic diagram illustrating a detection result of a Raw _ M defect sample by a current multi-class NMS method of a multi-class non-maximum suppression method based on an inter-class coverage ratio according to a first embodiment of the present invention;
FIG. 16 is a schematic diagram illustrating the detection result of the method for multi-class non-maximum rejection based on the inter-class coverage ratio for the Foreign _ M defect samples according to the first embodiment of the present invention;
FIG. 17 is a diagram illustrating the detection result of a Gold _ P defect sample by the method of the present invention based on the inter-class coverage ratio multi-class non-maximum suppression method according to the first embodiment of the present invention;
fig. 18 is a schematic diagram illustrating a detection result of a Raw _ M defect sample according to a multi-class non-maximum suppression method based on an inter-class coverage ratio according to a first embodiment of the present invention;
FIG. 19 is a diagram illustrating the calculation method IoU of a multi-class non-maximum suppression method based on the coverage ratio between classes according to the first embodiment of the present invention;
fig. 20 is a schematic diagram illustrating a detection result of a multi-class non-maximum suppression method using IoU metric overlap detection Gold _ P according to a first embodiment of the present invention;
fig. 21 is a schematic diagram illustrating a detection result of a multi-class non-maximum suppression method using COP measurement overlap detection Gold _ P according to a first embodiment of the present invention;
fig. 22 is a schematic diagram illustrating a detection result of the current multi-class NMS method for Incomplete _ B based on a multi-class non-maximum suppression method for inter-class coverage ratio according to the first embodiment of the present invention;
fig. 23 is a schematic diagram illustrating a detection result of the method for multi-class non-maximum suppression based on inter-class coverage ratio to Incomplete _ B according to the first embodiment of the present invention;
FIG. 24 is a COP for calculating the degree of overlap according to a second embodiment of the present inventionBoMA schematic flow diagram of the method of (1);
FIG. 25 is a COP for calculating the degree of overlap according to a second embodiment of the present inventionBoMMethod of (1) COPBoMA schematic diagram of a calculation method;
FIG. 26 is a block diagram illustrating a multi-class non-maximum rejection system based on inter-class coverage ratio according to a third embodiment of the present invention;
fig. 27 is a schematic diagram of a network topology of a multi-class non-maximum suppression method based on inter-class coverage ratio according to a third embodiment of the present invention;
FIG. 28 is a schematic diagram of the algorithm flow of the Score-NMS module 300 according to a third embodiment of the present invention;
fig. 29 is a schematic flowchart of an Area-NMS module 400 algorithm of a multi-class non-maximum suppression method based on an inter-class coverage ratio according to a third embodiment of the present invention;
FIG. 30 is a diagram illustrating a prediction box of the input of a multi-class non-maximum suppression method based on an inter-class coverage ratio according to a third embodiment of the present invention;
fig. 31 is a schematic diagram illustrating the result of the rough selection module 200 performing the prediction box screening according to the multi-class non-maximum suppression method based on the inter-class coverage ratio according to the third embodiment of the present invention;
FIG. 32 is a diagram illustrating the results of the Score-NMS module 300 performing the prediction box screening according to the third embodiment of the present invention;
fig. 33 is a diagram illustrating the result of the Area-NMS module 400 performing the prediction box screening according to the third embodiment of the present invention, in a multi-class non-maximum suppression method based on the inter-class coverage ratio;
FIG. 34 is a block diagram illustrating a distribution of multi-class non-maximum suppression systems according to a fourth embodiment of the present invention;
fig. 35 is a schematic diagram of a network topology of a multi-class non-maximum suppression system according to a fourth embodiment of the present invention.
Detailed Description
In order to make the aforementioned objects, features and advantages of the present invention comprehensible, specific embodiments accompanied with figures are described in detail below, and it is apparent that the described embodiments are a part of the embodiments of the present invention, not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without making creative efforts based on the embodiments of the present invention, shall fall within the protection scope of the present invention.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, but the present invention may be practiced in other ways than those specifically described and will be readily apparent to those of ordinary skill in the art without departing from the spirit of the present invention, and therefore the present invention is not limited to the specific embodiments disclosed below.
Furthermore, reference herein to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one implementation of the invention. The appearances of the phrase "in one embodiment" in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments.
The present invention will be described in detail with reference to the drawings, wherein the cross-sectional views illustrating the structure of the device are not enlarged partially in general scale for convenience of illustration, and the drawings are only exemplary and should not be construed as limiting the scope of the present invention. In addition, the three-dimensional dimensions of length, width and depth should be included in the actual fabrication.
Meanwhile, in the description of the present invention, it should be noted that the terms "upper, lower, inner and outer" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings, and are only for convenience of describing the present invention and simplifying the description, but do not indicate or imply that the referred device or element must have a specific orientation, be constructed in a specific orientation and operate, and thus, cannot be construed as limiting the present invention. Furthermore, the terms first, second, or third are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
The terms "mounted, connected and connected" in the present invention are to be understood broadly, unless otherwise explicitly specified or limited, for example: can be fixedly connected, detachably connected or integrally connected; they may be mechanically, electrically, or directly connected, or indirectly connected through intervening media, or may be interconnected between two elements. The specific meanings of the above terms in the present invention can be understood in specific cases to those skilled in the art.
Example 1
Referring to fig. 1 to 23, a first embodiment of the present invention provides a multi-class non-maximum suppression method based on inter-class coverage ratio, including:
s1: and setting a threshold value of the confidence of the category, and deleting all the prediction boxes with the confidence of the category smaller than the threshold value in the prediction box set.
S2: and screening the prediction frames by calculating the overlapping degree between the prediction frames.
Specifically, the steps of screening the prediction box are as follows:
(1) assuming that the input prediction box set is B and the empty set is D;
(2) sorting the prediction frames in the set B in a descending order according to the category confidence;
(3) marking the prediction frame with the maximum class confidence as a reference frame and marking as M; marking the rest prediction boxes as comparison boxes and marking as Bo; defining the proportion of the overlapping area of the reference frame and the comparison frame in the comparison frame as COPBoMI.e., degree of overlap, as shown in fig. 2;
specifically, the degree of overlap COP is calculated as followsBoM：
Figure BDA0002790257540000071
Wherein S isMBoIs the area of the overlapping region of the reference frame M and the reference frame Bo, SBoThe COP (cover percent) is the proportion of the overlapping area of the two prediction frames in the specified prediction frame;
(4) set overlap threshold to 0.8 according to COPBoMJudging whether the reference frame M and the comparison frame Bo have an overlapping relation or not according to the relation with a set threshold value; if COPBoMIf the value is larger than the threshold value, the reference frame M and the comparison frame Bo are overlapped with each other, and the reference frame M and the comparison frame Bo are removed from the set B and added into the set D at the same time; if COPBoMIf the value is less than the threshold value, no overlapping relation is indicated, and the reference frame M is added into the set D while being removed from the set B;
(5) judging whether the set B is empty; if not, circularly screening the prediction frames in the set B according to the steps (2) to (4), and stopping circulation until the set B is empty; if the prediction frame is empty, the prediction frame in the set D is output.
S3: and judging the inclusion relation between the reference frame and the comparison frame by calculating the proportion of the overlapping area of the reference frame and the comparison frame in the reference frame.
Specifically, the step of determining the inclusion relationship between the reference frame and the comparison frame is as follows:
(1) sorting the prediction frames in the set B in an ascending order according to the areas of the prediction frames;
(2) marking the prediction box with the minimum area as a reference box M1; the remaining prediction boxes are labeled as control boxes Bo 1; defining the proportion of the overlapping area of the reference frame and the comparison frame in the reference frame as COPMBo；
Specifically, COP was calculated according to the following formulaMBo：
Figure BDA0002790257540000081
Wherein S isMIs the area of the reference frame, SMBoThe area of the overlapping region of the reference frame M1 and the control frame Bo 1;
(3) setting a proportional threshold according to COPMBoIn relation to the threshold, whether an inclusion relation exists between the reference frame M1 and the check frame Bo1 is judged; COP of reference frame and comparison frameMBoIf the ratio of the overlap area in the reference frame is larger than the set threshold, it is determined that the reference frame includes the reference frame, as shown in fig. 4; COP of reference frame and comparison frameMBoIf the comparison frame is smaller than the set threshold, the comparison frame is judged not to contain the reference frame.
S4: if the reference frame is not included in the check frame, the reference frame M1 is removed from the prediction frame set B and the reference frame M1 is added to the empty set D.
S5: if the check box Bo1 contains the benchmark box M1, then the minimum area box N is selected in the check box Bo1, and the benchmark box M1 and the minimum area box N are screened by using an inter-class preferred selection strategy.
It should be noted that the inter-class preference policy is divided into preference between prediction frames of different classes and preference between prediction frames of the same class.
Specifically, it is determined whether the reference frame M1 is the same type as the minimum frame N, see fig. 5;
(1) if the categories are different, the prediction frames with high category confidence degrees are reserved, and the prediction frames with low category confidence degrees are removed;
(2) and if the types are the same, calculating the difference value of the confidence degrees of the types of the two prediction frames.
If the difference value is larger than the set difference value threshold value, reserving the prediction frame with larger category confidence coefficient, and removing the other prediction frame;
and if the difference is smaller than the set threshold, merging the two prediction boxes, namely reserving the prediction box with larger area and reserving a larger class confidence value.
It should be noted that, in this embodiment, three times of inter-class preference are completed, fig. 6 is a defect detection diagram before implementation of inter-class preference, four prediction frames (numbered 1-4) with the same class (all "Foreign _ M") are retained in the diagram, and the prediction frames and the class confidence coefficients thereof are sequentially: 1: 0.94, 2: 0.60, 3: 0.58, 4: 0.44.
in the first inter-class selection, the prediction frame 1 with the minimum mark area is used as a reference frame, the prediction frame 2 is used as a comparison frame, the comparison frame can be judged to be larger in area than the reference frame through calculation, the confidence coefficient of the reference frame is larger than that of the comparison frame, and meanwhile, the difference value of the confidence coefficient of the reference frame and the confidence coefficient of the comparison frame is smaller than a set threshold value; the two prediction boxes are merged, i.e. the prediction box 2 is retained while the class confidence value of 0.94 for the prediction box 1 is given to the prediction box 2, and the prediction box 1 is removed.
As shown in fig. 7, after the first inter-class preference is completed, the retained prediction frames and the class confidence coefficients thereof are: 2: 0.94, 3: 0.58, 4: 0.44;
in the second inter-class preference selection, the difference between the confidence degrees of the classes of the prediction boxes 2 and 3 is also smaller than the set threshold, so that the confidence degree of the class of the prediction box 3 is kept at 0.94, and the prediction box 2 is removed, as shown in fig. 8;
in the third inter-class selection, the class confidence difference between the prediction frames 3 and 4 is greater than the set threshold, so that the prediction frame 3 is directly reserved, and the prediction frame 2 is removed, as shown in fig. 9;
preferably, defect localization accuracy is improved by inter-class preference.
S6: judging whether the prediction frame set B is empty or not, and if not, continuously screening out the rest prediction frames; otherwise, an empty set is output.
If the prediction box set B is not empty, circulating the prediction boxes in the set B according to the steps S3-S5 until the prediction box set B is empty, and stopping circulation; if the prediction frame is empty, the prediction frame in the set D is output.
In order to verify the technical effect adopted in the method, the present embodiment selects the current multi-class NMS method and adopts the method to perform a comparison test, and compares the test results by a scientific demonstration means to verify the real effect of the method.
The current multi-class NMS method is not sufficient for removing redundancy and positioning defects;
in order to verify that the method has higher accuracy for measuring the overlapping degree and accuracy for locating the defect than the current multi-class NMS (Non-Maximum Suppression) method, the current multi-class NMS method and the method are adopted in the embodiment to respectively perform training and test comparison in the actual defect sample.
The data set comprises 2200 defect samples and 8 category defects in total, wherein the training set comprises 2000 samples, the test set comprises 200 samples, three defect samples are respectively shown in fig. 10, 11 and 12, wherein fig. 10 is sample 1 and contains a Foreign _ M defect; FIG. 11 is sample 2, containing Gold _ P defects; FIG. 12 is sample 3, containing a Raw _ M defect; the detection results of the current multi-class NMS method in the three defect samples are shown in fig. 13, 14, and 15, respectively, and the detection results of the method in the three defect samples are shown in fig. 16, 17, and 18.
(1) Referring to fig. 13, it can be seen that 4 prediction boxes are reserved after the defect detection of the current multi-class NMS algorithm, the prediction boxes are mutually overlapped, the redundancy removal is not sufficient, and the defect location is not accurate enough; after the detection of the method, as shown in fig. 16, a prediction box is reserved, which indicates that the confidence of the defect type is high, the defect location is more accurate, and other samples all have the effect.
(2) In order to verify that the method can accurately measure the overlapping degree of prediction frames with different areas, taking the current multi-class NMS method and the detection result of the method in a 'Gold _ P' defect sample as an example, the IoU (Intersection over Unit, the ratio of the Intersection area of two prediction frames to the Union area) parameter calculation method and the COP parameter calculation method of the previous multi-class NMS method are respectively adopted to measure the overlapping degree of the prediction frames; fig. 20 shows the detection result of the current multi-class NMS method, where the coordinates of prediction boxes 1 and 2 are:
Figure BDA0002790257540000101
the respective areas and intersection areas of the prediction boxes 1, 2 can be calculated as:
Figure BDA0002790257540000102
IoU of prediction blocks 1 and 2 can be obtained by the calculation method of IoU parameter in the current multi-class NMS method (as in fig. 19):
Figure BDA0002790257540000103
it can be seen that IoU of the current prediction blocks 1 and 2 is smaller than the set threshold (usually about 0.7), and the prediction blocks 1 and 2 actually overlap each other, however IoU fails to measure the overlapping degree accurately, resulting in incomplete redundant block removal.
Secondly, the COP of the prediction frames 1 and 2 can be obtained by the calculation method of the COP parameters in the method:
Figure BDA0002790257540000104
it can be found that the COP of prediction boxes 1 and 2 obtained by the COP parameter calculation method in the method can accurately measure the overlapping degree, and the result after the processing by the method is shown in fig. 21.
(3) Taking the detection result of the current multi-class NMS algorithm and the technical scheme of the invention in the Incomplate _ B defect sample as an example; as shown in fig. 22, after detecting a defect sample by the current multi-class NMS method, two different classes of prediction boxes are retained at the same defect location, namely, Incomplete _ B: 0.96 and Gold _ P: 0.11, however, this does not meet the requirement of defect type incompatibility in defect detection.
The non-maximum suppression scheme with incompatible categories constructed by the invention can ensure that only one category prediction box is reserved after detection, namely Incomplite _ B: 0.96, as in FIG. 23.
Example 2
Referring to fig. 24 to 25, a second embodiment of the present invention is a 2 nd embodiment, which is different from embodiment 1 in that the present embodiment proposes a COP calculating an overlap degreeBoMThe method of (1), comprising:
s1: and taking the prediction frame with the maximum class confidence as a reference frame, and taking the other prediction frames except the prediction frame with the maximum class confidence as comparison frames.
S2: calculating the proportion of the overlapping area of the reference frame and the comparison frame in the reference frame, namely COPBoM。
The calculation formula is as follows:
Figure BDA0002790257540000111
example 3
Referring to fig. 26 to 33, in a 3 rd embodiment of the present invention, unlike embodiments 1 and 2, this embodiment proposes a multi-class non-maximum suppression system based on an inter-class coverage ratio, including:
an input module 100 is used for inputting the prediction box set and the empty set into the system.
And a roughing module 200 connected with the input module 100 and used for screening out redundant boxes with very low category confidence.
The Score-NMS module 300 is connected with the roughing module 200 and used for receiving the prediction frame set and the empty set transmitted by the roughing module 200 and screening out the prediction frames with higher overlapping degree; specifically, referring to fig. 28, the Score-NMS module 300 screens out the prediction frames with higher overlap (not less than 0.8) by calculating COP to measure the overlap between the prediction frames based on the non-maximum suppression of the category confidence (Score) and using the category confidence as a standard.
An Area-NMS module 400 connected to the Score-NMS module 300, for further screening the prediction frames left by the Score-NMS module 300; specifically, referring to fig. 29, the Area-NMS module 400 determines whether there is an inclusion relationship between prediction frames by calculating COP based on non-maximum suppression of an Area (Area) and using the Area of the prediction frames as a standard, and preferably selects the prediction frames included with each other by an inter-class preferred selection method, so that the remaining prediction frames are better positioned and the class confidence is higher.
To verify the technical effect of the system, the present embodiment tests the roughing module 200, the Score-NMS module 300 and the Area-NMS module 400 respectively, and compares the test results with scientific demonstration means to verify the real effect of the system.
As shown in fig. 30, 1000 prediction boxes are input in total, and the position coordinates and the class confidence are as follows.
Position coordinates of 1000 prediction frames are:
Figure BDA0002790257540000121
the class confidence of 1000 prediction boxes is:
Figure BDA0002790257540000122
(1) the rough selection module 200 screens out the prediction frames with the class confidence lower than 0.05 by setting the class confidence threshold, for example, set to 0.05 in this embodiment, and retains 87 prediction frames after the screening, for example, as shown in fig. 31, and the position coordinates and the class confidence of the 87 prediction frames are as follows.
Position coordinates of the 87 prediction frames are:
Figure BDA0002790257540000131
class confidence of 87 prediction boxes is as follows:
Figure BDA0002790257540000132
(2) the prediction blocks with larger overlap are screened out by the Score-NMS module 300, and COP is setBoMThe threshold, set to 0.8 in this embodiment, is continuously iterated to calculate the COP of the prediction box with the maximum class confidence and the remaining prediction boxesBoMRemoval of COPBoMThe corresponding prediction box with value greater than 0.8, and finally 4 prediction boxes are reserved, as shown in fig. 32.
(3) Then, the prediction boxes with the inclusion relationship are merged by an inter-class preferred selection method in the Area-NMS module, as shown in FIG. 33, and the last retained prediction box has higher class confidence (0.94) and better defect localization.
Example 4
Referring to fig. 34 to 35, a 4 th embodiment of the present invention is different from embodiment 3 in that the present embodiment proposes a multi-class non-maximum suppression system, including:
an input module 100 is used for inputting the prediction box set and the empty set into the system.
And a roughing module 200 connected with the input module 100 and used for screening out redundant boxes with very low category confidence.
The Area-NMS module 300 is connected with the roughing module 200 and is used for receiving the prediction frame set and the empty set transmitted by the roughing module 200 and screening out the prediction frames with inclusion relation; specifically, the Area-NMS module 400 determines whether there is an inclusion relationship between prediction frames by calculating COP based on non-maximum inhibition of Area (Area) and using the Area of the prediction frames as a standard, and preferably, preferentially selects the mutually included prediction frames by an inter-class preferred selection method, so that the remaining prediction frames are better positioned and the class confidence is higher.
A Score-NMS module 400 connected to the Area-NMS module 300, for receiving the prediction frame set and the empty set from the Area-NMS module 300 and for screening out the prediction frames with higher overlapping degree; specifically, the Score-NMS module 400 screens out the prediction frames with higher overlap (not less than 0.8) by calculating COP to measure the overlap degree between the prediction frames based on the non-maximum suppression of the category confidence (Score) and using the category confidence as a standard.
It should be recognized that embodiments of the present invention can be realized and implemented by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer readable memory. The methods may be implemented in a computer program using standard programming techniques, including a non-transitory computer-readable storage medium configured with the computer program, where the storage medium so configured causes a computer to operate in a specific and predefined manner, according to the methods and figures described in the detailed description. Each program may be implemented in a high level procedural or object oriented programming language to communicate with a computer system. However, the program(s) can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language. Furthermore, the program can be run on a programmed application specific integrated circuit for this purpose.
Further, the operations of processes described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes described herein (or variations and/or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) collectively executed on one or more processors, by hardware, or combinations thereof. The computer program includes a plurality of instructions executable by one or more processors.
Further, the method may be implemented in any type of computing platform operatively connected to a suitable interface, including but not limited to a personal computer, mini computer, mainframe, workstation, networked or distributed computing environment, separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, and the like. Aspects of the invention may be embodied in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optically read and/or write storage medium, RAM, ROM, or the like, such that it may be read by a programmable computer, which when read by the storage medium or device, is operative to configure and operate the computer to perform the procedures described herein. Further, the machine-readable code, or portions thereof, may be transmitted over a wired or wireless network. The invention described herein includes these and other different types of non-transitory computer-readable storage media when such media include instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor. The invention also includes the computer itself when programmed according to the methods and techniques described herein. A computer program can be applied to input data to perform the functions described herein to transform the input data to generate output data that is stored to non-volatile memory. The output information may also be applied to one or more output devices, such as a display. In a preferred embodiment of the invention, the transformed data represents physical and tangible objects, including particular visual depictions of physical and tangible objects produced on a display.
As used in this application, the terms "component," "module," "system," and the like are intended to refer to a computer-related entity, either hardware, firmware, a combination of hardware and software, or software in execution. For example, a component may be, but is not limited to being: a process running on a processor, an object, an executable, a thread of execution, a program, and/or a computer. By way of example, both an application running on a computing device and the computing device can be a component. One or more components can reside within a process and/or thread of execution and a component can be localized on one computer and/or distributed between two or more computers. In addition, these components can execute from various computer readable media having various data structures thereon. The components may communicate by way of local and/or remote processes such as in accordance with a signal having one or more data packets (e.g., data from one component interacting with another component in a local system, distributed system, and/or across a network such as the internet with other systems by way of the signal).
It should be noted that the above-mentioned embodiments are only for illustrating the technical solutions of the present invention and not for limiting, and although the present invention has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions may be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, which should be covered by the claims of the present invention.

Claims (9)

1. A multi-class non-maximum suppression method based on inter-class coverage ratio is characterized in that: comprises the steps of (a) preparing a mixture of a plurality of raw materials,
setting a category confidence threshold, and deleting all the prediction frames in the prediction frame set, wherein the category confidence is smaller than the threshold;
screening the prediction frames by calculating the overlapping degree between the prediction frames;
judging the inclusion relation between the reference frame and the comparison frame by calculating the proportion of the overlapping area of the reference frame and the comparison frame in the reference frame; if the proportion is smaller than a set proportion threshold value, judging that the reference frame is not contained in the comparison frame, removing the reference frame from the prediction frame set, and adding the reference frame to an empty set;
otherwise, judging that the comparison frame comprises the reference frame, selecting a frame with the smallest area in the comparison frame, and screening the reference frame and the smallest frame by utilizing an inter-class preferred selection strategy;
judging whether the prediction frame set is empty or not, and if not, continuously screening out the rest prediction frames; otherwise, outputting the empty set.
2. The inter-class coverage ratio-based multi-class non-maximum suppression method according to claim 1, wherein: the calculating of the degree of overlap includes,
sorting the prediction frames in the prediction frame set B in a descending order according to the category confidence degrees;
the degree of overlap is as follows:
Figure FDA0002790257530000011
wherein the COPBoMThe ratio S of the overlapping area of the reference frame M and the comparison frame Bo in the comparison frameMBoIs the area of the overlapping region of the reference frame M and the reference frame Bo, SBoThe reference frame M is the prediction frame with the maximum class confidence coefficient, and the comparison frame Bo is the rest prediction frames.
3. The inter-class coverage ratio-based multi-class non-maximum suppression method according to claim 2, wherein: the screening prediction box comprises a filter-based prediction box,
if the COPBoMIf the reference frame M and the comparison frame Bo are larger than the overlap threshold value, removing the reference frame M and the comparison frame Bo from the prediction frame set, and adding the reference frame M to the empty set D;
otherwise, the reference frame M is removed from the prediction frame set B, while the reference frame M is added to the empty set D.
4. The inter-class coverage ratio-based multi-class non-maximum suppression method according to claim 3, wherein: the screening prediction box further comprises a filter prediction box,
judging whether the prediction frame set B is empty or not, and if not, continuously screening the prediction frames; otherwise, outputting the prediction frame in the empty set D.
5. The inter-class coverage ratio-based multi-class non-maximum suppression method according to claim 3 or 4, wherein: the overlap threshold value may comprise a value of,
the overlap threshold is equal to 0.8.
6. The inter-class coverage ratio-based multi-class non-maximum suppression method according to any one of claims 1, 2 and 3, wherein: the calculating of the ratio includes calculating a ratio of,
sorting the prediction frames in the prediction frame set B in an ascending order according to the areas of the prediction frames;
calculating the proportion COP of the overlapped area of the reference frame and the comparison frame in the reference frame according to the following formulaMBo：
Figure FDA0002790257530000021
Wherein S isMThe area of the reference frame is marked as a reference frame M1; remaining predictionThe box is labeled as control box Bo 1.
7. The inter-class coverage ratio-based multi-class non-maximum suppression method according to claim 6, wherein: the inter-class preference policy includes,
judging whether the reference frame M1 is the same as the minimum frame N in type, if so, retaining the prediction frame with high type confidence coefficient, and removing the prediction frame with low type confidence coefficient;
and if the categories are the same, calculating a category confidence difference value of the two prediction frames.
8. The inter-class coverage ratio-based multi-class non-maximum suppression method according to claim 7, wherein: the category confidence differences include, for example,
if the difference is larger than the set difference threshold, the prediction frame with higher category confidence coefficient is reserved, and the other prediction frame is removed;
otherwise, reserving the prediction box with larger area and the prediction box with higher class confidence value.
9. A multi-class non-maximum suppression system based on inter-class coverage ratio is characterized in that: comprises the steps of (a) preparing a mixture of a plurality of raw materials,
an input module (100) for inputting a set of prediction boxes and an empty set to the system;
a roughing module (200) connected with the input module (100) and used for screening out redundant boxes with very low category confidence;
a Score-NMS module (300) connected to the roughing module (200) for screening out the prediction frames with higher overlap;
an Area-NMS module (400) connected to the Score-NMS module (300) for further screening the prediction box left by the Score-NMS module (300).
CN202011312543.9A 2020-11-20 2020-11-20 Multi-class non-maximum inhibition method and system based on inter-class coverage ratio Active CN112419263B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011312543.9A CN112419263B (en) 2020-11-20 2020-11-20 Multi-class non-maximum inhibition method and system based on inter-class coverage ratio

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011312543.9A CN112419263B (en) 2020-11-20 2020-11-20 Multi-class non-maximum inhibition method and system based on inter-class coverage ratio

Publications (2)

Publication Number Publication Date
CN112419263A true CN112419263A (en) 2021-02-26
CN112419263B CN112419263B (en) 2023-04-14

Family

ID=74778447

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011312543.9A Active CN112419263B (en) 2020-11-20 2020-11-20 Multi-class non-maximum inhibition method and system based on inter-class coverage ratio

Country Status (1)

Country Link
CN (1) CN112419263B (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113281707A (en) * 2021-05-26 2021-08-20 上海电力大学 Sound source positioning method based on windowed LASSO under strong noise
CN113869307A (en) * 2021-09-16 2021-12-31 中国农业科学院农业资源与农业区划研究所 Cross-category fruit data set automatic labeling system and method

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108268869A (en) * 2018-02-13 2018-07-10 北京旷视科技有限公司 Object detection method, apparatus and system
CN108875577A (en) * 2018-05-11 2018-11-23 深圳市易成自动驾驶技术有限公司 Object detection method, device and computer readable storage medium
CN108960174A (en) * 2018-07-12 2018-12-07 广东工业大学 A kind of object detection results optimization method and device
CN109101897A (en) * 2018-07-20 2018-12-28 中国科学院自动化研究所 Object detection method, system and the relevant device of underwater robot
CN109359577A (en) * 2018-10-08 2019-02-19 福州大学 A kind of Complex Background number detection system based on machine learning
CN110414417A (en) * 2019-07-25 2019-11-05 电子科技大学 A kind of traffic mark board recognition methods based on multi-level Fusion multi-scale prediction
CN111310759A (en) * 2020-02-13 2020-06-19 中科智云科技有限公司 Target detection suppression optimization method and device for dual-mode cooperation
CN111368600A (en) * 2018-12-26 2020-07-03 北京眼神智能科技有限公司 Method and device for detecting and identifying remote sensing image target, readable storage medium and equipment
CN111738263A (en) * 2020-08-24 2020-10-02 北京易真学思教育科技有限公司 Target detection method and device, electronic equipment and storage medium

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108268869A (en) * 2018-02-13 2018-07-10 北京旷视科技有限公司 Object detection method, apparatus and system
CN108875577A (en) * 2018-05-11 2018-11-23 深圳市易成自动驾驶技术有限公司 Object detection method, device and computer readable storage medium
CN108960174A (en) * 2018-07-12 2018-12-07 广东工业大学 A kind of object detection results optimization method and device
CN109101897A (en) * 2018-07-20 2018-12-28 中国科学院自动化研究所 Object detection method, system and the relevant device of underwater robot
CN109359577A (en) * 2018-10-08 2019-02-19 福州大学 A kind of Complex Background number detection system based on machine learning
CN111368600A (en) * 2018-12-26 2020-07-03 北京眼神智能科技有限公司 Method and device for detecting and identifying remote sensing image target, readable storage medium and equipment
CN110414417A (en) * 2019-07-25 2019-11-05 电子科技大学 A kind of traffic mark board recognition methods based on multi-level Fusion multi-scale prediction
CN111310759A (en) * 2020-02-13 2020-06-19 中科智云科技有限公司 Target detection suppression optimization method and device for dual-mode cooperation
CN111738263A (en) * 2020-08-24 2020-10-02 北京易真学思教育科技有限公司 Target detection method and device, electronic equipment and storage medium

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
罗鹏等: "基于组合式目标检测框架的低漏报率缺陷识别方法", 《高电压技术》 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113281707A (en) * 2021-05-26 2021-08-20 上海电力大学 Sound source positioning method based on windowed LASSO under strong noise
CN113869307A (en) * 2021-09-16 2021-12-31 中国农业科学院农业资源与农业区划研究所 Cross-category fruit data set automatic labeling system and method

Also Published As

Publication number Publication date
CN112419263B (en) 2023-04-14

Similar Documents

Publication Publication Date Title
CN111862067B (en) Welding defect detection method and device, electronic equipment and storage medium
KR102171491B1 (en) Method for sorting products using deep learning
CN112419263B (en) Multi-class non-maximum inhibition method and system based on inter-class coverage ratio
CN109685199B (en) Method and apparatus for creating table containing information on pooling type, and test method and test apparatus using the same
EP2734946B1 (en) Systems and methods for three dimensional printing
CN108921166A (en) Medical bill class text detection recognition method and system based on deep neural network
CN110598761A (en) Dispensing detection method and device and computer readable storage medium
TWI715051B (en) Machine learning method and automatic optical inspection device using the method thereof
CN110264444A (en) Damage detecting method and device based on weak segmentation
KR20210024767A (en) Method for non-destructive inspection based on image and Computer-readable storage medium
CN113222913B (en) Circuit board defect detection positioning method, device and storage medium
CN113763340A (en) Automatic grading method based on multitask deep learning ankylosing spondylitis
CN117372424B (en) Defect detection method, device, equipment and storage medium
US20130231897A1 (en) Systems and methods for efficient analysis of topographical models
CN110489830A (en) The design detection method and computer storage medium of pcb board card
CN103837135A (en) Workpiece detecting method and system
JP2019158684A (en) Inspection system, identification system, and discriminator evaluation device
CN116468702A (en) Chloasma assessment method, device, electronic equipment and computer readable storage medium
CN112800870B (en) Non-maximum inhibition method and system based on interlayer coverage ratio
CN106326098A (en) Method and device for testing page perceived performance
JPH1185950A (en) Picture processing device and method therefor and computer readable recording medium
CN114418945A (en) Photovoltaic module hot spot detection method and device
CN112418590A (en) Printed circuit board component detection method and system
CN110163841A (en) Detection method, device, equipment and the storage medium of body surface defect
CN113994378A (en) Inspection device, inspection method, inspection program, learning device, learning method, and learning program

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant