CN112419263B - Multi-class non-maximum inhibition method and system based on inter-class coverage ratio - Google Patents
Multi-class non-maximum inhibition method and system based on inter-class coverage ratio Download PDFInfo
- Publication number
- CN112419263B CN112419263B CN202011312543.9A CN202011312543A CN112419263B CN 112419263 B CN112419263 B CN 112419263B CN 202011312543 A CN202011312543 A CN 202011312543A CN 112419263 B CN112419263 B CN 112419263B
- Authority
- CN
- China
- Prior art keywords
- frame
- prediction
- class
- reference frame
- inter
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 91
- 230000005764 inhibitory process Effects 0.000 title abstract description 7
- 238000012216 screening Methods 0.000 claims abstract description 36
- 230000001629 suppression Effects 0.000 claims description 50
- 230000001174 ascending effect Effects 0.000 claims description 3
- 239000000203 mixture Substances 0.000 claims 1
- 239000002994 raw material Substances 0.000 claims 1
- 239000000758 substrate Substances 0.000 claims 1
- 230000007547 defect Effects 0.000 abstract description 55
- 238000001514 detection method Methods 0.000 description 38
- 238000010586 diagram Methods 0.000 description 36
- 238000004364 calculation method Methods 0.000 description 12
- PCHJSUWPFVWCPO-UHFFFAOYSA-N gold Chemical compound [Au] PCHJSUWPFVWCPO-UHFFFAOYSA-N 0.000 description 8
- 239000010931 gold Substances 0.000 description 8
- 229910052737 gold Inorganic materials 0.000 description 8
- 238000004422 calculation algorithm Methods 0.000 description 7
- 238000013527 convolutional neural network Methods 0.000 description 6
- 238000003860 storage Methods 0.000 description 6
- 238000012360 testing method Methods 0.000 description 6
- 238000004590 computer program Methods 0.000 description 5
- 230000000694 effects Effects 0.000 description 5
- 230000000717 retained effect Effects 0.000 description 5
- 238000010187 selection method Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 2
- 230000004807 localization Effects 0.000 description 2
- 238000012549 training Methods 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 239000012212 insulator Substances 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 239000002184 metal Substances 0.000 description 1
- 229910052751 metal Inorganic materials 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000002245 particle Substances 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
- G06T7/0006—Industrial image inspection using a design-rule based approach
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/60—Analysis of geometric attributes
- G06T7/62—Analysis of geometric attributes of area, perimeter, diameter or volume
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Geometry (AREA)
- Quality & Reliability (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a multi-class non-maximum inhibition method and a system based on an inter-class coverage ratio, which comprises the steps of setting a class confidence coefficient threshold value, and deleting all prediction frames with class confidence coefficients smaller than the threshold value; screening the prediction frames by calculating the overlapping degree between the prediction frames; judging the inclusion relation between the reference frame and the comparison frame by calculating the proportion of the overlapping area of the reference frame and the comparison frame in the reference frame; if the proportion is smaller than the proportion threshold value, judging that the reference frame does not contain the reference frame, removing the reference frame from the prediction frame set, and adding the reference frame to the empty set; otherwise, judging that the comparison frame comprises a reference frame, selecting a frame with the minimum area in the comparison frame, and screening the reference frame and the minimum frame by utilizing an inter-class preferred selection strategy; if the prediction frame set is not empty, continuously screening out the rest prediction frames; otherwise, an empty set is output. The invention can accurately measure the overlapping degree of prediction frames with different areas and is suitable for detecting various defects.
Description
Technical Field
The invention relates to the technical field of machine vision, in particular to a multi-class non-maximum inhibition method and system based on an inter-class coverage ratio.
Background
The defect detection utilizes machine vision equipment to acquire images to judge whether defects exist in the acquired images and simultaneously realizes the output of the positions and the types of the detected defects; in recent years, with the rapid development of deep learning technology, the defect detection algorithm has also shifted from the traditional algorithm based on artificial features to the detection technology based on a deep Neural network, and a deep learning model represented by a Convolutional Neural Network (CNN) is also successfully applied in defect detection of many products.
The CNN-based defect detection method has many similarities with the current mainstream CNN-based target detection method, taking the fast R-CNN Network as target detection or defect detection as an example, extracting a feature map through a convolution Network, sending the feature map into an area extraction Network (RPN) to generate a large number of prediction boxes possibly containing targets to be detected or defects, classifying and regressing to reserve about 2000 prediction boxes, and removing redundant prediction boxes through some algorithms.
However, the CNN-based target detection method cannot be directly applied to defect detection of products, because in the target detection, one large-sized target may include one or several small-sized targets, but in the defect detection, the inside of one large-sized defect does not include other small-sized defects, i.e., the defect types are incompatible; in addition, several homogeneous defects should be identified as a whole when they are grouped together.
Disclosure of Invention
This section is for the purpose of summarizing some aspects of embodiments of the invention and to briefly introduce some preferred embodiments. In this section, as well as in the abstract and the title of the invention of this application, simplifications or omissions may be made to avoid obscuring the purpose of the section, the abstract and the title, and such simplifications or omissions are not intended to limit the scope of the invention.
The present invention has been made in view of the above-mentioned conventional problems.
Therefore, the invention provides a multi-class non-maximum inhibition method based on inter-class coverage ratio, which can accurately reflect the overlapping degree between prediction frames with large area difference, can screen out different types of prediction frames and combine the same type of prediction frames in the detection process, and solves the problems of inaccurate screening of the prediction frames and limited application range in the prior art.
In order to solve the technical problems, the invention provides the following technical scheme: setting a category confidence threshold value, and deleting all the prediction frames with the category confidence values smaller than the threshold value in the prediction frame set; screening the prediction frames by calculating the overlapping degree between the prediction frames; judging the inclusion relation between the reference frame and the comparison frame by calculating the proportion of the overlapping area of the reference frame and the comparison frame in the reference frame; if the proportion is smaller than a set proportion threshold value, judging that the reference frame is not contained in the comparison frame, removing the reference frame from the prediction frame set, and adding the reference frame to an empty set; otherwise, judging that the comparison frame comprises the reference frame, selecting a frame with the smallest area in the comparison frame, and screening the reference frame and the smallest frame by utilizing an inter-class preferred selection strategy; judging whether the prediction frame set is empty or not, and if not, continuously screening out the rest prediction frames; otherwise, outputting the empty set.
As a preferable scheme of the multi-class non-maximum suppression method based on the inter-class coverage ratio, in the present invention, wherein: the calculating the overlapping degree comprises sorting the prediction frames in the prediction frame set B in a descending order according to the category confidence degrees; the degree of overlap is as follows:
wherein the COP BoM The ratio S of the overlapping area of the reference frame M and the comparison frame Bo in the comparison frame MBo Is the area of the overlapping region of the reference frame M and the reference frame Bo, S Bo The reference frame M is the prediction frame with the maximum class confidence coefficient, and the comparison frame Bo is the rest prediction frames.
As a preferred embodiment of the multi-class non-maximum suppression method based on the inter-class coverage ratio, in the present invention: the screening prediction box includes, if the COP BoM If the reference frame M and the comparison frame Bo are larger than the overlap threshold value, removing the reference frame M and the comparison frame Bo from the prediction frame set, and adding the reference frame M to the empty set D; otherwise, the reference frame M is removed from the prediction frame set B, while the reference frame M is added to the empty set D.
As a preferred embodiment of the multi-class non-maximum suppression method based on the inter-class coverage ratio, in the present invention: the screening of the prediction frame further comprises the steps of judging whether the prediction frame set B is empty, and if not, continuing to screen the prediction frame; otherwise, outputting the prediction frame in the empty set D.
As a preferable scheme of the multi-class non-maximum suppression method based on the inter-class coverage ratio, in the present invention, wherein: the overlap threshold comprises the overlap threshold being equal to 0.8.
As a preferred embodiment of the multi-class non-maximum suppression method based on the inter-class coverage ratio, in the present invention: the calculation of the proportion comprises the steps of sorting the prediction frames in the prediction frame set B in an ascending order according to the areas of the prediction frames; calculating the proportion COP of the overlapped area of the reference frame and the comparison frame in the reference frame according to the following formula MBo ：
Wherein S is M The area of the reference frame is marked as the reference frame M1 by the prediction frame with the minimum area; the rest of the prediction boxes are marked as reference boxes Bo1;
as a preferable scheme of the multi-class non-maximum suppression method based on the inter-class coverage ratio, in the present invention, wherein: the inter-class preferred selection strategy comprises the steps of judging whether the classes of the reference frame M1 and the minimum frame N are the same or not, if the classes are different, reserving a prediction frame with high class confidence coefficient, and removing the prediction frame with low class confidence coefficient; and if the categories are the same, calculating a category confidence difference value of the two prediction frames.
As a preferred embodiment of the multi-class non-maximum suppression method based on the inter-class coverage ratio, in the present invention: the category confidence difference comprises that if the difference is larger than a set difference threshold, a prediction frame with higher category confidence is reserved, and another prediction frame is removed; otherwise, reserving the prediction box with larger area and the prediction box with higher class confidence value.
As a preferable solution of the multi-class non-maximum suppression system based on the inter-class coverage ratio according to the present invention, wherein: the system comprises an input module, a prediction module and a prediction module, wherein the input module is used for inputting a prediction box set and an empty set into the system; the rough selection module is connected with the input module and is used for screening out redundant boxes with very low category confidence coefficients; the Score-NMS module is connected with the rough selection module and is used for screening out the prediction frames with higher overlapping degree; and the Area-NMS module is connected with the Score-NMS module and is used for further screening the prediction frames screened by the Score-NMS module.
The invention has the beneficial effects that: the COP parameter is provided, and the overlapping degree of prediction frames with different areas can be accurately measured; the defect positioning accuracy is improved by applying an inter-class preferred selection strategy; in addition, the invention is suitable for detecting various defects, and is also suitable for detecting appearance defects of various objects such as semiconductors, high-speed railway line fasteners, insulators of transmission towers, texture surfaces, metal surfaces and the like.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the description below are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without inventive labor. Wherein:
fig. 1 is a schematic flowchart of a multi-class non-maximum suppression method based on inter-class coverage ratio according to a first embodiment of the present invention;
FIG. 2 is a diagram illustrating COP of a multi-class non-maximum suppression method based on inter-class coverage ratio according to a first embodiment of the present invention BoM A schematic diagram of a calculation method;
FIG. 3 is a diagram illustrating COP of a multi-class non-maximum suppression method based on inter-class coverage ratio according to a first embodiment of the present invention MBo A schematic diagram of a calculation method;
FIG. 4 is a diagram illustrating the predicted box containment relationship of a multi-class non-maximum suppression method based on inter-class coverage ratio according to a first embodiment of the present invention;
fig. 5 is a schematic flowchart of an inter-class preferred selection method of a multi-class non-maximum suppression method based on an inter-class coverage ratio according to a first embodiment of the present invention;
fig. 6 is a schematic diagram illustrating defect detection before inter-class selection is performed by a multi-class non-maximum suppression method based on inter-class coverage ratio according to a first embodiment of the present invention;
fig. 7 is a schematic diagram illustrating defect detection after completion of first inter-class selection of a multi-class non-maximum suppression method based on inter-class coverage ratio according to a first embodiment of the present invention;
FIG. 8 is a schematic diagram illustrating defect detection after completion of second inter-class preferential selection of a multi-class non-maximum suppression method based on inter-class coverage ratio according to a first embodiment of the present invention;
fig. 9 is a schematic diagram illustrating defect detection after completion of third inter-class preferential selection in a multi-class non-maximum suppression method based on inter-class coverage ratio according to a first embodiment of the present invention;
FIG. 10 is a sample diagram of a Foreign _ M-containing defect based on inter-class coverage ratio multi-class non-maximum suppression method according to a first embodiment of the present invention;
FIG. 11 is a diagram illustrating a sample containing Gold _ P defects of a multi-class non-maximum suppression method based on an inter-class coverage ratio according to a first embodiment of the present invention;
FIG. 12 is a diagram illustrating a sample containing a Raw _ M defect according to a first embodiment of the present invention;
fig. 13 is a schematic diagram illustrating a detection result of the method of multi-class NMS for detecting a forking _ M defect sample according to the present method of multi-class non-maximum suppression based on the inter-class coverage ratio according to the first embodiment of the present invention;
fig. 14 is a schematic diagram illustrating a detection result of Gold _ P defect samples by the current multi-class NMS method of a multi-class non-maximum suppression method based on an inter-class coverage ratio according to a first embodiment of the present invention;
fig. 15 is a schematic diagram illustrating a detection result of a Raw _ M defect sample by a current multi-class NMS method of a multi-class non-maximal suppression method based on an inter-class coverage ratio according to a first embodiment of the present invention;
FIG. 16 is a schematic diagram illustrating the detection result of the method for multi-class non-maximum rejection based on the inter-class coverage ratio for the Foreign _ M defect samples according to the first embodiment of the present invention;
FIG. 17 is a diagram illustrating the detection result of a Gold _ P defect sample by the method of the present invention based on the inter-class coverage ratio multi-class non-maximum suppression method according to the first embodiment of the present invention;
fig. 18 is a schematic diagram illustrating a detection result of a Raw _ M defect sample according to a multi-class non-maximum suppression method based on an inter-class coverage ratio according to a first embodiment of the present invention;
fig. 19 is a schematic diagram of a method for calculating IoU in a multi-class non-maximum suppression method based on inter-class coverage ratio according to a first embodiment of the present invention;
fig. 20 is a schematic diagram of a detection result of Gold _ P detected by using IoU to measure the degree of overlap according to a multi-class non-maximum suppression method based on an inter-class coverage ratio according to a first embodiment of the present invention;
fig. 21 is a schematic diagram illustrating a detection result of a multi-class non-maximum suppression method using COP measurement overlap detection Gold _ P according to a first embodiment of the present invention;
fig. 22 is a schematic diagram illustrating a detection result of the current multi-class NMS method for Incomplete _ B based on a multi-class non-maximum suppression method for inter-class coverage ratio according to the first embodiment of the present invention;
fig. 23 is a schematic diagram illustrating a detection result of the method for multi-class non-maximum suppression based on inter-class coverage ratio to Incomplete _ B according to the first embodiment of the present invention;
FIG. 24 is a COP for calculating the degree of overlap according to a second embodiment of the present invention BoM A schematic flow diagram of the method of (1);
FIG. 25 is a COP calculated as the degree of overlap according to the second embodiment of the present invention BoM Method of (3) COP BoM A schematic diagram of a calculation method;
FIG. 26 is a block diagram illustrating a multi-class non-maximum rejection system based on inter-class coverage ratio according to a third embodiment of the present invention;
fig. 27 is a schematic diagram of a network topology of a multi-class non-maximum suppression method based on inter-class coverage ratio according to a third embodiment of the present invention;
FIG. 28 is a schematic diagram of the algorithm flow of the Score-NMS module 300 according to a third embodiment of the present invention;
fig. 29 is a schematic flowchart of an Area-NMS module 400 algorithm of a multi-class non-maximum suppression method based on an inter-class coverage ratio according to a third embodiment of the present invention;
FIG. 30 is a diagram illustrating a prediction box of the input of a multi-class non-maximum suppression method based on inter-class coverage ratio according to a third embodiment of the present invention;
fig. 31 is a schematic diagram illustrating the result of the rough selection module 200 performing the prediction box screening according to the multi-class non-maximum suppression method based on the inter-class coverage ratio according to the third embodiment of the present invention;
FIG. 32 is a diagram illustrating the results of the Score-NMS module 300 performing the prediction box screening according to the third embodiment of the present invention;
fig. 33 is a diagram illustrating the result of the Area-NMS module 400 performing the prediction box screening according to the third embodiment of the present invention, in a multi-class non-maximum suppression method based on the inter-class coverage ratio;
fig. 34 is a schematic diagram illustrating a distribution of a multi-class non-maximum suppression system according to a fourth embodiment of the present invention;
fig. 35 is a schematic diagram of a network topology of a multi-class non-maximum suppression system according to a fourth embodiment of the present invention.
Detailed Description
In order to make the aforementioned objects, features and advantages of the present invention comprehensible, specific embodiments accompanied with figures are described in detail below, and it is apparent that the described embodiments are a part of the embodiments of the present invention, not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without making creative efforts based on the embodiments of the present invention, shall fall within the protection scope of the present invention.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, but the present invention may be practiced in other ways than those specifically described and will be readily apparent to those of ordinary skill in the art without departing from the spirit of the present invention, and therefore the present invention is not limited to the specific embodiments disclosed below.
Furthermore, reference herein to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one implementation of the invention. The appearances of the phrase "in one embodiment" in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments.
The present invention will be described in detail with reference to the drawings, wherein the cross-sectional views illustrating the structure of the device are not enlarged partially in general scale for convenience of illustration, and the drawings are only exemplary and should not be construed as limiting the scope of the present invention. In addition, the three-dimensional dimensions of length, width and depth should be included in the actual fabrication.
Meanwhile, in the description of the present invention, it should be noted that the terms "upper, lower, inner and outer" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings, and are only for convenience of describing the present invention and simplifying the description, but do not indicate or imply that the referred device or element must have a specific orientation, be constructed in a specific orientation and operate, and thus, cannot be construed as limiting the present invention. Furthermore, the terms first, second, or third are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
The terms "mounted, connected and connected" in the present invention are to be understood broadly, unless otherwise explicitly specified or limited, for example: can be fixedly connected, detachably connected or integrally connected; they may be mechanically, electrically, or directly connected, or indirectly connected through intervening media, or may be interconnected between two elements. The specific meanings of the above terms in the present invention can be understood in a specific case to those of ordinary skill in the art.
Example 1
Referring to fig. 1 to 23, a first embodiment of the present invention provides a multi-class non-maximum suppression method based on inter-class coverage ratio, including:
s1: and setting a threshold value of the confidence of the category, and deleting all the prediction boxes with the confidence of the category smaller than the threshold value in the prediction box set.
S2: and screening the prediction frames by calculating the overlapping degree between the prediction frames.
Specifically, the steps of screening the prediction box are as follows:
(1) Assuming that the input prediction box set is B and the empty set is D;
(2) Sorting the prediction frames in the set B in a descending order according to the category confidence;
(3) Marking the prediction frame with the maximum class confidence as a reference frame and marking as M; marking the rest prediction boxes as comparison boxes and marking as Bo; defining the proportion of the overlapping area of the reference frame and the comparison frame in the comparison frame as COP BoM I.e., degree of overlap, as shown in fig. 2;
specifically, the COP is calculated as follows BoM ：
Wherein S is MBo Is the area of the overlapping region of the reference frame M and the reference frame Bo, S Bo COP (COver percentage) is the proportion of the overlapping area of the two prediction frames in the specified prediction frame;
(4) Set overlap threshold to 0.8 according to COP BoM Judging whether the reference frame M and the comparison frame Bo have an overlapping relation or not according to the relation with a set threshold value; if COP BoM If the difference is larger than the threshold value, the reference frame M and the comparison frame Bo are overlapped, the reference frame M and the comparison frame Bo are removed from the set B, and M is added into the set D; if COP BoM If the value is less than the threshold value, no overlapping relation is indicated, and the reference frame M is added into the set D while being removed from the set B;
(5) Judging whether the set B is empty; if not, circularly screening the prediction frames in the set B according to the steps (2) to (4) and stopping circulation until the set B is empty; if the prediction frame is empty, the prediction frame in the set D is output.
S3: and judging the inclusion relation between the reference frame and the comparison frame by calculating the proportion of the overlapping area of the reference frame and the comparison frame in the reference frame.
Specifically, the step of determining the inclusion relationship between the reference frame and the comparison frame is as follows:
(1) Sorting the prediction frames in the set B in an ascending order according to the areas of the prediction frames;
(2) Marking the prediction frame with the minimum area as a reference frame M1; the rest of the prediction boxes are marked as reference boxes Bo1; defining the proportion of the overlapping area of the reference frame and the comparison frame in the reference frame as COP MBo ；
Specifically, COP was calculated according to the following formula MBo ：
Wherein S is M As reference frame area, S MBo The area of the overlapping region of the reference frame M1 and the comparison frame Bo1 is set;
(3) Setting a proportional threshold according to COP MBo Judging whether an inclusion relationship exists between the reference frame M1 and the comparison frame Bo1 or not according to the relationship with the threshold value; COP of reference frame and comparison frame MBo If the ratio of the overlap area in the reference frame is larger than the set threshold, it is determined that the reference frame includes the reference frame, as shown in fig. 4; COP of reference frame and comparison frame MBo If the comparison frame is smaller than the set threshold, the comparison frame is judged not to contain the reference frame.
S4: if the reference frame is not included in the comparison frame, the reference frame M1 is removed from the prediction frame set B, and the reference frame M1 is added to the empty set D.
S5: if the comparison frame Bo1 contains the reference frame M1, selecting a frame N with the minimum area in the comparison frame Bo1, and screening the reference frame M1 and the minimum frame N by using an inter-class preferred selection strategy.
It should be noted that the inter-class preference strategy is divided into preference between prediction frames of different classes and preference between prediction frames of the same class.
Specifically, it is determined whether the reference frame M1 and the minimum frame N are the same in type, see fig. 5;
(1) If the categories are different, the prediction frames with high category confidence degrees are reserved, and the prediction frames with low category confidence degrees are removed;
(2) And if the types are the same, calculating the difference value of the confidence degrees of the types of the two prediction frames.
(1) If the difference is larger than the set difference threshold, reserving the prediction box with larger category confidence and removing the other prediction box;
(2) and if the difference value is smaller than the set threshold value, merging the two prediction boxes, namely, reserving the prediction box with a larger area and reserving a larger class confidence value at the same time.
It should be noted that, in this embodiment, three times of inter-class preference are completed, fig. 6 is a defect detection diagram before implementation of inter-class preference, four prediction frames (numbered 1-4) with the same class (all "Foreign _ M") are retained in the diagram, and the prediction frames and the class confidence coefficients thereof are sequentially: 1:0.94, 2:0.60, 3:0.58, 4:0.44.
in the first inter-class selection, the prediction frame 1 with the minimum mark area is used as a reference frame, the prediction frame 2 is used as a comparison frame, the comparison frame can be judged to be larger in area than the reference frame through calculation, the confidence coefficient of the reference frame is larger than that of the comparison frame, and meanwhile, the difference value of the confidence coefficient of the reference frame and the confidence coefficient of the comparison frame is smaller than a set threshold value; the two prediction boxes are merged, i.e. the prediction box 2 is retained while the class confidence value of 0.94 for the prediction box 1 is given to the prediction box 2, and the prediction box 1 is removed.
As shown in fig. 7, after the first inter-class preference is completed, the retained prediction frames and the class confidence coefficients thereof are: 2:0.94, 3:0.58, 4:0.44;
in the second inter-class preference selection, the difference between the confidence degrees of the classes of the prediction boxes 2 and 3 is also smaller than the set threshold, so that the confidence degree of the class of the prediction box 3 is kept at 0.94, and the prediction box 2 is removed, as shown in fig. 8;
in the third inter-class selection, the class confidence difference between the prediction frames 3 and 4 is greater than the set threshold, so that the prediction frame 3 is directly reserved, and the prediction frame 2 is removed, as shown in fig. 9;
preferably, defect localization accuracy is improved by inter-class preference.
S6: judging whether the prediction frame set B is empty or not, and if not, continuously screening out the rest prediction frames; otherwise, an empty set is output.
If the prediction box set B is not empty, circulating the prediction boxes in the set B according to the steps S3-S5 until the prediction box set B is empty, and stopping circulating; if the prediction frame is empty, the prediction frame in the set D is output.
In order to verify the technical effect adopted in the method, the present embodiment selects the current multi-class NMS method and adopts the method to perform a comparison test, and compares the test results by a scientific demonstration means to verify the real effect of the method.
The current multi-class NMS method is not sufficient for removing redundancy and positioning defects;
in order to verify that the method has higher accuracy for measuring the overlapping degree and accuracy for locating the defect than the current multi-class NMS (Non-Maximum Suppression) method, the current multi-class NMS method and the method are adopted in the embodiment to respectively perform training and test comparison in the actual defect sample.
The data set comprises 2200 defect samples and 8 class defects, wherein the training set comprises 2000 samples, the test set comprises 200 samples, three defect samples are respectively shown in fig. 10, fig. 11 and fig. 12, wherein fig. 10 is sample 1 containing a forign _ M defect; FIG. 11 is sample 2, containing Gold _ P defects; FIG. 12 is sample 3, containing a Raw _ M defect; the detection results of the current multi-class NMS method in the three defect samples are shown in fig. 13, 14, and 15, respectively, and the detection results of the method in the three defect samples are shown in fig. 16, 17, and 18.
(1) Referring to fig. 13, it can be seen that 4 prediction boxes are reserved after the defect detection of the current multi-class NMS algorithm, the prediction boxes are mutually overlapped, the redundancy removal is not sufficient, and the defect location is not accurate enough; after the detection of the method, as shown in fig. 16, a prediction box is reserved, which indicates that the confidence of the defect type is high, the defect location is more accurate, and other samples all have the effect.
(2) In order to verify that the method can accurately measure the overlapping degree of prediction frames with different areas, taking the current multi-class NMS method and the detection result of the method in a 'Gold _ P' defect sample as an example, a IoU (Intersection over Unit, the ratio of the Intersection area and the Union area of two prediction frames) parameter calculation method and a COP parameter calculation method of the current multi-class NMS method are respectively adopted to measure the overlapping degree of the prediction frames; fig. 20 shows the detection result of the current multi-class NMS method, where the coordinates of prediction boxes 1 and 2 are:
the respective areas and intersection areas of the prediction boxes 1 and 2 can be calculated as follows:
(1) the IoU of prediction blocks 1 and 2 can be obtained by the calculation method of IoU parameter in the current multi-class NMS method (as shown in fig. 19):
it can be seen that IoU of the current prediction frames 1 and 2 is smaller than the set threshold (usually about 0.7), and at this time, the prediction frames 1 and 2 actually overlap with each other, however IoU fails to measure the overlapping degree accurately, resulting in incomplete redundant frame removal.
(2) The COP of the prediction frames 1 and 2 can be obtained by the calculation method of the COP parameters in the method:
it can be found that the COP of prediction boxes 1 and 2 obtained by the COP parameter calculation method in the method can accurately measure the overlapping degree, and the result after the processing by the method is shown in fig. 21.
(3) Taking the detection result of the current multi-class NMS algorithm and the technical scheme of the invention in the Incomplate _ B defect sample as an example; as shown in fig. 22, after detecting a defect sample by the current multi-class NMS method, two different classes of prediction boxes are retained at the same defect location, namely, incomplete _ B:0.96 and Gold _ P:0.11, however, this does not meet the requirement of defect type incompatibility in defect detection.
The non-maximum suppression scheme with incompatible categories constructed by the invention can ensure that only one category prediction box is reserved after detection, namely Incomplite _ B:0.96, as in FIG. 23.
Example 2
Referring to fig. 24 to 25, in the second embodiment of the present invention, unlike embodiment 1, the present embodiment proposes a COP calculation method for calculating an overlap degree BoM The method of (1), comprising:
s1: and taking the prediction frame with the maximum class confidence as a reference frame, and taking the other prediction frames except the prediction frame with the maximum class confidence as comparison frames.
S2: calculating the proportion of the overlapping area of the reference frame and the comparison frame in the reference frame, namely COP BoM 。
The calculation formula is as follows:
example 3
Referring to fig. 26 to 33, in a 3 rd embodiment of the present invention, unlike embodiments 1 and 2, this embodiment proposes a multi-class non-maximum suppression system based on an inter-class coverage ratio, including:
an input module 100 is used for inputting the prediction box set and the empty set into the system.
And a roughing module 200 connected with the input module 100 and used for screening out redundant boxes with very low category confidence.
The Score-NMS module 300 is connected with the roughing module 200 and used for receiving the prediction frame set and the empty set transmitted by the roughing module 200 and screening out the prediction frames with higher overlapping degree; specifically, referring to fig. 28, the Score-NMS module 300 screens out the prediction frames with higher overlap (not less than 0.8) by calculating COP to measure the overlap between the prediction frames based on the non-maximum suppression of the category confidence (Score) and using the category confidence as a standard.
The Area-NMS module 400 is connected with the Score-NMS module 300 and is used for further screening the prediction frames screened by the Score-NMS module 300; specifically, referring to fig. 29, the area-NMS module 400 determines whether there is an inclusion relationship between prediction frames by calculating COP based on non-maximum inhibition of area (area) and using area of the prediction frames as a standard, and preferably, preferentially selects the mutually included prediction frames by an inter-class preferred selection method, so that the remaining prediction frames are better positioned and the class confidence is higher.
To verify the technical effect of the system, the present embodiment tests the roughing module 200, the Score-NMS module 300 and the Area-NMS module 400 respectively, and compares the test results with scientific demonstration means to verify the real effect of the system.
As shown in fig. 30, 1000 prediction boxes are input in total, and the position coordinates and the class confidence are as follows.
(1) The position coordinates of the 1000 prediction boxes are:
(2) the class confidence for the 1000 prediction boxes is:
(1) The rough selection module 200 screens out the prediction frames with the class confidence lower than 0.05 by setting the class confidence threshold, for example, set to 0.05 in this embodiment, and retains 87 prediction frames after the screening, for example, as shown in fig. 31, and the position coordinates and the class confidence of the 87 prediction frames are as follows.
(1) The position coordinates of the 87 prediction boxes are:
(2) the class confidence for the 87 prediction boxes is:
(2) The prediction blocks with larger overlap are screened out by the Score-NMS module 300, and COP is set BoM The threshold, set to 0.8 in this embodiment, is continuously iterated to calculate the COP of the prediction box with the maximum class confidence and the remaining prediction boxes BoM Removal of COP BoM The corresponding prediction blocks with values greater than 0.8, and finally 4 prediction blocks are reserved, as shown in fig. 32.
(3) Then, the prediction boxes with the inclusion relationship are merged by an inter-class preferred selection method in the Area-NMS module, as shown in FIG. 33, and the last retained prediction box has higher class confidence (0.94) and better defect localization.
Example 4
Referring to fig. 34 to 35, a 4 th embodiment of the present invention is different from embodiment 3 in that the present embodiment proposes a multi-class non-maximum suppression system, including:
an input module 100, configured to input the prediction box set and the empty set into the system.
And a roughing module 200 connected with the input module 100 and used for screening out redundant boxes with very low category confidence.
The Area-NMS module 300 is connected with the roughing module 200 and is used for receiving the prediction frame set and the empty set transmitted by the roughing module 200 and screening the prediction frames with inclusion relation; specifically, the Area-NMS module 400 determines whether there is an inclusion relationship between prediction frames by calculating COP based on non-maximum inhibition of Area (Area) and using the Area of the prediction frames as a standard, and preferably, preferentially selects the mutually included prediction frames by an inter-class preferred selection method, so that the remaining prediction frames are better positioned and the class confidence is higher.
The Score-NMS module 400 is connected with the Area-NMS module 300 and is used for receiving the prediction frame set and the empty set transmitted by the Area-NMS module 300 and screening the prediction frames with higher overlapping degree; specifically, the Score-NMS module 400 screens out the prediction frames with higher overlap (not less than 0.8) by calculating COP to measure the overlap degree between the prediction frames based on the non-maximum suppression of the category confidence (Score) and using the category confidence as a standard.
It should be recognized that embodiments of the present invention can be realized and implemented by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer readable memory. The methods may be implemented in a computer program using standard programming techniques, including a non-transitory computer-readable storage medium configured with the computer program, where the storage medium so configured causes a computer to operate in a specific and predefined manner, according to the methods and figures described in the detailed description. Each program may be implemented in a high level procedural or object oriented programming language to communicate with a computer system. However, the program(s) can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language. Furthermore, the program can be run on a programmed application specific integrated circuit for this purpose.
Further, the operations of processes described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes described herein (or variations and/or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) collectively executed on one or more processors, by hardware, or combinations thereof. The computer program includes a plurality of instructions executable by one or more processors.
Further, the method may be implemented in any type of computing platform operatively connected to a suitable interface, including but not limited to a personal computer, mini computer, mainframe, workstation, networked or distributed computing environment, separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, and the like. Aspects of the invention may be embodied in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optically read and/or write storage medium, RAM, ROM, or the like, such that it may be read by a programmable computer, which when read by the storage medium or device, is operative to configure and operate the computer to perform the procedures described herein. Further, the machine-readable code, or portions thereof, may be transmitted over a wired or wireless network. The invention described herein includes these and other different types of non-transitory computer-readable storage media when such media includes instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor. The invention also includes the computer itself when programmed according to the methods and techniques described herein. A computer program can be applied to input data to perform the functions described herein to transform the input data to generate output data that is stored to non-volatile memory. The output information may also be applied to one or more output devices, such as a display. In a preferred embodiment of the invention, the transformed data represents physical and tangible objects, including particular visual depictions of physical and tangible objects produced on a display.
As used in this application, the terms "component," "module," "system," and the like are intended to refer to a computer-related entity, either hardware, firmware, a combination of hardware and software, or software in execution. For example, a component may be, but is not limited to being: a process running on a processor, an object, an executable, a thread of execution, a program, and/or a computer. By way of example, both an application running on a computing device and the computing device can be a component. One or more components can reside within a process and/or thread of execution and a component can be localized on one computer and/or distributed between two or more computers. In addition, these components can execute from various computer readable media having various data structures thereon. The components may communicate by way of local and/or remote processes such as in accordance with a signal having one or more data packets (e.g., data from one component interacting with another component in a local system, distributed system, and/or across a network such as the internet with other systems by way of the signal).
It should be noted that the above-mentioned embodiments are only for illustrating the technical solutions of the present invention and not for limiting, and although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, which should be covered by the claims of the present invention.
Claims (7)
1. A multi-class non-maximum suppression method based on inter-class coverage ratio is characterized in that: comprises the steps of (a) preparing a substrate,
setting a threshold value of the confidence coefficient of the category, and deleting all the prediction frames of which the confidence coefficients of the categories are smaller than the threshold value in the prediction frame set;
screening the prediction frames by calculating the overlapping degree between the prediction frames;
judging the inclusion relation between the reference frame and the comparison frame by calculating the proportion of the overlapping area of the reference frame and the comparison frame in the reference frame; if the proportion is smaller than a set proportion threshold value, judging that the reference frame is not contained in the comparison frame, removing the reference frame from the prediction frame set, and adding the reference frame to an empty set;
otherwise, judging that the comparison frame comprises the reference frame, selecting a frame with the smallest area in the comparison frame, and screening the reference frame and the smallest frame by utilizing an inter-class preferred selection strategy, wherein the inter-class preferred selection strategy is as follows: judging whether the types of the reference frame M1 and the minimum frame N are the same or not, if the types are different, reserving a prediction frame with high type confidence coefficient, and removing the prediction frame with low type confidence coefficient; if the categories are the same, calculating a category confidence difference value of the two prediction frames; if the difference is larger than the set difference threshold, reserving the prediction frame with higher category confidence coefficient, and removing the other prediction frame; otherwise, reserving a prediction box with a larger area and a prediction box with a higher category confidence value;
judging whether the prediction frame set is empty or not, and if not, continuously screening out the rest prediction frames; otherwise, outputting the empty set.
2. The inter-class coverage ratio-based multi-class non-maximum suppression method according to claim 1, wherein: the calculating of the degree of overlap may include,
sorting the prediction frames in the prediction frame set B in a descending order according to the category confidence degrees;
the degree of overlap is as follows:
wherein the COP BoM The ratio S of the overlapping area of the reference frame M and the comparison frame Bo in the comparison frame MBo Is the overlapping area of the reference frame M and the comparison frame Bo, S Bo And the reference frame M is the prediction frame with the maximum category confidence coefficient as the area of the comparison frame, and the comparison frame Bo is the rest prediction frames.
3. The inter-class coverage ratio-based multi-class non-maximum suppression method according to claim 2, wherein: the screening prediction box comprises a filter-based prediction box,
if the COP BoM If the reference frame M and the comparison frame Bo are larger than the overlapping threshold value, removing the reference frame M and the comparison frame Bo from the prediction frame set, and adding the reference frame M to the empty set D;
otherwise, the reference frame M is removed from the prediction frame set B while the reference frame M is added to the empty set D.
4. The inter-class coverage ratio-based multi-class non-maximum suppression method according to claim 3, wherein: the screening prediction box further comprises a filter prediction box,
judging whether the prediction frame set B is empty or not, and if not, continuously screening the prediction frames; otherwise, outputting the prediction frame in the empty set D.
5. The inter-class coverage ratio-based multi-class non-maximum suppression method according to claim 3 or 4, wherein: the overlap threshold value may comprise a threshold value of,
the overlap threshold is equal to 0.8.
6. The inter-class coverage ratio-based multi-class non-maximum suppression method according to any one of claims 1, 2 and 3, wherein: the calculating of the ratio includes calculating a ratio of,
sorting the prediction frames in the prediction frame set B in an ascending order according to the areas of the prediction frames;
calculating the proportion COP of the overlapped area of the reference frame and the comparison frame in the reference frame according to the following formula MBo ：
Wherein S is M The area of the reference frame is marked as the reference frame M1 by the prediction frame with the minimum area; the remaining prediction boxes are labeled as control boxes Bo1.
7. A system applied to the inter-class coverage ratio-based multi-class non-maximum suppression method according to claim 1, wherein: comprises the steps of (a) preparing a mixture of a plurality of raw materials,
an input module (100) for inputting a set of prediction boxes and an empty set to the system;
a roughing module (200) connected with the input module (100) and used for screening out redundant boxes with very low category confidence;
a Score-NMS module (300) connected to the roughing module (200) for screening out the prediction frames with higher overlap;
an Area-NMS module (400) connected to the Score-NMS module (300) for further screening the prediction box left by the Score-NMS module (300).
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011312543.9A CN112419263B (en) | 2020-11-20 | 2020-11-20 | Multi-class non-maximum inhibition method and system based on inter-class coverage ratio |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011312543.9A CN112419263B (en) | 2020-11-20 | 2020-11-20 | Multi-class non-maximum inhibition method and system based on inter-class coverage ratio |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112419263A CN112419263A (en) | 2021-02-26 |
| CN112419263B true CN112419263B (en) | 2023-04-14 |
Family
ID=74778447
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011312543.9A Active CN112419263B (en) | 2020-11-20 | 2020-11-20 | Multi-class non-maximum inhibition method and system based on inter-class coverage ratio |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112419263B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113281707B (en) * | 2021-05-26 | 2022-10-21 | 上海电力大学 | Sound source positioning method based on windowed LASSO under strong noise |
| CN113869307A (en) * | 2021-09-16 | 2021-12-31 | 中国农业科学院农业资源与农业区划研究所 | Cross-category fruit data set automatic labeling system and method |
Family Cites Families (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108268869B (en) * | 2018-02-13 | 2021-11-16 | 北京旷视科技有限公司 | Target detection method, device and system |
| CN108875577A (en) * | 2018-05-11 | 2018-11-23 | 深圳市易成自动驾驶技术有限公司 | Object detection method, device and computer readable storage medium |
| CN108960174A (en) * | 2018-07-12 | 2018-12-07 | 广东工业大学 | A kind of object detection results optimization method and device |
| CN109101897A (en) * | 2018-07-20 | 2018-12-28 | 中国科学院自动化研究所 | Object detection method, system and the relevant device of underwater robot |
| CN109359577B (en) * | 2018-10-08 | 2021-06-29 | 福州大学 | System for detecting number of people under complex background based on machine learning |
| CN111368600B (en) * | 2018-12-26 | 2023-10-31 | 北京眼神智能科技有限公司 | Remote sensing image target detection and identification method and device, readable storage medium and equipment |
| CN110414417B (en) * | 2019-07-25 | 2022-08-12 | 电子科技大学 | Traffic sign board identification method based on multi-level fusion multi-scale prediction |
| CN111310759B (en) * | 2020-02-13 | 2024-03-01 | 中科智云科技有限公司 | Target detection inhibition optimization method and device for dual-mode cooperation |
| CN111738263A (en) * | 2020-08-24 | 2020-10-02 | 北京易真学思教育科技有限公司 | Target detection method and device, electronic equipment and storage medium |
-
2020
- 2020-11-20 CN CN202011312543.9A patent/CN112419263B/en active Active
Non-Patent Citations (1)
| Title |
|---|
| 基于组合式目标检测框架的低漏报率缺陷识别方法;罗鹏等;《高电压技术》;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112419263A (en) | 2021-02-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR102171491B1 (en) | Method for sorting products using deep learning | |
| CN109685199B (en) | Method and apparatus for creating table containing information on pooling type, and test method and test apparatus using the same | |
| CN112419263B (en) | Multi-class non-maximum inhibition method and system based on inter-class coverage ratio | |
| JP7265592B2 (en) | Techniques for Measuring Overlay Between Layers in Multilayer Structures | |
| JP5953842B2 (en) | Image inspection method and inspection area setting method | |
| CN111709948B (en) | Method and device for detecting defects of container | |
| CN108921166A (en) | Medical bill class text detection recognition method and system based on deep neural network | |
| CN105074896A (en) | Pattern-measuring apparatus and semiconductor-measuring system | |
| CN104112284B (en) | The similarity detection method and equipment of a kind of picture | |
| CN110264444A (en) | Damage detecting method and device based on weak segmentation | |
| CN109993700B (en) | Data processing method, device, electronic equipment and computer readable storage medium | |
| CN107578021A (en) | Pedestrian detection method, apparatus and system based on deep learning network | |
| TWI715051B (en) | Machine learning method and automatic optical inspection device using the method thereof | |
| CN110910445B (en) | Object size detection method, device, detection equipment and storage medium | |
| KR20210024767A (en) | Method for non-destructive inspection based on image and Computer-readable storage medium | |
| CN113222913A (en) | Circuit board defect detection positioning method and device and storage medium | |
| CN112800870B (en) | Non-maximum inhibition method and system based on interlayer coverage ratio | |
| CN110375736A (en) | Paths planning method, system, equipment and the readable storage medium storing program for executing of smart machine | |
| CN110489830A (en) | The design detection method and computer storage medium of pcb board card | |
| CN117372424B (en) | Defect detection method, device, equipment and storage medium | |
| TWI849144B (en) | Inspection device, inspection method, and recording medium recording inspection program, as well as learning device, learning method, and recording medium recording learning program | |
| CN117808852A (en) | Construction period digital monitoring method and system based on mapping point cloud | |
| KR102075872B1 (en) | Method for non-destructive inspection based on image and apparatus thereof | |
| US8306311B2 (en) | Method and system for automated ball-grid array void quantification | |
| KR20220111214A (en) | Method, apparatus and computer program for inspection of product based on artificial intelligence |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |