CN113256581A - Automatic defect sample labeling method and system based on visual attention modeling fusion - Google Patents
Automatic defect sample labeling method and system based on visual attention modeling fusion Download PDFInfo
- Publication number
- CN113256581A CN113256581A CN202110555658.9A CN202110555658A CN113256581A CN 113256581 A CN113256581 A CN 113256581A CN 202110555658 A CN202110555658 A CN 202110555658A CN 113256581 A CN113256581 A CN 113256581A
- Authority
- CN
- China
- Prior art keywords
- defect
- image
- saliency map
- fusion
- labeling
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000007547 defect Effects 0.000 title claims abstract description 116
- 238000002372 labelling Methods 0.000 title claims abstract description 68
- 230000004927 fusion Effects 0.000 title claims abstract description 60
- 230000000007 visual effect Effects 0.000 title claims abstract description 40
- 238000000034 method Methods 0.000 claims abstract description 79
- 230000011218 segmentation Effects 0.000 claims abstract description 64
- 239000003550 marker Substances 0.000 claims abstract description 13
- 230000003993 interaction Effects 0.000 claims abstract description 10
- 238000003860 storage Methods 0.000 claims description 15
- 239000013598 vector Substances 0.000 claims description 12
- 238000004891 communication Methods 0.000 claims description 9
- 238000000354 decomposition reaction Methods 0.000 claims description 8
- 238000000605 extraction Methods 0.000 claims description 6
- 238000004422 calculation algorithm Methods 0.000 claims description 5
- 239000000126 substance Substances 0.000 claims description 5
- 238000001514 detection method Methods 0.000 abstract description 18
- 238000004590 computer program Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 230000008569 process Effects 0.000 description 7
- 238000004364 calculation method Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 5
- 238000012545 processing Methods 0.000 description 5
- 238000012549 training Methods 0.000 description 5
- 238000010801 machine learning Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 238000011160 research Methods 0.000 description 4
- 230000009466 transformation Effects 0.000 description 3
- 238000004519 manufacturing process Methods 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000010008 shearing Methods 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 238000013519 translation Methods 0.000 description 2
- 230000003190 augmentative effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000009776 industrial production Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000007306 turnover Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
-
- G06T5/94—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
- G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Data Mining & Analysis (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Quality & Reliability (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention belongs to the technical field of part surface damage detection, and particularly relates to a defect sample automatic labeling method and system based on visual attention modeling fusion, aiming at solving the problem that the automatic labeling of the surface defects of precision part samples cannot be realized in the prior art; respectively calculating a saliency map based on visual attention modeling fusion for each image region, and segmenting the saliency regions in the saliency map by using a marker array segmentation method to serve as defect candidate regions; extracting the characteristics of the defect candidate area, clustering to obtain a plurality of defect samples of different categories, labeling each defect sample according to a small number of labels or known label definition rules, and manually labeling unknown categories through human-computer interaction; the invention can greatly reduce the workload of manual marking, and has high marking efficiency and good accuracy.
Description
Technical Field
The invention belongs to the technical field of part surface damage detection, and particularly relates to a method and a system for automatically labeling a defect sample based on visual attention modeling fusion.
Background
The defect detection of precise devices is an important link of device production, and because the defects are not in fixed forms, have low contrast with the background, are difficult to find slight defects and the like, the defects are detected by artificial naked eyes for a long time in the actual industrial production. As a key technology for image processing, and for guaranteeing the quality of automated production, research on appearance detection methods has been started as early as the 80 s in the 20 th century, and has attracted a great deal of research by many scholars at home and abroad. The vision-based defect sample detection techniques can be divided into two categories: the defect detection method comprises the steps of carrying out feature extraction manually and then carrying out classification according to the features; the defect detection method based on learning is a method for automatically learning appearance defect characteristics by a machine learning method and carrying out detection and classification; due to the fact that the appearance defect forms are different and the contrast is low, the traditional defect detection method is difficult to achieve high detection precision.
While an important assumption that needs to be met for successful application of machine learning is: the number of training samples is large enough; however, in practical applications, it is difficult to collect a sufficient amount of labeled data samples on many occasions, and for this reason, a limited sample learning method is proposed in the prior art, which is mainly divided into two types, one type is based on training data to solve the sample limited problem, and the other type is based on a model to solve the sample limited problem, wherein the limited sample learning method based on data is to expand limited sample data through the existing priori knowledge to meet the requirements of model training on the number of samples, such as translation, inversion, shearing, scale transformation, mirror image, and the like.
Therefore, the problem of sample shortage can be fundamentally solved by collecting a large number of real samples and then marking typical defect targets, and at present, no related method for automatically marking defect samples of precision devices exists.
Disclosure of Invention
In order to solve the problems, namely solving the problem that the automatic marking of the surface defects of the precision part sample cannot be realized in the prior art, the invention provides a method and a system for automatically marking the defect sample based on visual attention modeling fusion.
The invention provides a first aspect of a method for automatically labeling a defect sample based on visual attention modeling fusion, which comprises the following steps: step S100, performing superpixel segmentation on an input original sample image, and automatically segmenting the image into a plurality of image areas with preset characteristics;
step S200, respectively calculating a saliency map based on visual attention modeling fusion for each image region, and segmenting the saliency region in the saliency map by using a marker array segmentation method to serve as a defect candidate region;
and step S300, extracting the characteristics of the defect candidate region, clustering to obtain a plurality of defect samples of different categories, labeling each defect sample according to a small number of labels or known label definition rules, and manually labeling unknown categories through man-machine interaction.
In some preferred embodiments, the superpixel segmentation in step S100 is a SLIC superpixel segmentation algorithm;
the image area is a super pixel; the superpixel is composed of adjacent pixel points with similar characteristics.
In some preferred embodiments, the method for calculating the saliency map in step S200 is specifically: calculating a first saliency map based on background contrast for a superpixel region obtained after each superpixel segmentation;
calculating a second saliency map based on sparse low-rank decomposition for a super-pixel region obtained after each super-pixel is segmented;
and performing fusion reconstruction on the first saliency map and the second saliency map by using a saliency difference modeling method to obtain a final saliency map.
In some preferred embodiments, the fused saliency value corresponding to the saliency map is Mi，
Wherein the content of the first and second substances,
is the saliency value corresponding to said first saliency map,
the corresponding significant value of the second significant map is obtained;
In some preferred embodiments, the method for obtaining the defect candidate region by segmentation specifically includes: sorting the obtained N fusion significant values from small to large, and if the corresponding fusion significant value is greater than a preset significant threshold, determining the marker array Seq corresponding to the super pixel regionj1, otherwise Seqj0 to obtain a tag array Seq; the image is mapped into a binary image by using the mark array, wherein the area with the value of 1 is the defect candidate area.
In some preferred embodiments, the clustering in step S300 specifically includes:
performing image enhancement on the defect candidate area image by utilizing histogram statistics;
performing feature extraction on the defect candidate region image after image enhancement;
normalizing the extracted features to construct feature vectors;
and clustering the feature vectors to obtain a plurality of different categories.
The invention provides a defect sample automatic labeling system based on visual attention modeling fusion, which comprises an image super-pixel segmentation module, a defect candidate region segmentation module and a candidate region automatic labeling module, wherein the defect candidate region segmentation module is in communication connection with the image super-pixel segmentation module;
the image super-pixel segmentation module is configured to perform super-pixel segmentation on an input original sample image and automatically segment the image into a plurality of image areas with preset characteristics;
the defect candidate region segmentation module is configured to respectively calculate a saliency map based on visual attention modeling fusion for each image region, and segment the saliency region in the saliency map by using a marker array segmentation method to serve as a defect candidate region;
the automatic candidate region labeling module is configured to extract the characteristics of the defect candidate regions, perform clustering to obtain a plurality of different classes, label the class of samples according to a small number of labels or known label definition rules, and perform manual labeling on unknown classes through man-machine interaction.
In some preferred embodiments, the method for calculating the saliency map specifically includes: calculating a first saliency map based on background contrast for a superpixel region obtained after each superpixel segmentation;
calculating a second saliency map based on sparse low-rank decomposition for a super-pixel region obtained after each super-pixel is segmented;
and performing fusion reconstruction on the first saliency map and the second saliency map by using a saliency difference modeling method to obtain a final saliency map.
A third aspect of the present invention provides an electronic device comprising: at least one processor; and
a memory communicatively coupled to at least one of the processors; wherein the content of the first and second substances,
the memory stores instructions executable by the processor for execution by the processor to implement the method for automatic labeling of defect samples based on visual attention modeling fusion as set forth in any one of the above.
In a fourth aspect of the present invention, a computer-readable storage medium is provided, where computer instructions are stored in the computer-readable storage medium for execution by the computer to implement any one of the above methods for automatically labeling a defect sample based on visual attention modeling fusion.
The automatic labeling method and system for the defect sample based on visual attention modeling fusion can greatly reduce the workload of manual labeling, provide enough samples for a machine learning model, play an important role in industrial application and provide effective data support for research of an appearance detection positioning method.
Drawings
Other features, objects and advantages of the present application will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, made with reference to the accompanying drawings in which:
FIG. 1 is a schematic diagram illustrating the steps of an embodiment of a method for automatically labeling a defect sample based on visual attention modeling fusion according to the present invention;
fig. 2 is a specific flowchart of the method for automatically labeling a defect sample based on visual attention modeling fusion in the present invention.
FIG. 3 is a block diagram of a computer system of a server for implementing embodiments of the method, system, and apparatus of the present application.
Detailed Description
In order to make the embodiments, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be described clearly and completely with reference to the accompanying drawings, and it is apparent that the described embodiments are some, but not all embodiments of the present invention. It should be understood by those skilled in the art that these embodiments are only for explaining the technical principle of the present invention, and are not intended to limit the scope of the present invention.
The invention provides a first aspect of a method for automatically labeling a defect sample based on visual attention modeling fusion, which comprises the following steps: step S100, performing superpixel segmentation on an input original sample image, and automatically segmenting the image into a plurality of image areas with preset characteristics; step S200, respectively calculating a saliency map based on visual attention modeling fusion for each image region, and segmenting the saliency region in the saliency map by using a marker array segmentation method to serve as a defect candidate region; and step S300, extracting the characteristics of the defect candidate region, clustering to obtain a plurality of defect samples of different categories, labeling each defect sample according to a small number of labels or known label definition rules, and manually labeling unknown categories through man-machine interaction.
The invention provides a system for automatically labeling a defect sample based on visual attention modeling fusion, which comprises an image super-pixel segmentation module, a defect candidate region segmentation module and a candidate region automatic labeling module, wherein the defect candidate region segmentation module is in communication connection with the image super-pixel segmentation module; the image super-pixel segmentation module is configured to perform super-pixel segmentation on an input original sample image and automatically segment the image into a plurality of image areas with preset characteristics; the defect candidate region segmentation module is configured to respectively calculate a saliency map based on visual attention modeling fusion for each image region, and segment the saliency region in the saliency map by using a marker array segmentation method to serve as a defect candidate region; the automatic candidate region labeling module is configured to extract the characteristics of the defect candidate regions, perform clustering to obtain a plurality of different classes, label the class of samples according to a small number of labels or known label definition rules, and perform manual labeling on unknown classes through human-computer interaction.
By the automatic defect sample labeling method and system based on visual attention modeling fusion, the problems that the efficiency of a traditional method for manually labeling defect samples of precision devices is low, labeling standards cannot be unified due to human subjective factors, and labeling workload is large can be effectively solved; the method comprises the following steps of performing superpixel segmentation on an input original sample image, automatically segmenting the image into a plurality of image regions with certain consistency, calculating significance detection based on background contrast and significance detection based on sparse low-rank decomposition for each image region respectively to obtain two significance maps, performing fusion reconstruction on the two significance maps by using a significance difference modeling method, and segmenting the significance regions by using a mark segmentation method; thirdly, extracting the characteristics of the significant region, clustering to obtain N different classes, and finally labeling the samples according to a small number of labels or known label definition rules; the method is applied to automatic labeling of the sample in the defect detection of the precision device, and has high labeling efficiency and good accuracy.
The invention is further described with reference to the following detailed description of embodiments with reference to the accompanying drawings.
Referring to fig. 1 and 2, a first aspect of the present invention provides a method for automatically labeling a defect sample based on visual attention modeling fusion, the method comprising the following steps: step S100, performing superpixel segmentation on an input original sample image, and automatically segmenting the image into a plurality of image areas with preset characteristics, wherein the image areas with the preset characteristics refer to certain consistent image areas in the embodiment; wherein, the superpixel is divided into SLIC superpixel division algorithm; the image area is a super pixel, and the super pixel is composed of adjacent pixel points with similar characteristics; for an input image I, an image is segmented into a plurality of superpixels P by using a SLIC superpixel segmentation algorithmi(ii) a Each super pixel PiThe image acquisition system is composed of adjacent pixel points with similar characteristics, so that the integrity of a subsequent detection obvious object is ensured, and the number of image blocks can be reduced.
Step S200, respectively calculating a saliency map based on visual attention modeling fusion for each image region, and segmenting the saliency region in the saliency map by using a marker array segmentation method to serve as a defect candidate region; specifically, the method comprises taking all the super-pixels P connected with the image framejForming an image background set B, and calculating a super pixel P outside the backgroundiSignificance M to background set BB iThe calculation formula is as follows:
wherein d (c)i,cj) Is a super pixel PiAnd PjEuclidean distance in color space, d (l)i,lj) Is a super pixel PiAnd PjEuclidean distance in location space; sigmalFor controlling the weight of the spatial distance to the saliency value calculation. At the same time, for each super-pixel P obtained in the previous stepiComputing a significance value M based on sparse low rank decompositionL i(ii) a And for the two saliency maps obtained in the steps, performing fusion reconstruction on the two saliency maps by using a saliency difference modeling method.
The method for calculating the saliency map specifically comprises the following steps (namely the method for fusion reconstruction specifically comprises the following steps of): for each super pixel PiCalculating a first saliency map based on background contrast from a super-pixel region obtained after segmentation; calculating a second saliency map based on sparse low-rank decomposition for a super-pixel region obtained after each super-pixel is segmented; and performing fusion reconstruction on the first saliency map and the second saliency map by using a saliency difference modeling method to obtain a final saliency map.
The fusion significance value corresponding to the significance map is Mi，
Wherein the content of the first and second substances,
is the saliency value corresponding to the first saliency map,
the corresponding significant value of the second significant map;
And (3) segmenting the salient regions of the final salient image obtained by fusion reconstruction by using a marker array segmentation method, wherein each salient region is a defect candidate region.
The method for acquiring the defect candidate area by segmentation specifically comprises the following steps: sorting the obtained N fusion significant values from small to large, and if the corresponding fusion significant value is greater than a preset significant threshold, determining the marker array Seq corresponding to the super pixel regionj1, otherwise Seqj0 to obtain a tag array Seq; and mapping the image into a binary image by using the mark array, wherein the area with the value of 1 is a defect candidate area, and the area with the value of 0 is a normal surface area.
And step S300, extracting the characteristics of the defect candidate region, clustering to obtain a plurality of defect samples of different categories, labeling each defect sample according to a small number of labels or known label definition rules, and manually labeling unknown categories through man-machine interaction.
Wherein, clustering specifically comprises: performing image enhancement on the defect candidate area image by utilizing histogram statistics; extracting the features of the defect candidate region image after image enhancement, wherein five features including color, gray scale, area, size and texture are taken in the embodiment; normalizing the extracted features to construct a feature vector xi；xi＝(xi1，xi2，xi3，xi4，xi5) And i represents the ith super pixel.
Clustering the feature vectors by a DBSCAN clustering method to obtain m clusters (C) of different classes1，C2，……，CmObtained by clusteringDefect samples in the same category have similar characteristic parameters;
for a known defect sample set D with a label, calculating a feature vector y of each samplej＝(yj1，yj2，yj3，yj4，yj5) And j represents the jth known sample labeled Labelj；
Calculating the known sample yjAnd cluster C obtained by clusteringmIf y isj∈CmThen C ismAll sample tags within a sample are defined as Labelj；
Cluster C obtained by clusteringk，Ck+1，...，Ck+NIf there is no sample in the known sample set D, it is labeled as Unknown _0, Unknown _1, and Unknown _ N;
for the categories, the categories can be named according to a preset naming rule, or after manual naming is completed through human-computer interaction, new labels are uniformly modified.
The invention provides a defect sample automatic labeling system based on visual attention modeling fusion, which comprises an image super-pixel segmentation module, a defect candidate region segmentation module and a candidate region automatic labeling module, wherein the defect candidate region segmentation module is in communication connection with the image super-pixel segmentation module, and the candidate region automatic labeling module is in communication connection with the defect candidate region segmentation module.
The image super-pixel segmentation module is used for carrying out super-pixel segmentation on an input original sample image and automatically segmenting the image into a plurality of image areas with certain consistency.
The defect candidate region segmentation module is used for respectively calculating a saliency map based on visual attention modeling fusion for each image region, and segmenting the saliency regions in the saliency map by using a marker array segmentation method, wherein each saliency region is a defect candidate region.
And the automatic candidate region labeling module is used for extracting the characteristics of the defect candidate regions, clustering the characteristics to obtain a plurality of different classes, labeling the class of samples according to a small number of labels or known label definition rules, and manually labeling unknown classes through man-machine interaction.
Preferably, the image super-pixel segmentation module divides adjacent pixel points with similar characteristics in the input sample image into the same image area through SLIC super-pixel segmentation, so that the integrity of the subsequent detection of the significant object is ensured, and the number of image blocks can be reduced.
Preferably, the calculation method of the saliency map is specifically: calculating a first saliency map based on background contrast for a superpixel region obtained after each superpixel segmentation; calculating a second saliency map based on sparse low-rank decomposition for a super-pixel region obtained after each super-pixel is segmented; and performing fusion reconstruction on the first saliency map and the second saliency map by using a saliency difference modeling method to obtain a final saliency map.
Further, the specific calculation process of the defect candidate region segmentation module for calculating the saliency map based on visual attention modeling fusion for each image region respectively includes:
the fusion significance value corresponding to the significance map is Mi，
Wherein the content of the first and second substances,
is the saliency value corresponding to the first saliency map,
the corresponding significant value of the second significant map;
Marked array segmentation method is utilized in defect candidate region segmentation module to segment salient regionsThe segmentation to obtain all defect candidate regions specifically comprises: all super-pixel image blocks P in the imageiThe fusion significant value M obtained by the calculationiSorting from small to large, if the significance value of the image block is greater than the set significance threshold value THThen the mark array Seq corresponding to the image blockj1 is ═ 1; otherwise Seqj0. Thereby obtaining a tag array Seq; and mapping the image into a binary image by using the mark array, wherein the area with the value of 1 is a defect candidate area, and the area with the value of 0 is a normal surface area.
Further, the automatic candidate region labeling module specifically includes: performing feature extraction on the defect candidate region segmented by the previous module, wherein the feature extraction comprises color, gray scale, contour, entropy and the like, and obtaining a feature vector of the candidate region; clustering the feature vectors to obtain a plurality of different classes; automatically labeling the defect candidate areas according to labeled sample labels or known label definition rules; for classes different from known tags, the default definition is a new class, which can be named manually or be updated by waiting for a definition rule.
The clustering method specifically comprises the following steps: performing image enhancement on the defect candidate area image by utilizing histogram statistics; and performing feature extraction on the processed candidate region image, including but not limited to: color, gray scale, area, size, texture, etc.; normalizing the extracted features to construct feature vectors; and clustering the feature vectors to obtain a plurality of different categories.
The labeling method specifically comprises the following steps: the candidate defect regions, namely the extracted defect samples can be divided into a plurality of different categories through a clustering algorithm, and the defect samples in the same category have similar characteristic parameters;
calculating a feature vector of a known defect sample image with a label;
classifying the known defect samples with labels according to the clustering rule;
labeling all candidate samples classified as the same category as the known sample as the labels carried by the known sample;
after clustering, the classes which have no same characteristic parameters with the known samples are labeled Unknown _1 and Unknown _ 2;
for the clustering categories without known samples, the categories can be named according to a preset naming rule, or manually named through human-computer interaction, and then uniformly modified into new labels.
In practical application, the defect detection method based on deep learning is difficult to collect enough labeled data samples for model training. The traditional data capacity expansion method is to expand limited sample data through the prior knowledge so as to meet the requirements of model training on the number of samples, such as translation, turnover, shearing, scale transformation, mirror image and the like. The method is characterized in that each original sample is augmented by an artificially defined conversion rule, and the method only performs combined transformation on the original image and does not increase the diversity of the sample, so that more new features cannot be provided. The invention provides a precision device defect sample automatic labeling method based on visual attention modeling fusion, which is used for rapidly and effectively automatically labeling the defects on the surface of a precision device sample by acquiring a large number of real samples and labeling typical defect targets, so that the problem of sample shortage is fundamentally solved. The automatic sample labeling can greatly reduce the workload of manual labeling and provide enough samples for the machine learning model. The method not only can play an important role in industrial application, but also can provide effective data support for the research of an appearance detection positioning method.
An electronic apparatus according to a third embodiment of the present invention includes: at least one processor; and a memory communicatively coupled to at least one of the processors; wherein the memory stores instructions executable by the processor for execution by the processor to implement the method for automatic labeling of defect samples based on visual attention modeling fusion as set forth in any one of the above.
A computer-readable storage medium of a fourth embodiment of the present invention stores computer instructions for execution by the computer to implement the method for automatically labeling a defect sample based on visual attention modeling fusion as described in any one of the above.
It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes and related descriptions of the storage system and the processing system described above may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
Reference is now made to FIG. 3, which illustrates a block diagram of a computer system of a server for implementing embodiments of the method, system, and apparatus of the present application. The server shown in fig. 3 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.
As shown in fig. 3, the computer system includes a Central Processing Unit (CPU)601, which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 602 or a program loaded from a storage section 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data necessary for system operation are also stored. The CPU 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An Input/Output (I/O) interface 605 is also connected to bus 604.
The following components are connected to the I/O interface 605: an input portion 606 including a keyboard, a mouse, and the like; an output section 607 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, a speaker, and the like; a storage section 608 including a hard disk and the like; and a communication section 609 including a Network interface card such as a LAN (Local Area Network) card, a modem, or the like. The communication section 609 performs communication processing via a network such as the internet. The driver 610 is also connected to the I/O interface 605 as needed. A removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 610 as necessary, so that a computer program read out therefrom is mounted in the storage section 608 as necessary.
In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 609, and/or installed from the removable medium 611. The computer program performs the above-described functions defined in the method of the present application when executed by a Central Processing Unit (CPU) 601. It should be noted that the computer readable medium mentioned above in the present application may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, or device. In this application, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present application may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The terms "first," "second," and the like are used for distinguishing between similar elements and not necessarily for describing or implying a particular order or sequence.
It should be noted that in the description of the present invention, the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. indicating the directions or positional relationships are based on the directions or positional relationships shown in the drawings, which are only for convenience of description, and do not indicate or imply that the system or the element must have a specific orientation, be constructed and operated in a specific orientation, and thus, should not be construed as limiting the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
Furthermore, it should be noted that, in the description of the present invention, unless otherwise explicitly specified or limited, the terms "mounted," "connected," and "connected" are to be construed broadly, and may be, for example, fixedly connected, detachably connected, or integrally connected; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meanings of the above terms in the present invention can be understood by those skilled in the art according to specific situations.
The terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, article, or apparatus/system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, article, or apparatus/system.
So far, the technical solutions of the present invention have been described in connection with the preferred embodiments shown in the drawings, but it is easily understood by those skilled in the art that the scope of the present invention is obviously not limited to these specific embodiments. Equivalent changes or substitutions of related technical features can be made by those skilled in the art without departing from the principle of the invention, and the technical scheme after the changes or substitutions can fall into the protection scope of the invention.
Claims (10)
1. A method for automatically labeling a defect sample based on visual attention modeling fusion is characterized by comprising the following steps:
step S100, performing superpixel segmentation on an input original sample image, and automatically segmenting the image into a plurality of image areas with preset characteristics;
step S200, respectively calculating a saliency map based on visual attention modeling fusion for each image region, and segmenting the saliency region in the saliency map by using a marker array segmentation method to serve as a defect candidate region;
and step S300, extracting the characteristics of the defect candidate region, clustering to obtain a plurality of defect samples of different categories, labeling each defect sample according to a small number of labels or known label definition rules, and manually labeling unknown categories through human-computer interaction.
2. The method for automatically labeling defect samples based on visual attention modeling fusion as claimed in claim 1, wherein said superpixel segmentation in step S100 is SLIC superpixel segmentation algorithm;
the image area is a super pixel; the superpixel is composed of adjacent pixel points with similar characteristics.
3. The method for automatically labeling defect samples based on visual attention modeling fusion according to claim 1, wherein the method for calculating the saliency map in step S200 specifically comprises:
calculating a first saliency map based on background contrast for a superpixel region obtained after each superpixel segmentation;
calculating a second saliency map based on sparse low-rank decomposition for a super-pixel region obtained after each super-pixel is segmented;
and performing fusion reconstruction on the first saliency map and the second saliency map by using a saliency difference modeling method to obtain a final saliency map.
4. The method for automatically labeling defect samples based on visual attention modeling fusion as claimed in claim 3, wherein the fusion saliency value corresponding to the saliency map is
Wherein the content of the first and second substances,
is the saliency value corresponding to said first saliency map,
the corresponding significant value of the second significant map is obtained;
5. The method for automatically labeling the defect sample based on the visual attention modeling fusion as claimed in claim 1, wherein the method for segmenting and acquiring the defect candidate region specifically comprises:
sorting the obtained N fusion significant values from small to large, and if the corresponding fusion significant value is greater than a preset significant threshold, determining the marker array Seq corresponding to the super pixel regionj1, otherwise Seqj0 to obtain a tag array Seq; the image is mapped into a binary image by using the mark array, wherein the area with the value of 1 is the defect candidate area.
6. The method for automatically labeling defect samples based on visual attention modeling fusion according to claim 1, wherein the clustering in step S300 specifically comprises:
performing image enhancement on the defect candidate area image by utilizing histogram statistics;
performing feature extraction on the defect candidate region image after image enhancement;
normalizing the extracted features to construct feature vectors;
and clustering the feature vectors to obtain a plurality of different categories.
7. A defect sample automatic labeling system based on visual attention modeling fusion is characterized by comprising an image super-pixel segmentation module, a defect candidate region segmentation module and a candidate region automatic labeling module, wherein the defect candidate region segmentation module is in communication connection with the image super-pixel segmentation module;
the image super-pixel segmentation module is configured to perform super-pixel segmentation on an input original sample image and automatically segment the image into a plurality of image areas with preset characteristics;
the defect candidate region segmentation module is configured to respectively calculate a saliency map based on visual attention modeling fusion for each image region, and segment the saliency region in the saliency map by using a marker array segmentation method to serve as a defect candidate region;
the automatic labeling module of the candidate area is configured to extract the characteristics of the defect candidate area, perform clustering to obtain a plurality of different types, and label the type of sample according to a small number of labels or known label definition rules.
8. The method for automatically labeling the defect sample based on the visual attention modeling fusion as claimed in claim 6, wherein the method for calculating the saliency map is specifically as follows:
calculating a first saliency map based on background contrast for a superpixel region obtained after each superpixel segmentation;
calculating a second saliency map based on sparse low-rank decomposition for a super-pixel region obtained after each super-pixel is segmented;
and performing fusion reconstruction on the first saliency map and the second saliency map by using a saliency difference modeling method to obtain a final saliency map.
9. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to at least one of the processors; wherein the memory stores instructions executable by the processor for implementing the method for automatic labeling of defect samples based on visual attention modeling fusion of any of claims 1-6.
10. A computer-readable storage medium storing computer instructions for execution by the computer to implement the method for automatic labeling of defect samples based on visual attention modeling fusion according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110555658.9A CN113256581B (en) | 2021-05-21 | 2021-05-21 | Automatic defect sample labeling method and system based on visual attention modeling fusion |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110555658.9A CN113256581B (en) | 2021-05-21 | 2021-05-21 | Automatic defect sample labeling method and system based on visual attention modeling fusion |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113256581A true CN113256581A (en) | 2021-08-13 |
| CN113256581B CN113256581B (en) | 2022-09-02 |
Family
ID=77183465
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110555658.9A Active CN113256581B (en) | 2021-05-21 | 2021-05-21 | Automatic defect sample labeling method and system based on visual attention modeling fusion |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113256581B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140254922A1 (en) * | 2013-03-11 | 2014-09-11 | Microsoft Corporation | Salient Object Detection in Images via Saliency |
| CN109522908A (en) * | 2018-11-16 | 2019-03-26 | 董静 | Image significance detection method based on area label fusion |
| CN110717896A (en) * | 2019-09-24 | 2020-01-21 | 东北大学 | Plate strip steel surface defect detection method based on saliency label information propagation model |
| WO2020107717A1 (en) * | 2018-11-30 | 2020-06-04 | 长沙理工大学 | Visual saliency region detection method and apparatus |
-
2021
- 2021-05-21 CN CN202110555658.9A patent/CN113256581B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140254922A1 (en) * | 2013-03-11 | 2014-09-11 | Microsoft Corporation | Salient Object Detection in Images via Saliency |
| CN109522908A (en) * | 2018-11-16 | 2019-03-26 | 董静 | Image significance detection method based on area label fusion |
| WO2020107717A1 (en) * | 2018-11-30 | 2020-06-04 | 长沙理工大学 | Visual saliency region detection method and apparatus |
| CN110717896A (en) * | 2019-09-24 | 2020-01-21 | 东北大学 | Plate strip steel surface defect detection method based on saliency label information propagation model |
Non-Patent Citations (2)
| Title |
|---|
| RUI HUANG 等: "Saliency and co-saliency detection by low-rank multiscale fusion", 《2015 IEEE INTERNATIONAL CONFERENCE ON MULTIMEDIA AND EXPO》 * |
| 王东 等: "基于机器视觉的微小零件表面缺陷检测研究", 《应用科技》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113256581B (en) | 2022-09-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Xu et al. | Voxel-based representation of 3D point clouds: Methods, applications, and its potential use in the construction industry | |
| WO2023083059A1 (en) | Road surface defect detection method and apparatus, and electronic device and readable storage medium | |
| CN111460927B (en) | Method for extracting structured information of house property evidence image | |
| WO2023202381A1 (en) | Three-dimensional model segmentation method and apparatus, computing device, and readable storage medium | |
| US10685443B2 (en) | Cloud detection using images | |
| CN112989995B (en) | Text detection method and device and electronic equipment | |
| CN112070135A (en) | Power equipment image detection method and device, power equipment and storage medium | |
| CN113052295B (en) | Training method of neural network, object detection method, device and equipment | |
| CN115376130A (en) | Method, device, equipment and medium for generating electric power construction site identification model | |
| CN114120141A (en) | All-weather remote sensing monitoring automatic analysis method and system thereof | |
| CN116486296A (en) | Target detection method, device and computer readable storage medium | |
| CN115311469A (en) | Image labeling method, training method, image processing method and electronic equipment | |
| CN117572457B (en) | Cross-scene multispectral point cloud classification method based on pseudo tag learning | |
| CN115019163A (en) | City factor identification method based on multi-source big data | |
| CN117437647B (en) | Oracle character detection method based on deep learning and computer vision | |
| CN113033386B (en) | High-resolution remote sensing image-based transmission line channel hidden danger identification method and system | |
| Sun et al. | Fast location and recognition of green apple based on RGB-D image | |
| CN113256581B (en) | Automatic defect sample labeling method and system based on visual attention modeling fusion | |
| CN112785548A (en) | Pavement crack detection method based on vehicle-mounted laser point cloud | |
| CN117146841A (en) | Semantic map construction method based on combination of laser radar and binocular vision | |
| WO2023220913A1 (en) | Cell image processing method, electronic device and storage medium | |
| Luo et al. | RBD-Net: robust breakage detection algorithm for industrial leather | |
| CN115937492A (en) | Transformer equipment infrared image identification method based on feature identification | |
| Jia et al. | Polar-Net: Green fruit instance segmentation in complex orchard environment | |
| Zhang et al. | A YOLOv3-Based Industrial Instrument Classification and Reading Recognition Method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |