CN113554592A - Image difference detection method and device - Google Patents

Image difference detection method and device Download PDF

Info

Publication number
CN113554592A
CN113554592A CN202110646644.8A CN202110646644A CN113554592A CN 113554592 A CN113554592 A CN 113554592A CN 202110646644 A CN202110646644 A CN 202110646644A CN 113554592 A CN113554592 A CN 113554592A
Authority
CN
China
Prior art keywords
image
difference detection
detection model
images
model
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110646644.8A
Other languages
Chinese (zh)
Inventor
陈南博
赵军明
张伟
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Suzhou Xinliancheng Software Co ltd
Original Assignee
Suzhou Xinliancheng Software Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Suzhou Xinliancheng Software Co ltd filed Critical Suzhou Xinliancheng Software Co ltd
Priority to CN202110646644.8A priority Critical patent/CN113554592A/en
Publication of CN113554592A publication Critical patent/CN113554592A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/0002Inspection of images, e.g. flaw detection
    • G06T7/0004Industrial image inspection
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/08Learning methods
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00Image enhancement or restoration
    • G06T5/90Dynamic range modification of images or parts thereof
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20081Training; Learning
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20084Artificial neural networks [ANN]
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/30Subject of image; Context of image processing
    • G06T2207/30108Industrial image inspection
    • G06T2207/30148Semiconductor; IC; Wafer
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/30Subject of image; Context of image processing
    • G06T2207/30204Marker

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Biophysics (AREA)
  • Evolutionary Computation (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • Quality & Reliability (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses an image difference detection method and device. The method is applied to an integrated circuit and comprises the following steps: preprocessing the collected sample image; constructing a difference detection model based on a neural network; performing model training on the difference detection model according to the preprocessed sample image to obtain a target difference detection model; and detecting images to be detected according to the target difference detection model, determining difference information between the images to be detected, and determining abnormal image data according to the difference information. According to the technical scheme of the embodiment, the image difference is detected through the target difference detection model constructed by the neural network, so that the efficiency and the accuracy of detecting the difference of mass images can be improved, and finally, the extraction rate and the accuracy of the super-large-scale digital circuit are greatly improved.

Description

Image difference detection method and device
Technical Field
The embodiment of the invention relates to the technical field of image processing, in particular to the technical field of integrated circuit image processing, and particularly relates to an image difference detection method and device.
Background
With the rapid development of IC design, the integration level of integrated circuits is higher and higher, and the scale of digital circuits is larger and larger.
For circuits with scales of million gates or even more than ten million gates, circuits with the same basic unit type but different driving and size in digital gates are common, the basic units have slightly different images (see area a in fig. 1a and area B in fig. 1B), and the existing example verification process and algorithm which are manually executed cannot meet the requirements.
Moreover, subjective factors exist in manual perspective confirmation, visual fatigue easily causes false detection, the existing example confirmation flow and algorithm consume huge human resources, the final circuit extraction rate and accuracy are not high, and the vision of high-efficiency circuit analysis is hindered.
Disclosure of Invention
The invention provides an image difference detection method and device, which are used for improving the efficiency and accuracy of the difference detection of mass images and finally greatly improving the extraction rate and accuracy of a super-large-scale digital circuit.
In a first aspect, an embodiment of the present invention provides an image difference detection method, applied to an integrated circuit, including:
preprocessing the collected sample image;
constructing a difference detection model based on a neural network;
performing model training on the difference detection model according to the preprocessed sample image to obtain a target difference detection model;
and detecting images to be detected according to the target difference detection model, determining difference information between the images to be detected, and determining abnormal image data according to the difference information.
In a second aspect, an embodiment of the present invention further provides an abnormal image detection apparatus, applied in an integrated circuit, where the apparatus includes:
the preprocessing module is used for preprocessing the acquired sample image;
the model construction module is used for constructing a difference detection model based on a neural network;
the model training module is used for carrying out model training on the difference detection model according to the preprocessed sample image so as to obtain a target difference detection model;
and the detection module is used for detecting the images to be detected according to the target difference detection model, determining the difference information between the images to be detected and determining abnormal image data according to the difference information.
According to the method, the collected sample images are preprocessed, the difference detection model based on the neural network is constructed, the difference detection model is trained according to the preprocessed sample images to obtain the target difference detection model, and finally the trained target difference detection model is used for carrying out difference detection on the images to be detected. The image difference is detected through the target difference detection model constructed by the neural network, so that the efficiency and the accuracy of detecting the difference of massive images can be improved, and finally, the extraction rate and the accuracy of the super-large-scale digital circuit are greatly improved.
Drawings
FIGS. 1a and 1b are contrasts of the subtle differences of the microscopic images;
fig. 2 is a flowchart of an image difference detection method according to an embodiment of the present invention;
FIG. 3 is a diagram of a base unit diff (active) layer image provided in accordance with a second embodiment of the present invention;
fig. 4 is a binary diagram of a basic unit according to a second embodiment of the present invention;
FIG. 5a is a basic unit template image according to a second embodiment of the present invention;
FIG. 5b is an example image of a basic unit according to a second embodiment of the present invention;
fig. 6 is a schematic structural diagram of an image difference detection apparatus according to a third embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Example one
Fig. 2 is a flowchart of an image difference detection method according to an embodiment of the present invention, which is applicable to a situation of detecting a slight difference of basic unit images in a digital gate circuit, and the method can be executed by an image difference detection apparatus, and specifically includes the following steps:
and S110, preprocessing the acquired sample image.
Wherein, the sample image is the correct image in the basic unit image referenced by the example. Because the original image to be detected has multiple data information such as contours, bright spots and backgrounds, and the original image also has the problems of uneven black and white, distortion and the like, in order to improve the accuracy of difference detection, the acquired sample image needs to be preprocessed to eliminate the problems in the sample image.
Specifically, the preprocessing the acquired sample image includes: marking the collected sample image; and performing feature enhancement on the marked sample image.
The purpose of labeling the sample image is to remove irrelevant information such as background in the sample image so as to obtain image characteristic information in the sample image. The sample images can be collected and labeled by EDA software for IC competitive analysis, the type and number of labels labeled in this embodiment can be adjusted according to specific requirements, and each label includes the coating information of the type.
Further, the feature enhancement of the labeled sample image includes: and carrying out random noise addition, random brightness change, random cropping and random scaling on the marked image.
Illustratively, randomly adding noise to the annotated image includes: the Gaussian noise with the total pixel number of 1/10 sample images is randomly added to the image, and the operation can be used for balancing the sample images with uneven black and white or uneven exposure.
Random luminance variation: the image data is randomly changed to an adding or subtracting operation of values between 0-255 to equalize the brightness of the image.
And (3) random cutting: and randomly searching a size area which does not exceed the boundary for the image to cut, and because the size of the acquired sample image is larger, finding an area corresponding to the characteristic point on the image to cut so as to realize the characteristic enhancement of the sample image.
Random scaling: the image data is randomly subjected to 1-3 times of magnification or reduction and rotation operations to increase the number of samples.
And S120, constructing a difference detection model based on the neural network.
The difference detection model comprises a multi-level feature extraction network and a multi-level fusion network, and feature information extracted by the difference detection model can be compared with standard image features, so that the difference between the images is detected.
And S130, performing model training on the difference detection model according to the preprocessed sample image to obtain a target difference detection model.
In this embodiment, the feature-enhanced sample image is used to train the difference detection model to construct the target difference detection model.
Specifically, performing model training on a difference detection model constructed based on a neural network according to the preprocessed sample image to obtain a target difference detection model, including:
and setting the learning rate, the number of images in single training and the number of convergence steps of the difference detection model during model training, and updating the weight parameters of the difference detection model by using a gradient function to obtain a target difference detection model.
As an alternative embodiment, the learning rate is set to 0.001 from 0, the learning rate is reduced to 0.001 × 0.8 every 1000 steps, the weight parameters of the difference detection model are updated by adopting a random gradient reduction method, the number of single training images is set to 32, the current best model is saved when the value of the model loss function is not reduced within 100 steps, the training is finished, and the current best model is used as the best target difference detection model.
S140, detecting the images to be detected according to the target difference detection model, determining difference information between the images to be detected, and determining abnormal image data according to the difference information.
After the target difference detection model is obtained through training, the target difference detection model is deployed on a server to detect the image to be detected.
Detecting images to be detected according to a target difference detection model to determine difference information between the images to be detected and determine abnormal image data according to the difference information, wherein the method comprises the following steps: sending the standard image of each basic unit category into the target difference detection model to obtain a feature expression vector of the standard image; and sequentially sending all the images under the corresponding categories into the target difference detection model to obtain the feature expression vector of each image, comparing the feature expression vector of the standard image under each category with the feature expression vector of each image under the corresponding category, and determining abnormal image data under the corresponding category according to the comparison result.
For example, the output results of the images to be detected may be sorted in the order of decreasing the degree of difference between the feature representation vector of the standard image and the feature representation vectors of the other images in the corresponding category, if the feature representation vector of the image ranked first is compared with the feature representation vector of the standard image and is determined as an abnormal image, the images are continuously and sequentially determined until the first normal image is found, and all the images ranked before the normal image are abnormal images; if the image ranked first is not an abnormal image, all images are not abnormal images.
Example two
The embodiment of the invention provides a specific application example of the technical scheme of the application, wherein a digital area of a chip in the example comprises 240 types of basic units, each type of basic unit comprises a certain number of examples, and the total number of the examples is 30w plus. The elementary cells referenced by these examples may be erroneous, which cannot be found quickly based on the prior art. That is, the chip entry number region corresponds to 240 types of sample images, each corresponding to a certain amount of example image data. By the mass data detection method based on the neural network, the difference between the example image and the basic unit template is calculated from the mass microscopic image, and the example with the wrong reference can be rapidly excavated.
Referring to fig. 3, fig. 3 is a basic unit diff (active) layer image, that is, an image to be detected, and the basic unit image is labeled and feature-enhanced to obtain the basic unit binary image of fig. 4.
Next, a neural network-based difference detection model was constructed with an input image size of 200 x 200 and a first convolution layer of the model consisting of convolution kernels of 3 x 64. Four identical neural network blocks are then connected in series, each consisting of 3 parallel neural networks. The first neural network is input and directly output; the second neural network is a convolution layer of 3 x 128, and the convolution result is used as an output; the third neural network is connected with the convolution layer of 1 × 128, then connected with the convolution layer of 3 × 64, then connected with the convolution layer of 1 × 128, and the result is output. The output of each neural network block is a stack of three parallel outputs. The serially connected neural network blocks are up-sampled to 200 x 200, respectively, and then each up-sampled result is stacked. Then connecting 4 neural network blocks which are the same as the neural network blocks in series, finally connecting two full-connection layers, wherein the output of the last full-connection layer is b data of (b,128), and 128 characteristic representation vectors extracted from each data.
And obtaining feature expression vectors of all example original images of each basic unit through the model, comparing the difference between the feature expression vectors of the template images of the basic units and the feature expression vectors of each example image, and detecting abnormal data. Fig. 5a is a basic unit template image according to a second embodiment of the present invention. By the above method, the difference between the region D in FIG. 5b and the region C in FIG. 5a can be rapidly identified, and the example reference error is confirmed by analysis.
The invention can simplify the existing complicated execution steps into an end-to-end operation mode by depending on the high fitting capability of the neural network, and the final accuracy can be improved to 99.997 percent through experimental verification.
EXAMPLE III
Fig. 6 is an image difference detecting apparatus according to a third embodiment of the present invention, which is capable of performing an image difference detecting method according to any embodiment of the present invention.
The device includes: the device comprises a preprocessing module, a model building module, a model training module and a detection module.
The preprocessing module 210 is configured to preprocess the acquired sample image;
a model construction module 220, configured to construct a neural network-based difference detection model;
a model training module 230, configured to perform model training on the difference detection model according to the preprocessed sample image to obtain a target difference detection model;
the detection module 240 is configured to detect an image to be detected according to a target difference detection model, determine difference information between the images to be detected, and determine abnormal image data according to the difference information.
Specifically, the preprocessing module 210 is specifically configured to:
marking the collected sample image;
and performing feature enhancement on the marked sample image.
Wherein, carrying out characteristic enhancement on the marked sample image comprises the following steps:
and carrying out random noise addition, random brightness change, random cropping and random scaling on the marked image.
The difference detection model constructed based on the neural network comprises a multi-level feature extraction network and a multi-level fusion network.
Further, the model training module 230 is specifically configured to: and setting the learning rate, the number of images in single training and the number of convergence steps of the difference detection model during model training, and updating the weight parameters of the difference detection model by using a gradient function to obtain a target difference detection model.
The model training module 230 is further specifically configured to: setting the learning rate to be 0.001 from 0, reducing the learning rate to be 0.001 x 0.8 every 1000 steps, updating the weight parameters of the difference detection model by adopting a random gradient descent mode, setting the number of single training images to be 32, saving the current model when the value of the model loss function does not descend within 100 steps, and taking the current model as the optimal target difference detection model.
The detection module 240 is specifically configured to: sending the standard image of each basic unit category into the target difference detection model to obtain a feature expression vector of the standard image;
sequentially sending all images under the corresponding category into the target difference detection model to obtain a feature expression vector of each image;
and comparing the feature representation vector of the standard image under each category with the feature representation vector of each image under the corresponding category, and determining abnormal image data under the corresponding category according to the comparison result.
Comparing the feature representation vector of the standard image in each category with the feature representation vector of each image in the corresponding category, and determining abnormal image data in the corresponding category according to the comparison result, wherein the method comprises the following steps:
sorting the output results of the images to be detected according to the sequence of the difference degree from large to small between the feature expression vectors of the standard images and the feature expression vectors of other images in the corresponding categories, if the image arranged at the first position is judged as an abnormal image, continuing to judge the images in sequence until the first normal image is found, wherein all the images arranged in front of the normal image are abnormal images;
if the image ranked first is not an abnormal image, all the images are not abnormal images.
The image difference detection device provided by the embodiment of the invention can execute the image difference detection method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims (10)

1. An image difference detection method applied to an integrated circuit includes:
preprocessing the collected sample image;
constructing a difference detection model based on a neural network;
performing model training on the difference detection model according to the preprocessed sample image to obtain a target difference detection model;
and detecting images to be detected according to the target difference detection model, determining difference information between the images to be detected, and determining abnormal image data according to the difference information.
2. The method of claim 1, wherein preprocessing the acquired sample image comprises:
marking the collected sample image;
and performing feature enhancement on the marked sample image.
3. The method of claim 2, wherein feature enhancing the annotated sample image comprises:
and carrying out random noise addition, random brightness change, random cropping and random scaling on the marked image.
4. The method of claim 1, wherein the neural network-based difference detection model comprises a multi-level feature extraction network and a multi-level fusion network.
5. The method of claim 1, wherein model training the neural network-based difference detection model according to the preprocessed sample images to obtain a target difference detection model comprises:
and setting the learning rate, the number of images in single training and the number of convergence steps of the difference detection model during model training, and updating the weight parameters of the difference detection model by using a gradient function to obtain a target difference detection model.
6. The method according to claim 5, wherein the setting of the learning rate, the number of images per training and the number of convergence steps in model training of the difference detection model, and the updating of the weight parameters of the difference detection model by using a gradient function to obtain a target difference detection model comprises:
setting the learning rate to be 0.001 from 0, reducing the learning rate to be 0.001 x 0.8 every 1000 steps, updating the weight parameters of the difference detection model by adopting a random gradient descent mode, setting the number of single training images to be 32, saving the current model when the value of the model loss function does not descend within 100 steps, and taking the current model as the optimal target difference detection model.
7. The method of claim 1, wherein detecting the image to be detected according to a target difference detection model, determining difference information between the images to be detected, and determining abnormal image data according to the difference information comprises:
sending the standard image of each basic unit category into the target difference detection model to obtain a feature expression vector of the standard image;
sequentially sending all images under the corresponding category into the target difference detection model to obtain a feature expression vector of each image;
and comparing the feature representation vector of the standard image under each category with the feature representation vector of each image under the corresponding category, and determining abnormal image data under the corresponding category according to the comparison result.
8. The method according to claim 7, wherein comparing the feature representation vector of the standard image in each category with the feature representation vector of each image in the corresponding category, and determining abnormal image data in the corresponding category according to the comparison result comprises:
sorting the output results of the images to be detected according to the sequence of the difference degree from large to small between the feature expression vectors of the standard images and the feature expression vectors of other images in the corresponding categories, if the image arranged at the first position is judged as an abnormal image, continuing to judge the images in sequence until the first normal image is found, wherein all the images arranged in front of the normal image are abnormal images;
if the image ranked first is not an abnormal image, all the images are not abnormal images.
9. An image difference detection device applied to an integrated circuit, the device comprising:
the preprocessing module is used for preprocessing the acquired sample image;
the model construction module is used for constructing a difference detection model based on a neural network;
the model training module is used for carrying out model training on the difference detection model according to the preprocessed sample image so as to obtain a target difference detection model;
and the detection module is used for detecting the images to be detected according to the target difference detection model, determining the difference information between the images to be detected and determining abnormal image data according to the difference information.
10. The apparatus of claim 9, wherein the detection module is specifically configured to:
sending the standard image of each basic unit category into the target difference detection model to obtain a feature expression vector of the standard image;
sending all the images under the corresponding category into the target difference detection model in sequence to obtain the characteristic expression vector of each image,
and comparing the feature representation vector of the standard image under each category with the feature representation vector of each image under the corresponding category, and determining abnormal image data under the corresponding category according to the comparison result.
CN202110646644.8A 2021-06-10 2021-06-10 Image difference detection method and device Pending CN113554592A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110646644.8A CN113554592A (en) 2021-06-10 2021-06-10 Image difference detection method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110646644.8A CN113554592A (en) 2021-06-10 2021-06-10 Image difference detection method and device

Publications (1)

Publication Number Publication Date
CN113554592A true CN113554592A (en) 2021-10-26

Family

ID=78130455

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110646644.8A Pending CN113554592A (en) 2021-06-10 2021-06-10 Image difference detection method and device

Country Status (1)

Country Link
CN (1) CN113554592A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115082400A (en) * 2022-06-21 2022-09-20 北京字跳网络技术有限公司 Image processing method and device, computer equipment and readable storage medium

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115082400A (en) * 2022-06-21 2022-09-20 北京字跳网络技术有限公司 Image processing method and device, computer equipment and readable storage medium

Similar Documents

Publication Publication Date Title
CN111179251B (en) Defect detection system and method based on twin neural network and by utilizing template comparison
US10878283B2 (en) Data generation apparatus, data generation method, and data generation program
CN110060237B (en) Fault detection method, device, equipment and system
CN109919908B (en) Method and device for detecting defects of light-emitting diode chip
CN111507958A (en) Target detection method, training method of detection model and electronic equipment
CN114862845B (en) Defect detection method, device and equipment for mobile phone touch screen and storage medium
CN111680705B (en) MB-SSD method and MB-SSD feature extraction network suitable for target detection
CN111242899B (en) Image-based flaw detection method and computer-readable storage medium
CN111209858A (en) Real-time license plate detection method based on deep convolutional neural network
CN109389105A (en) A kind of iris detection and viewpoint classification method based on multitask
CN117355038B (en) X-shaped hole processing method and system for circuit board soft board
CN115953373B (en) Glass defect detection method, device, electronic equipment and storage medium
CN114494780A (en) Semi-supervised industrial defect detection method and system based on feature comparison
CN117392042A (en) Defect detection method, defect detection apparatus, and storage medium
CN116385430A (en) Machine vision flaw detection method, device, medium and equipment
CN113554592A (en) Image difference detection method and device
CN111178445A (en) Image processing method and device
CN115984158A (en) Defect analysis method and device, electronic equipment and computer readable storage medium
CN115830302B (en) Multi-scale feature extraction fusion power distribution network equipment positioning identification method
CN116934737A (en) Weld joint combination defect identification and classification method
CN116363136A (en) On-line screening method and system for automatic production of motor vehicle parts
CN105809180A (en) Image processing method and device
CN107403428A (en) Plate defect recognition methods and device
CN114581407A (en) Self-adaptive defect detection method for photovoltaic module
CN111352827A (en) Automatic testing method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination