CN114119557B - Optical element rapid dark field detection method based on convolutional neural network - Google Patents
Optical element rapid dark field detection method based on convolutional neural network Download PDFInfo
- Publication number
- CN114119557B CN114119557B CN202111429842.5A CN202111429842A CN114119557B CN 114119557 B CN114119557 B CN 114119557B CN 202111429842 A CN202111429842 A CN 202111429842A CN 114119557 B CN114119557 B CN 114119557B
- Authority
- CN
- China
- Prior art keywords
- dark field
- model
- training
- neural network
- optical element
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 40
- 230000003287 optical effect Effects 0.000 title claims abstract description 36
- 238000013527 convolutional neural network Methods 0.000 title claims abstract description 26
- 238000012549 training Methods 0.000 claims abstract description 71
- 238000000034 method Methods 0.000 claims abstract description 50
- 230000007547 defect Effects 0.000 claims abstract description 42
- 230000008569 process Effects 0.000 claims description 28
- 238000012795 verification Methods 0.000 claims description 13
- 238000012545 processing Methods 0.000 claims description 11
- 238000007664 blowing Methods 0.000 claims description 8
- 239000000428 dust Substances 0.000 claims description 8
- 238000007781 pre-processing Methods 0.000 claims description 8
- 230000002950 deficient Effects 0.000 claims description 6
- 238000002372 labelling Methods 0.000 claims description 4
- 238000013508 migration Methods 0.000 claims description 4
- 230000005012 migration Effects 0.000 claims description 4
- 230000001678 irradiating effect Effects 0.000 claims description 3
- 239000003344 environmental pollutant Substances 0.000 abstract description 15
- 231100000719 pollutant Toxicity 0.000 abstract description 15
- 238000013526 transfer learning Methods 0.000 abstract description 9
- 230000011218 segmentation Effects 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 9
- 230000000694 effects Effects 0.000 description 9
- 230000006870 function Effects 0.000 description 9
- 230000033001 locomotion Effects 0.000 description 9
- 230000008014 freezing Effects 0.000 description 4
- 238000007710 freezing Methods 0.000 description 4
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 description 3
- VYPSYNLAJGMNEJ-UHFFFAOYSA-N Silicium dioxide Chemical compound O=[Si]=O VYPSYNLAJGMNEJ-UHFFFAOYSA-N 0.000 description 3
- 239000005350 fused silica glass Substances 0.000 description 3
- 239000011159 matrix material Substances 0.000 description 3
- 238000012546 transfer Methods 0.000 description 3
- 238000002679 ablation Methods 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000013145 classification model Methods 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 230000004913 activation Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 239000000356 contaminant Substances 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000008034 disappearance Effects 0.000 description 1
- 238000012854 evaluation process Methods 0.000 description 1
- 238000004880 explosion Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000003709 image segmentation Methods 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 210000002569 neuron Anatomy 0.000 description 1
- 230000002035 prolonged effect Effects 0.000 description 1
- 230000008439 repair process Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 229920006395 saturated elastomer Polymers 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/048—Activation functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/60—Analysis of geometric attributes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30108—Industrial image inspection
- G06T2207/30164—Workpiece; Machine component
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Biomedical Technology (AREA)
- Health & Medical Sciences (AREA)
- Biophysics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Quality & Reliability (AREA)
- Geometry (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
A fast dark field detection method of an optical element based on a convolutional neural network relates to the technical field of optical element detection and is used for solving the problems of low accuracy and low efficiency of identifying surface defects of a large-caliber element in the prior art. The technical key points of the invention include: scanning and collecting the element surface in a dark field environment, and adjusting exposure values to obtain dark field image sets corresponding to different exposure values; inputting the preprocessed dark field image set into an identification model based on a convolutional neural network for training; inputting the image to be identified into a trained identification model to obtain an identification result; the method comprises the steps of carrying out target segmentation and image interception by using high exposure value data, carrying out recognition and classification by using low exposure value data, introducing transfer learning in a model training stage, reducing model training times and improving model recognition accuracy. The invention identifies the defect area through the dark field stage, eliminates a large amount of pollutants, and greatly reduces the whole detection period of the optical element.
Description
Technical Field
The invention relates to the technical field of optical element detection, in particular to a fast dark field detection method of an optical element based on a convolutional neural network.
Background
The large-caliber fused quartz optical element is the most commonly applied optical element in the terminal optical component of a high-power solid laser device, but in a high-power solid laser system, when the element is irradiated by strong laser, micro defects such as micro cracks, micro pits and the like of a ablation point are easy to generate. Research shows that after micro defects such as microcracks or ablation points are generated, the size of the micro defects on the rear surface of the optical element increases exponentially with the increase of the laser irradiation times. When the number of microdefects increases to some extent, the optical element will be rejected and not continue to be used. For the large-caliber fused quartz optical element, the processing time period is long, the cost is high, and in order to prolong the service life of the optical element, the main solution adopted at home and abroad is to carry out laser micro-repair on the generated micro defects, so that the damage resistance of the optical element is greatly improved, the damage growth is restrained, the service life of the element is prolonged, and the cost is reduced.
The fused silica optical element inevitably introduces a large amount of pollutants in the process of transportation, installation and use, and the pollutants are attached to the element, have similar shapes to damage, and can interfere with the detection of the damage on the surface of the element. Therefore, in order to obtain damage information on the surface of the element, damage and contaminants are distinguished. The main method adopted at present is to obtain the position information of all defects and pollutants in dark field detection, then transfer the element to a microscope camera for clearer photographing, and determine whether the point is damage. However, moving the microscope camera to locate the target points one by one and identifying it consumes a lot of time, making the detection process longer.
Disclosure of Invention
In view of the problems, the invention provides a convolution neural network-based optical element rapid dark field detection method, which is used for solving the problems of low accuracy and low efficiency of identifying surface defects of a large-caliber element in the prior art.
An optical element rapid dark field detection method based on a convolutional neural network comprises the following steps:
step one, scanning and collecting the surface of an element in a dark field environment, and adjusting exposure values to obtain dark field image sets corresponding to different exposure values;
step two, preprocessing the dark field image set;
Inputting the preprocessed dark field image set into a recognition model based on a convolutional neural network for training, and obtaining a trained recognition model;
Inputting an image to be identified containing the element surface defect area into a trained identification model to obtain an identification result; the identification result includes whether the defective area is a pseudo defective area.
Further, the specific steps of the first step include:
The method comprises the steps of irradiating the surface of an element at a low angle by adopting an annular light source to form a dark field environment, performing row-by-row and column-by-column moving scanning on the surface of the element in the dark field environment, and acquiring a plurality of subgraphs of a plurality of preset photographing positions; wherein, each preset photographing position correspondingly collects a plurality of subgraphs with different exposure values, and the different exposure values are divided into a high exposure value and a low exposure value;
step two, processing a plurality of subgraphs to obtain a plurality of element surface defect area images;
And thirdly, carrying out dust blowing treatment on the defect areas on the surfaces of the elements, and correspondingly labeling the images of the defect areas on the surfaces of the elements according to the dust blowing treatment result so as to obtain positive sample and negative sample image data.
Further, the specific process of the step two comprises the following steps: binarization processing is carried out on the subgraph corresponding to the high exposure value of each preset photographing position, the outline of the defect area is extracted, the minimum external square of the outline is calculated, and the central position and the size of the minimum external square are respectively used for capturing images in the subgraph corresponding to the high exposure value and the low exposure value, so that the defect area images corresponding to the high exposure value and the low exposure value are respectively obtained.
Further, the preprocessing in the second step includes data enhancement of image data with flipping, rotation and noise disturbance.
Further, the specific steps of the third step include:
Step three, taking the minimum circumcircle diameter of the outline of the envelope defect area after binarization processing as the pixel size of the defect area, and dividing the preprocessed dark field image set according to a size range based on the pixel size;
step three, dividing dark field image data corresponding to a plurality of size ranges into a training set and a verification set according to a proportion;
thirdly, inputting the defect area image corresponding to the low exposure value in the training set into an identification model based on a convolutional neural network ResNet for training;
And thirdly, inputting the verification set into the model after each training to adjust model parameters until the model identification accuracy is not improved, and stopping training to obtain a trained identification model.
Further, in step three, the error between the predicted value and the true value is calculated using the following cross entropy function as the loss function:
wherein y i represents a sample label, the positive sample is 1, and the negative sample is 0; p i denotes the probability that sample i is predicted as a positive sample; n represents the total number of samples.
Further, in the third and fourth steps, the model recognition accuracy is calculated by using the following formula:
Wherein TP, FP, FN, TN represents the number of true positives, false positives, true negatives, and false negatives in the recognition result, respectively.
Further, in the third step, a residual structure is used for building ResNet a network model, and the node weight of the ResNet network model pre-trained under the ImageNet data set is migrated and loaded in the initial training stage.
And in the third step, freezing part of the convolution layers corresponding to the node weights loaded by migration in the training process, and performing error back propagation on the unfrozen convolution layers.
The beneficial technical effects of the invention are as follows:
According to the invention, the high exposure value data is applied to target segmentation and image interception, so that the efficiency and accuracy of target identification and image interception are improved; the low exposure value data is applied to classification, so that the model identification accuracy is increased; the target point in the dark field picture is automatically extracted by using the target identification method, and the position and size information of the target point are obtained, so that the process of acquiring the data set is quickened; by training the data sets in different size ranges, the optimal size range of model classification is obtained, and the model identification accuracy is high and the time consumption is short in the range; the transfer learning is introduced in the dark field detection stage, so that the number of times that the model needs to be trained is reduced, and the model identification accuracy is improved; the target points are identified through the dark field stage, so that a large amount of pollutants are removed, the number of the target points required to be identified by the microscope station is greatly reduced, and the whole detection period of the optical element is greatly reduced.
Drawings
The invention may be better understood by reference to the following description taken in conjunction with the accompanying drawings, which are included to provide a further illustration of the preferred embodiments of the invention and to explain the principles and advantages of the invention, together with the detailed description below.
FIG. 1is a schematic overall flow chart of a fast dark field detection method for an optical element according to an embodiment of the invention;
FIG. 2 is a schematic diagram of a dark field data acquisition device according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a residual structure in an embodiment of the present invention;
FIG. 4 is a diagram of two specific residual structure examples in an embodiment of the present invention;
FIG. 5 is a diagram illustrating an example of a process for constructing ResNet networks for two specific residual structures in an embodiment of the present invention; wherein, figure (a) is an 18-layer and 34-layer network; figure (b) is a 50, 101 and 152 layer network;
FIG. 6 is a schematic diagram of forward and reverse transfer of a convolutional neural network in an embodiment of the present invention;
FIG. 7 is a schematic diagram of a training model for transfer learning in an embodiment of the present invention;
FIG. 8 is a schematic diagram of the combination of different variables in an embodiment of the present invention;
FIG. 9 is a schematic diagram of a confusion matrix in an embodiment of the invention;
FIG. 10 is an exemplary graph of a dark field image object recognition result in an embodiment of the present invention;
FIG. 11 is a graph showing the comparison of the same target recognition results at different exposure values according to the embodiment of the present invention;
FIG. 12 is a schematic diagram of data set expansion in an embodiment of the invention; wherein, figure (a) is an original figure; FIG. (b) is a horizontally mirrored view; FIG. (c) is a vertically mirrored view; fig. (d) is a graph after random rotation.
Detailed Description
In order that those skilled in the art will better understand the present invention, exemplary embodiments or examples of the present invention will be described below with reference to the accompanying drawings. It is apparent that the described embodiments or examples are only implementations or examples of a part of the invention, not all. All other embodiments or examples, which may be made by one of ordinary skill in the art without undue burden, are intended to be within the scope of the present invention based on the embodiments or examples herein.
In order to accelerate the detection process of the surface defects of the element, the invention introduces deep learning into the dark field detection part of the optical element, and identifies and eliminates pollutants as much as possible in the dark field detection part by the methods of transfer learning, data set enhancement, changing the exposure value of the picture and setting the size threshold value so as to accelerate the detection process of the element. The method can effectively identify the pollutants in the dark field stage, reduces the number of target points identified by a microscopic camera, and effectively shortens the element detection period.
The embodiment of the invention provides a fast dark field detection method based on a convolutional neural network, which comprises the following steps:
step one, scanning and collecting the surface of an element in a dark field environment, and adjusting exposure values to obtain dark field image sets corresponding to different exposure values;
step two, preprocessing a dark field image set;
Inputting the preprocessed dark field image set into a recognition model based on a convolutional neural network for training, and obtaining a trained recognition model;
Inputting an image to be identified containing the element surface defect area into a trained identification model to obtain an identification result; the identification result includes whether the defective area is a pseudo defective area.
In this embodiment, optionally, the specific steps of the first step include:
The method comprises the steps of irradiating the surface of an element at a low angle by adopting an annular light source to form a dark field environment, performing row-by-row and column-by-column moving scanning on the surface of the element in the dark field environment, and acquiring a plurality of subgraphs of a plurality of preset photographing positions; wherein, each preset photographing position correspondingly collects a plurality of subgraphs with different exposure values, and the different exposure values are divided into a high exposure value and a low exposure value;
step two, processing a plurality of subgraphs to obtain a plurality of element surface defect area images;
And thirdly, carrying out dust blowing treatment on the defect areas on the surfaces of the elements, and correspondingly labeling the images of the defect areas on the surfaces of the elements according to the dust blowing treatment result so as to obtain positive sample and negative sample image data.
In this embodiment, optionally, the specific process of the step one includes: binarization processing is carried out on the subgraph corresponding to the high exposure value of each preset photographing position, the outline of the defect area is extracted, the minimum external square of the outline is calculated, and the central position and the size of the minimum external square are respectively used for capturing images in the subgraph corresponding to the high exposure value and the low exposure value, so that the defect area images corresponding to the high exposure value and the low exposure value are respectively obtained.
In this embodiment, optionally, the preprocessing in the second step includes data enhancement of flipping, rotation, and noise disturbance of the image data.
In this embodiment, optionally, the specific steps of the third step include:
Step three, taking the minimum circumcircle diameter of the outline of the envelope defect area after binarization processing as the pixel size of the defect area, and dividing the preprocessed dark field image set according to the size range based on the pixel size;
step three, dividing dark field image data corresponding to a plurality of size ranges into a training set and a verification set according to a proportion;
thirdly, inputting the defect area image corresponding to the low exposure value in the training set into an identification model based on a convolutional neural network ResNet for training;
And thirdly, inputting the verification set into the model after each training to adjust model parameters until the model identification accuracy is not improved, and stopping training to obtain a trained identification model.
In this embodiment, optionally, in step three, the error between the predicted value and the true value is calculated using the following cross entropy function as the loss function:
wherein y i represents a sample label, the positive sample is 1, and the negative sample is 0; p i denotes the probability that sample i is predicted as a positive sample; n represents the total number of samples.
In this embodiment, optionally, in the third and fourth steps, the model recognition accuracy is calculated using the following formula:
Wherein TP, FP, FN, TN represents the number of true positives, false positives, true negatives, and false negatives in the recognition result, respectively.
In this embodiment, optionally, in step three, a residual structure is used to build ResNet a network model, and the node weights of the ResNet network model obtained by pre-training under the ImageNet dataset are migrated and loaded in the initial stage of training.
In this embodiment, optionally, in the third step, a part of the convolution layers corresponding to the node weights loaded by migration is frozen in the training process, and error back propagation is performed on the unfrozen convolution layers.
The invention provides a fast dark field detection method based on a convolutional neural network, which mainly relates to the acquisition of data sets with different exposure time, the pretreatment of the data sets, the training by applying a classification model, the application of the model and the like, and the whole flow is schematically shown in figure 1. Firstly, acquiring dark field images on the surface of an element to obtain an original image, then carrying out target recognition to obtain target position and size information of defects and pollutants, and cutting the original image by using the information to obtain a data set. Preprocessing the obtained data set, removing the background information of the picture, dividing the data set into a training set and a verification set according to a certain proportion, using the training set and the verification set for training a model and verifying the validity of the model, and expanding the data set. And putting the obtained data set into a model for training, and changing various parameters to obtain a better classification result. The method is finally used in the detection of components. The method comprises the following specific steps:
and 1, collecting dark field images.
According to the embodiment of the invention, firstly, a full-caliber dark field image is obtained through scanning photographing, a plurality of images with different exposure time are photographed at each scanning photographing position, the gray value of a target area tends to be saturated under a high exposure value, the image segmentation is easy to realize, the method can be used for positioning and size measurement of a target point, the detail change of the target gray can be displayed under a low exposure value, and the classification is convenient due to the fact that more features are included. To obtain the best size for classification, dark field images were acquired at 7 exposure values in the range of 10ms to 40 ms.
Dark field data acquisition device as shown in fig. 2, the device comprises a motion platform and a dark field scanning system. The motion platform comprises X, Y, Z motion axes, and the motion directions of the X, Y, Z motion axes are respectively consistent with the directions of X, Y, Z coordinate axes of a machine tool coordinate system; the motion platform can carry an optical large-caliber element to realize the movement along the X, Y axis direction, and carry a dark field scanning system to realize the movement along the Z axis direction, so as to complete the focusing of the lens. The dark field scanning system comprises a high-resolution area array dark field camera, a double telecentric lens and an annular light source, can realize the undistorted detection within the range of 50mm multiplied by 50mm, and needs to move an element to collect and splice a plurality of pictures when detecting an optical element with a larger size; in order to form a dark field environment, the surface of the optical element is irradiated by an annular light source at a low angle, and the brightness and the on-off of the light source are automatically adjusted through a light source controller. Dark field pictures under different exposure times have different classification effects during convolutional network identification, and dark field data sets under different exposure times are acquired in order to obtain the optimal exposure time. The specific data set acquisition steps are as follows:
Step 1-1: the motion platform is controlled to move the image part to be acquired of the optical element to the dark field lens, the size of the optical element is 430mm multiplied by 430mm, and the whole element can be completely covered by 9 multiplied by 9 subgraphs. And (3) moving the element to the position of a dark field camera, adjusting exposure values at intervals of 5ms within 10ms-40ms to photograph the element, moving the element to the next position after photographing under various exposure values, and sequentially completing photographing at 81 positions to obtain full-caliber dark field images of the element under different exposure times.
Step 1-2: and (3) the image is close to a binarization image under high exposure, the image is subjected to target identification, and the position and the size of a target point in the image are obtained. And calculating the position information of the target point on the element according to the corresponding relation between the image and the real element. And writing the target point sequence number, the target point position and the target point size into the xml file.
Step 1-3: and (3) sequentially moving the microscopic camera to the position of a defect point according to the information in the xml file, blowing dust on the surface of the optical element, and if the target point disappears, indicating that the point is a pollutant, otherwise, the point is a defect. By the method, the specific category of the target point can be obtained, and therefore the dark field data set with the marked target point is obtained.
And 2, acquiring a data set.
According to the embodiment of the invention, a large number of pictures are needed during model training, and manual picture acquisition is very time-consuming. And carrying out binarization processing on the high-exposure image to extract a target area, carrying out contour extraction on the target area, calculating the minimum circumscribed square of the contour, and taking the square to intercept the low-exposure image as classification data. And positioning the target point to a microscopic station according to the corresponding relation between the dark field image and the element, and carrying out dust blowing and wiping treatment on the target point, wherein if the target point disappears, the target point is a pollutant, and if the target point does not disappear, the target point is a defect. And finishing the labeling through the steps, and expanding the labeled picture to obtain an expanded data set. The image preprocessing process comprises background information rejection and data set expansion. The specific process is as follows:
step 2-1: in order to improve the model recognition accuracy, firstly, the background information of the picture is removed, the background of the picture is filled with pure colors, and the background information is prevented from influencing the model recognition process.
Step 2-2: the convolutional neural network requires a large amount of data in the model training process, and in order to expand the data set without substantially increasing the pictures, the original data set is subjected to data enhancement modes such as mirroring, rotation and the like. Thereby improving the generalization capability of the model.
And 3, dividing the data set.
According to the embodiment of the invention, the data sets with different size ranges are obtained according to the size division of the target point, the data sets are divided into a training set and a verification set, the training set is used for training the model, and the verification set is used for evaluating the performance of the model and adjusting the parameters of the model. The specific process of data set division is as follows:
Step 3-1: to derive the size threshold, the expanded dataset is classified according to the size of the target point into a dataset with a size greater than 30um, a dataset with a size greater than 40um, a dataset with a size greater than 50um, a dataset with a size greater than 100um, a dataset with a size greater than 200um, a dataset with a size greater than 300um, and a dataset with a size greater than 500 um.
Step 3-2: carrying out 1 on the data sets of different size ranges obtained in the previous step: and 9 proportion division is respectively a verification set and a training set. Note that all pictures that are expanded from the same picture are placed in the same category.
And 4, training based on ResNet classification models.
According to the embodiment of the invention, as the number of the pictures of the training set is small, in order to accelerate the training speed and improve the model identification effect, the invention uses ResNet networks based on transfer learning for training, and the model is verified after each round of training is completed, and the prediction effect is evaluated. And comparing the migration learning results of different layers, and selecting a model with good prediction effect and less time consumption. The ResNet model contains a residual structure shown in fig. 3, and the residual structure can enable the network not to have gradient disappearance or gradient explosion when the number of layers is large, so that the residual structure enables the hierarchy of the network model to be deep. Models with multiple layers can be built by ResNet networks through two specific residual structures as shown in fig. 4, 18-layer and 34-layer networks apply the residual structure shown in fig. 4 (a), and more than 50-layer networks apply the residual structure shown in fig. 4 (b).
In fig. 5, a specific process of constructing ResNet network by residual structure is shown, fig. 5 (a) is 18 layers and 34 layers, fig. 5 (b) is 50 layers, 101 layers and 152 layers, in order to be suitable for the identification of dark field images, the invention modifies the number of parameters of the model full-connection layer to 2, and converts the two parameters into the prediction probability of damage and pollutant through Sigmoid function so as to output the result. In order to achieve better effects, 18 layers, 34 layers, 50 layers, 101 layers and 152 layers of ResNet models are respectively trained to select the model with the best comprehensive effect. The specific training process of the network is as follows:
step 4-1: because the number of dark field dataset pictures is limited, transfer learning is introduced to improve model accuracy. The method comprises the steps of importing all node weights of a network model obtained by training under an ImageNet data set, wherein a pre-training file of ResNet network model based on the ImageNet data set is downloaded on a network, the pre-training file comprises node weights of ResNet model trained under the ImageNet data set, the weights are applicable to identifying images of ImgeNet data set, and retraining is needed to be carried out when the weights are applied to dark field images so as to adjust part of node weights to enable the identifying effects of the node weights to be more consistent with the dark field images.
The shallower layers in the convolutional neural network are mainly used for acquiring general features, while the deeper layers are mainly used for acquiring semantic features. The difference between the ImageNet data set and the dark field data set is larger, but the general characteristics are similar, so that the process of directly using the weights of part of general layers to change the weights of the feature layers during transfer learning can be selected as shown in fig. 7, the process is completed by partially freezing the pre-training file in the training process, the pre-training file is firstly imported into a training model, and the weights of relevant layers are not changed when the model is reversely transferred through a freezing command, so that the aim of maintaining the pre-training weights is fulfilled. For the unfrozen layer number to be subjected to error back propagation during training, as can be seen from fig. 5, the ResNet network structures with different layer numbers are respectively obtained by splicing 4 basic modules with different numbers, pre-training weights of one, two and three residual blocks are respectively frozen from low to high during transfer learning, and the number of the frozen layers for transfer learning is selected according to training results. And finally, selecting three residual blocks before freezing, and retraining a fourth residual block to obtain weight parameters of the dark field image recognition model. Fig. 6 shows the convolutional neural network structure and parameters of each layer, where w is a weight parameter, and the purpose of the network training is to obtain a suitable weight parameter through continuous iteration, and equation (1) illustrates the forward propagation and output process of the model.
Wherein x i is an input parameter; The weight between the ith node of the k-1 layer and the jth node of the k layer is obtained; sigma is an activation function; bias for the j-th layer of i-th neurons; y i is the output parameter.
Step 4-2: step 1 shows that the dark field data sets have different exposure times, wherein the exposure times are longer and are closer to the binary image, and less information is contained and the dark field data sets are not suitable for image recognition; while images with shorter exposure times have more information, so several sets of data sets with shorter exposure times are used for model training. In step 3, the data sets are divided according to the size of the target point, and in order to obtain an ideal size threshold, the data sets in different size ranges are respectively imported into the model for training under the same exposure time. And loading different dark field datasets into ResNet models with different layers in sequence for training, and finally selecting a size threshold of 50um under the condition of simultaneously considering the recognition range and the recognition accuracy as large as possible, namely, the model is used for recognizing all target points above 50 um. This process is shown in fig. 8.
Step 4-3: the training set is used for evaluating the model every time training is completed, the confusion matrix shown in the formula 2 is used for evaluating the model, and the confusion matrix method shown in the figure 9 is used for evaluating the model effect through the number of true positives, false positives, true negatives and false negatives in the model prediction result. The invention aims to remove pollutants as much as possible in a dark field stage, but the defects are not hopefully misjudged as pollutants, the defects are taken as positive samples in a model, and the pollutants are taken as negative samples, so that the true positives are more important in the evaluation process, and the proportion of the true positives is increased as much as possible.
Wherein TP, FP, FN, TN represents the number of true positives, false positives, true negatives, and false negatives in the predicted result, respectively.
And (3) calculating the difference between the model predicted value and the true value through the cross entropy loss function shown in the formula 3, and modifying the model parameters through error reverse transfer in such a way that the derivative of the loss function on each weight parameter is firstly obtained through a chain rule.
Wherein y i represents a sample label, the positive sample is 1, and the negative sample is 0; p i denotes the probability that sample i is predicted as a positive sample; n represents the total number of samples.
And correcting the weight parameters in the mode shown in the formula 4, and repeating the steps to continuously reduce the model loss value until the model converges.
Wherein w is an optimized weight parameter; alpha is momentum coefficient, eta is learning rate; l is a loss function; v i is the amount of update to the weight parameters.
And 5, applying a model.
According to the embodiment of the invention, the trained model is loaded into the whole optical element detection and repair system, and the target points in the dark field stage are identified and classified. The specific process of model application is loading ResNet network models after training, inputting the obtained dark field target point pictures into a network for recognition, and the recognition result can play a role in deciding the work of a microscope station.
Another embodiment of the present invention provides an example analysis of a fast dark field detection method based on a convolutional neural network, which is described by taking training and application processes of a dark field detection model for a certain time as an example, and the specific processes are as follows:
1. Training of dark field detection models
(1) Dark field pictures at different exposure values are acquired. Firstly, adjusting exposure time of a high-resolution dark field camera, respectively acquiring dark field images under different exposure time, identifying the size and position of a target point in a dark field picture with a high exposure value by applying target identification, and cutting the picture according to the size and position to obtain the picture of the target point under different exposure values. The pictures under different exposure values for target recognition are shown in fig. 10 and 11, respectively.
(2) The data set is expanded, taking the target point ID-563 under a certain exposure value as an example, and the data enhancement modes of reverse horizontal mirror image, vertical mirror image and random rotation are adopted, as shown in FIG. 12.
(3) The dataset is divided into >30um, >40um, >50um, >100um, >200um, >300um, and >500um by size. The dataset was randomized as 1:9, dividing the proportion to obtain a verification set and a training set.
(4) And importing the training set and the verification set into ResNet network models with different layers to train, and selecting the network structure with the exposure time of 15ms, the size threshold of 50um and 50 layers ResNet as a final dark field detection model according to training results.
2. Application of dark field detection model
(1) Loading a trained dark field detection model.
(2) The exposure time of the dark field camera is adjusted to the optimal exposure time.
(3) And carrying out target recognition on the dark field picture, cutting to obtain a target point picture to be recognized, and inputting the target picture into a trained model for recognition.
(4) Writing the model identification result into a dark field file, and if the target point is identified as a pollutant, no identification is needed at a microscope station.
The invention realizes the identification and classification of the target points in the dark field detection stage through the process, greatly reduces the number of the target points to be identified in the microscope station, and effectively shortens the detection period of the whole optical element.
While the invention has been described with respect to a limited number of embodiments, those skilled in the art, having benefit of the above description, will appreciate that other embodiments are contemplated within the scope of the invention as described herein. The disclosure of the present invention is intended to be illustrative, but not limiting, of the scope of the invention, which is defined by the appended claims.
Claims (7)
1. The fast dark field detection method of the optical element based on the convolutional neural network is characterized by comprising the following steps of:
step one, scanning and collecting the surface of an element in a dark field environment, and adjusting exposure values to obtain dark field image sets corresponding to different exposure values; the method comprises the following specific steps:
The method comprises the steps of irradiating the surface of an element at a low angle by adopting an annular light source to form a dark field environment, performing row-by-row and column-by-column moving scanning on the surface of the element in the dark field environment, and acquiring a plurality of subgraphs of a plurality of preset photographing positions; wherein, each preset photographing position correspondingly collects a plurality of subgraphs with different exposure values, and the different exposure values are divided into a high exposure value and a low exposure value;
step two, processing a plurality of subgraphs to obtain a plurality of element surface defect area images;
Step three, carrying out dust blowing treatment on the defect areas on the surfaces of the elements, and correspondingly labeling the images of the defect areas on the surfaces of the elements according to the dust blowing treatment result so as to obtain positive sample and negative sample image data;
step two, preprocessing the dark field image set;
Inputting the preprocessed dark field image set into a recognition model based on a convolutional neural network for training, and obtaining a trained recognition model; the method comprises the following specific steps:
Step three, taking the minimum circumcircle diameter of the outline of the envelope defect area after binarization processing as the pixel size of the defect area, and dividing the preprocessed dark field image set according to a size range based on the pixel size;
step three, dividing dark field image data corresponding to a plurality of size ranges into a training set and a verification set according to a proportion;
thirdly, inputting the defect area image corresponding to the low exposure value in the training set into an identification model based on a convolutional neural network ResNet for training;
inputting the verification set into the model after each training to adjust model parameters until the model identification accuracy is no longer improved, and stopping training to obtain a trained identification model;
Inputting an image to be identified containing the element surface defect area into a trained identification model to obtain an identification result; the identification result includes whether the defective area is a pseudo defective area.
2. The method for fast dark-field detection of an optical element based on a convolutional neural network according to claim 1, wherein the specific process of step two comprises: binarization processing is carried out on the subgraph corresponding to the high exposure value of each preset photographing position, the outline of the defect area is extracted, the minimum external square of the outline is calculated, and the central position and the size of the minimum external square are respectively used for capturing images in the subgraph corresponding to the high exposure value and the low exposure value, so that the defect area images corresponding to the high exposure value and the low exposure value are respectively obtained.
3. The method for fast dark-field detection of an optical element based on a convolutional neural network according to claim 1, wherein the preprocessing in the second step includes data enhancement of image data with flipping, rotation, noise disturbance.
4. The method for fast dark-field detection of an optical element based on a convolutional neural network according to claim 1, wherein in step three, the error between the predicted value and the true value is calculated using the following cross entropy function as the loss function:
wherein y i represents a sample label, the positive sample is 1, and the negative sample is 0; p i denotes the probability that sample i is predicted as a positive sample; n represents the total number of samples.
5. The method for fast dark field detection of an optical element based on a convolutional neural network according to claim 1, wherein in the third and fourth steps, the model recognition accuracy is calculated by using the following formula:
Wherein TP, FP, FN, TN represents the number of true positives, false positives, true negatives, and false negatives in the recognition result, respectively.
6. The method for fast dark field detection of an optical element based on a convolutional neural network according to claim 1, wherein in the third step, a ResNet network model is built by using a residual structure, and node weights of a ResNet network model obtained by pre-training under an ImageNet data set are migrated and loaded in a training initial stage.
7. The method for fast dark field detection of an optical element based on a convolutional neural network according to claim 6, wherein in the third step, a part of the convolutional layer corresponding to the node weight loaded by migration is frozen in the training process, and error back propagation is performed on the unfrozen convolutional layer.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111429842.5A CN114119557B (en) | 2021-11-29 | 2021-11-29 | Optical element rapid dark field detection method based on convolutional neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111429842.5A CN114119557B (en) | 2021-11-29 | 2021-11-29 | Optical element rapid dark field detection method based on convolutional neural network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114119557A CN114119557A (en) | 2022-03-01 |
| CN114119557B true CN114119557B (en) | 2024-04-26 |
Family
ID=80371132
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111429842.5A Active CN114119557B (en) | 2021-11-29 | 2021-11-29 | Optical element rapid dark field detection method based on convolutional neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114119557B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114782286A (en) * | 2022-06-21 | 2022-07-22 | 奥蒂玛光学科技（深圳）有限公司 | Defect repairing method, optical repairing device, electronic device and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109934811A (en) * | 2019-03-08 | 2019-06-25 | 中国科学院光电技术研究所 | A kind of optical element surface defect inspection method based on deep learning |
| CN110031471A (en) * | 2019-05-17 | 2019-07-19 | 中国科学院自动化研究所 | Optical elements of large caliber surface defect Analysis in Growth method, system, device |
| WO2019191346A1 (en) * | 2018-03-28 | 2019-10-03 | Kla-Tencor Corporation | Training a neural network for defect detection in low resolution images |
| CN110706224A (en) * | 2019-10-09 | 2020-01-17 | 中国科学院自动化研究所 | Optical element weak scratch detection method, system and device based on dark field image |
| CN111476283A (en) * | 2020-03-31 | 2020-07-31 | 上海海事大学 | Glaucoma fundus image identification method based on transfer learning |
| WO2021139069A1 (en) * | 2020-01-09 | 2021-07-15 | 南京信息工程大学 | General target detection method for adaptive attention guidance mechanism |
| CN113256576A (en) * | 2021-05-18 | 2021-08-13 | 福州大学 | Automatic optical element detection system and method based on polarization imaging and machine learning |
-
2021
- 2021-11-29 CN CN202111429842.5A patent/CN114119557B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019191346A1 (en) * | 2018-03-28 | 2019-10-03 | Kla-Tencor Corporation | Training a neural network for defect detection in low resolution images |
| CN109934811A (en) * | 2019-03-08 | 2019-06-25 | 中国科学院光电技术研究所 | A kind of optical element surface defect inspection method based on deep learning |
| CN110031471A (en) * | 2019-05-17 | 2019-07-19 | 中国科学院自动化研究所 | Optical elements of large caliber surface defect Analysis in Growth method, system, device |
| CN110706224A (en) * | 2019-10-09 | 2020-01-17 | 中国科学院自动化研究所 | Optical element weak scratch detection method, system and device based on dark field image |
| WO2021139069A1 (en) * | 2020-01-09 | 2021-07-15 | 南京信息工程大学 | General target detection method for adaptive attention guidance mechanism |
| CN111476283A (en) * | 2020-03-31 | 2020-07-31 | 上海海事大学 | Glaucoma fundus image identification method based on transfer learning |
| CN113256576A (en) * | 2021-05-18 | 2021-08-13 | 福州大学 | Automatic optical element detection system and method based on polarization imaging and machine learning |
Non-Patent Citations (3)
| Title |
|---|
| 基于深度学习混合模型迁移学习的图像分类;石祥滨;房雪键;张德园;郭忠强;系统仿真学报;20160108;第28卷(第1期);全文 * |
| 大口径反射镜表面颗粒污染物暗场检测算法研究;尹朝阳;张德志;赵林杰;陈明君;程健;蒋晓东;苗心向;牛龙飞;光学学报;20201231(第007期);全文 * |
| 大口径精密表面疵病的数字化检测系统;范勇;陈念年;高玲玲;贾渊;王俊波;程晓锋;;强激光与粒子束;20090715(第07期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114119557A (en) | 2022-03-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111179251B (en) | Defect detection system and method based on twin neural network and by utilizing template comparison | |
| JP6866495B2 (en) | Image quality evaluation method and image quality evaluation system | |
| CN114120317B (en) | Optical element surface damage identification method based on deep learning and image processing | |
| CN112598672A (en) | Pavement disease image segmentation method and system based on deep learning | |
| KR20210107149A (en) | Methods and systems for inspection of semiconductor structures with automatically generated defect features | |
| CN113239930B (en) | Glass paper defect identification method, system, device and storage medium | |
| CN111507998B (en) | Depth cascade-based multi-scale excitation mechanism tunnel surface defect segmentation method | |
| CN111667455A (en) | AI detection method for various defects of brush | |
| CN111814850A (en) | Defect detection model training method, defect detection method and related device | |
| CN114119557B (en) | Optical element rapid dark field detection method based on convolutional neural network | |
| TWI809094B (en) | Cross layer common-unique analysis for nuisance filtering | |
| CN110927171A (en) | Bearing roller chamfer surface defect detection method based on machine vision | |
| CN114627383B (en) | Small sample defect detection method based on metric learning | |
| CN114359695A (en) | Insulator breakage identification method based on uncertainty estimation | |
| CN112150418A (en) | Intelligent identification method for magnetic powder inspection | |
| CN113240647A (en) | Mobile phone shell rear cover defect detection method and system based on deep learning | |
| CN116071294A (en) | Optical fiber surface defect detection method and device | |
| CN117256514B (en) | Method for removing unqualified hatching eggs for sorting system | |
| CN111160100A (en) | Lightweight depth model aerial photography vehicle detection method based on sample generation | |
| CN112785610B (en) | Lane line semantic segmentation method integrating low-level features | |
| CN114113116A (en) | Accurate detection process method for micro-defects on surface of large-diameter element | |
| CN113222901A (en) | Method for detecting surface defects of steel ball based on single stage | |
| CN117541568A (en) | Deep learning-based automobile brake disc surface defect detection method | |
| CN113516652A (en) | Battery surface defect and adhesive detection method, device, medium and electronic equipment | |
| CN114120318B (en) | Dark field image target point accurate extraction method based on integrated decision tree |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |