CN114119557A - Optical element rapid dark field detection method based on convolutional neural network - Google Patents

Optical element rapid dark field detection method based on convolutional neural network Download PDF

Info

Publication number
CN114119557A
CN114119557A CN202111429842.5A CN202111429842A CN114119557A CN 114119557 A CN114119557 A CN 114119557A CN 202111429842 A CN202111429842 A CN 202111429842A CN 114119557 A CN114119557 A CN 114119557A
Authority
CN
China
Prior art keywords
dark field
model
neural network
convolutional neural
training
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202111429842.5A
Other languages
Chinese (zh)
Other versions
CN114119557B (en
Inventor
陈明君
李小涛
尹朝阳
赵林杰
程健
袁晓东
郑万国
廖威
王海军
张传超
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Harbin Institute of Technology
Original Assignee
Harbin Institute of Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Harbin Institute of Technology filed Critical Harbin Institute of Technology
Priority to CN202111429842.5A priority Critical patent/CN114119557B/en
Publication of CN114119557A publication Critical patent/CN114119557A/en
Application granted granted Critical
Publication of CN114119557B publication Critical patent/CN114119557B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0004—Industrial image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/60—Analysis of geometric attributes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30108—Industrial image inspection
    • G06T2207/30164—Workpiece; Machine component

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Computational Linguistics (AREA)
  • Software Systems (AREA)
  • Biomedical Technology (AREA)
  • Health & Medical Sciences (AREA)
  • Biophysics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Quality & Reliability (AREA)
  • Geometry (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

A fast dark field detection method for an optical element based on a convolutional neural network relates to the technical field of optical element detection and is used for solving the problems of low accuracy and efficiency of surface defect identification of a large-caliber element in the prior art. The technical points of the invention comprise: scanning and collecting the surface of the element in a dark field environment, and adjusting exposure values to obtain dark field image sets corresponding to different exposure values; inputting the preprocessed dark field image set into a recognition model based on a convolutional neural network for training; inputting the image to be recognized into the trained recognition model to obtain a recognition result; the method comprises the steps of carrying out target segmentation and image interception by using high exposure value data, carrying out identification classification by using low exposure value data, and introducing transfer learning in a model training stage, so that the model training times are reduced, and the model identification accuracy is improved. The invention identifies the defect area through the dark field stage, eliminates a large amount of pollutants, and greatly reduces the whole detection period of the optical element.

Description

Optical element rapid dark field detection method based on convolutional neural network
Technical Field
The invention relates to the technical field of optical element detection, in particular to a fast dark field detection method of an optical element based on a convolutional neural network.
Background
The large-caliber fused quartz optical element is the most commonly applied optical element in a terminal optical component of a high-power solid laser device, but in a high-power solid laser system, when the element is irradiated by strong laser, ablation point micro defects such as micro cracks, micro pits and the like are easily generated. Studies have shown that when micro-defects such as micro-cracks or ablation spots are generated, the rear surface micro-defect size of the optical element grows exponentially as the number of laser shots increases. When the number of micro-defects increases to a certain extent, the optical element will be rejected and cannot be used further. For large-caliber fused quartz optical elements, the processing time period is long, the cost is high, and in order to prolong the service life of the optical elements, the main solution adopted at home and abroad is to carry out laser micro-repair on generated micro defects so as to greatly improve the damage resistance of the optical elements, thereby inhibiting the damage growth, prolonging the service life of the elements and reducing the cost.
The fused silica optical component inevitably introduces a large amount of pollutants in the process of transportation, installation and use, the pollutants are attached to the component, the shape and the damage are similar, and the detection of the damage on the surface of the component is interfered. Therefore, in order to obtain damage information on the surface of the component, damage and contamination are distinguished. The main method adopted at present is to obtain the position information of all defects and pollutants during dark field detection, and then transfer the element to a microscope camera for more clear photographing to determine whether the point is damaged. However, the moving microscope camera positions and identifies the target points one by one, which takes a lot of time, and lengthens the detection process.
Disclosure of Invention
In view of the above problems, the invention provides a fast dark field detection method for an optical element based on a convolutional neural network, which is used for solving the problems of low accuracy and efficiency of identifying surface defects of a large-caliber element in the prior art.
A fast dark field detection method for an optical element based on a convolutional neural network comprises the following steps:
scanning and collecting the surface of an element in a dark field environment, and adjusting exposure values to obtain dark field image sets corresponding to different exposure values;
secondly, preprocessing the dark field image set;
inputting the preprocessed dark field image set into a recognition model based on a convolutional neural network for training to obtain a trained recognition model;
inputting the image to be recognized containing the element surface defect area into a trained recognition model to obtain a recognition result; the identification result includes whether the defective area is a pseudo-defective area.
Further, the specific steps of the first step include:
step one, adopting an annular light source to irradiate the surface of an element at a low angle to form a dark field environment, carrying out row-by-row and column-by-column moving scanning on the surface of the element in the dark field environment, and acquiring a plurality of subgraphs of a plurality of preset photographing positions; the method comprises the following steps that each preset photographing position correspondingly collects a plurality of sub-images with different exposure values, and the plurality of different exposure values are divided into a high exposure value and a low exposure value;
step two, processing the multiple subgraphs to obtain multiple element surface defect area images;
and step three, carrying out dust blowing treatment on the element surface defect areas, and correspondingly labeling the images of the element surface defect areas according to the dust blowing treatment result so as to obtain image data of the positive sample and the negative sample.
Further, the specific process of the second step comprises the following steps: and performing binarization processing on the subgraph corresponding to the high exposure value of each preset photographing position, extracting the outline of the defect region, calculating the minimum external square of the outline, and respectively capturing pictures in the subgraphs corresponding to the high exposure value and the low exposure value according to the central position and the size of the minimum external square to respectively obtain images of the defect region corresponding to the high exposure value and the low exposure value.
Further, the preprocessing in the second step includes data enhancement of turning, rotating and noise disturbance on the image data.
Further, the third step specifically comprises:
thirdly, dividing a preprocessed dark field image set according to a size range by taking the minimum circumscribed circle diameter enveloping the outline of the defect region after binarization processing as the pixel size of the defect region and based on the pixel size;
step two, dividing dark field image data corresponding to a plurality of size ranges into a training set and a verification set according to a proportion;
inputting the defect area image corresponding to the low exposure value in the training set into a recognition model based on a convolutional neural network ResNet for training;
and step three, inputting the verification set into the model after each training to adjust model parameters, and stopping training until the model identification accuracy is not improved any more, so as to obtain the trained identification model.
Further, in the third step, the following cross entropy function is used as a loss function to calculate the error between the predicted value and the true value:
Figure BDA0003379856120000021
in the formula, yiRepresents a sample label, the positive sample is 1, and the negative sample is 0; p is a radical ofiRepresents the probability that sample i is predicted to be a positive sample; n represents the total number of samples.
Further, in the third step and the fourth step, the model identification accuracy is calculated by using the following formula:
Figure BDA0003379856120000022
in the formula, TP, FP, FN and TN respectively represent the number of true positive, false positive, true negative and false negative in the recognition result.
And further, in the third step, a ResNet network model is built by using a residual error structure, and the node weight of the ResNet network model obtained by pre-training under the ImageNet data set is migrated and loaded in the initial training stage.
Further, in the third step, in the training process, part of the convolutional layer corresponding to the node weight loaded by migration is frozen, and the error back propagation is performed on the convolutional layer which is not frozen.
The beneficial technical effects of the invention are as follows:
the invention provides a method for carrying out target segmentation and image interception by using high exposure value data, which improves the efficiency and accuracy of target identification and image interception; low exposure value data is used for classification, so that the model identification accuracy is increased; the target point in the dark field picture is automatically extracted by using a target identification method, and the position and size information of the target point is obtained, so that the data set acquisition process is accelerated; the optimal size range of model classification is obtained by training the data sets in different size ranges, and the model identification accuracy is high and the consumed time is short in the range; transfer learning is introduced in a dark field detection stage, so that the times of training of the model are reduced, and the model identification accuracy is improved; the target points are identified in the dark field stage, a large number of pollutants are removed, the number of the target points needing to be identified in the microscopic station is greatly reduced, and the whole detection period of the optical element is greatly reduced.
Drawings
The present invention may be better understood by reference to the following description taken in conjunction with the accompanying drawings, which are incorporated in and form a part of this specification, and which are used to further illustrate preferred embodiments of the present invention and to explain the principles and advantages of the present invention.
FIG. 1 is a schematic overall flow chart of a fast dark-field inspection method for an optical device according to an embodiment of the present invention;
FIG. 2 is a schematic structural diagram of a dark field data acquisition device in an embodiment of the invention;
FIG. 3 is a schematic diagram of a residual structure in an embodiment of the present invention;
FIG. 4 is a diagram of two specific residual structure examples in an embodiment of the present invention;
FIG. 5 is a diagram illustrating an example of a process for building a ResNet network by using two specific residual error structures according to an embodiment of the present invention; wherein, the graph (a) is an 18-layer and 34-layer network; graph (b) is a 50, 101, and 152 layer network;
FIG. 6 is a schematic diagram of forward and reverse transfer of a convolutional neural network in an embodiment of the present invention;
FIG. 7 is a diagram illustrating a transfer learning training model according to an embodiment of the present invention;
FIG. 8 is a schematic diagram of different variable combinations in an embodiment of the present invention;
FIG. 9 is a schematic diagram of an embodiment of a confusion matrix;
FIG. 10 is a diagram of an example of dark field image target identification result in the embodiment of the invention;
FIG. 11 is a comparison graph of the recognition results of the same target under different exposure values in the embodiment of the present invention;
FIG. 12 is a schematic diagram of data set expansion according to an embodiment of the present invention; wherein, the graph (a) is an original graph; FIG. (b) is a horizontally mirrored view; FIG. (c) is a vertically mirrored view; fig. (d) is a graph after random rotation.
Detailed Description
In order that those skilled in the art will better understand the disclosure, exemplary embodiments or examples of the disclosure are described below with reference to the accompanying drawings. It is obvious that the described embodiments or examples are only some, but not all embodiments or examples of the invention. All other embodiments or examples obtained by a person of ordinary skill in the art based on the embodiments or examples of the present invention without any creative effort shall fall within the protection scope of the present invention.
In order to accelerate the detection process of the surface defects of the elements, the invention introduces deep learning into a dark field detection part of the optical elements, and identifies and eliminates pollutants as much as possible in the dark field detection part by the methods of transfer learning, data set enhancement, picture exposure value change and size threshold setting so as to accelerate the detection process of the elements. The method can effectively identify the pollutants in the dark field stage, reduces the number of target points identified by the microscope camera, and effectively shortens the element detection period.
The embodiment of the invention provides a fast dark field detection method based on a convolutional neural network, which comprises the following steps:
scanning and collecting the surface of an element in a dark field environment, and adjusting exposure values to obtain dark field image sets corresponding to different exposure values;
step two, preprocessing the dark field image set;
inputting the preprocessed dark field image set into a recognition model based on a convolutional neural network for training to obtain a trained recognition model;
inputting the image to be recognized containing the element surface defect area into a trained recognition model to obtain a recognition result; the recognition result includes whether the defective area is a pseudo-defective area.
In this embodiment, optionally, the specific steps of the first step include:
step one, adopting an annular light source to irradiate the surface of an element at a low angle to form a dark field environment, carrying out row-by-row and column-by-column moving scanning on the surface of the element in the dark field environment, and acquiring a plurality of subgraphs of a plurality of preset photographing positions; the method comprises the following steps that each preset photographing position correspondingly collects a plurality of sub-images with different exposure values, and the plurality of different exposure values are divided into a high exposure value and a low exposure value;
step two, processing the multiple subgraphs to obtain multiple element surface defect area images;
and step three, carrying out dust blowing treatment on the element surface defect areas, and correspondingly labeling the images of the element surface defect areas according to the dust blowing treatment result so as to obtain image data of the positive sample and the negative sample.
In this embodiment, optionally, the specific process of the second step includes: and performing binarization processing on the subgraph corresponding to the high exposure value of each preset photographing position, extracting the outline of the defect region, calculating the minimum external square of the outline, and respectively capturing pictures in the subgraphs corresponding to the high exposure value and the low exposure value according to the central position and the size of the minimum external square to respectively obtain images of the defect region corresponding to the high exposure value and the low exposure value.
In this embodiment, optionally, the preprocessing in the second step includes data enhancement of flipping, rotating, and noise disturbance on the image data.
In this embodiment, optionally, the specific steps of step three include:
thirdly, dividing a preprocessed dark field image set according to a size range by taking the minimum circumscribed circle diameter enveloping the outline of the defect region after binarization processing as the pixel size of the defect region and based on the pixel size;
step two, dividing dark field image data corresponding to a plurality of size ranges into a training set and a verification set according to a proportion;
inputting the defect area image corresponding to the low exposure value in the training set into a recognition model based on a convolutional neural network ResNet for training;
and step three, inputting the verification set into the model after each training to adjust model parameters, and stopping training until the model identification accuracy is not improved any more, so as to obtain the trained identification model.
In this embodiment, optionally, in the third step, the following cross entropy function is used as the loss function to calculate an error between the predicted value and the true value:
Figure BDA0003379856120000051
in the formula, yiRepresents a sample label, the positive sample is 1, and the negative sample is 0; p is a radical ofiRepresents the probability that sample i is predicted to be a positive sample; n represents the total number of samples.
In this embodiment, optionally, in the third step, the model identification accuracy is calculated by using the following formula:
Figure BDA0003379856120000052
in the formula, TP, FP, FN and TN respectively represent the number of true positive, false positive, true negative and false negative in the recognition result.
In this embodiment, optionally, in the third step, a ResNet network model is built by using a residual structure, and the node weights of the ResNet network model pre-trained under the ImageNet data set are migrated and loaded at the initial training stage.
In this embodiment, optionally, in the third step, in the training process, part of the convolutional layer corresponding to the node weight loaded by migration is frozen, and error back propagation is performed on the convolutional layer that is not frozen.
Another embodiment of the present invention provides a fast dark field detection method based on a convolutional neural network, which mainly relates to the content of acquiring data sets with different exposure times, preprocessing the data sets, training by applying a classification model, applying the model, and the like, and the overall flow is schematically shown in fig. 1. Firstly, dark field image acquisition is carried out on the surface of an element to obtain an original image, then target identification is carried out to obtain position and size information of a defect and a pollutant target, and the information is used for cutting the original image to obtain a data set. Preprocessing the obtained data set, removing picture background information, dividing the data set into a training set and a verification set according to a certain proportion, training the model and verifying the validity of the model, and expanding the data set. And putting the obtained data set into a model for training, and changing various parameters to obtain a better classification result. Finally, the method is used in the detection process of the element. The method comprises the following specific steps:
step 1, collecting dark field images.
According to the embodiment of the invention, the full-aperture dark field image is obtained by scanning and photographing, a plurality of images with different exposure time are photographed at each scanning and photographing position, the gray value of the target area tends to be saturated at a high exposure value, the image segmentation is easy to realize, the method can be used for positioning and size measurement of a target point, the detail change of the target gray can be displayed at a low exposure value, and more features are included, so that the classification is convenient. In order to obtain the optimal size for classification, dark field images under 7 exposure values within the range of 10ms to 40ms are collected.
Dark field data acquisition apparatus as shown in fig. 2, the apparatus includes a motion stage and a dark field scanning system. The motion platform comprises X, Y, Z three motion axes, and the motion directions of X, Y, Z motion axes are respectively consistent with the directions of X, Y, Z coordinate axes of a machine tool coordinate system; the motion platform can carry an optical large-aperture element to realize the movement along the X, Y axis direction, and carry a dark field scanning system to realize the movement along the Z axis direction, thereby completing the focusing of the lens. The dark field scanning system comprises a high-resolution area array dark field camera, a double telecentric lens and an annular light source, can realize distortion-free detection in a range of 50mm multiplied by 50mm, and needs to move an element to collect and splice a plurality of pictures when detecting an optical element with a larger size; in order to form a dark field environment, an annular light source is adopted to irradiate the surface of the optical element at a low angle, and the brightness and the on-off state of the light source are automatically adjusted through a light source controller. Dark field images under different exposure times are different in classification effect during convolutional network identification, and dark field data sets under different exposure times are acquired in order to obtain the optimal exposure time. The specific data set acquisition steps are as follows:
step 1-1: and controlling the motion platform to move the image part to be acquired of the optical element to the dark field lens, wherein the size of the optical element is 430mm multiplied by 430mm, and 9 multiplied by 9 sub-images are needed to completely cover the whole element. And moving the element to a dark field camera position, adjusting the exposure value every 5ms within 10ms-40ms to photograph the element, moving the element to the next position after photographing under various exposure values is completed, and sequentially completing photographing at 81 positions to obtain a dark field image of the full aperture of the element under different exposure time.
Step 1-2: and (3) enabling the image under high exposure to be close to the binary image, carrying out target identification on the image, and acquiring the position and the size of a target point in the image. And calculating the position information of the target point on the element according to the corresponding relation between the image and the real element. And writing the target point sequence number, the target point position and the target point size into the xml file.
Step 1-3: and sequentially moving the microscope camera to the position of the defect point according to the information in the xml file, blowing dust on the surface of the optical element, if the target point disappears, indicating that the point is a pollutant, otherwise, indicating that the point is a defect. By the method, the specific type of the target point can be obtained, and a labeled dark field data set is obtained.
And 2, acquiring a data set.
According to the embodiment of the invention, a large number of pictures are needed during model training, and the manual picture acquisition is time-consuming. And carrying out binarization processing on the high-exposure image to extract a target area, carrying out contour extraction on the target area, calculating a minimum external square of the contour, and intercepting the low-exposure image as classification data by using the square. And positioning the target point to a microscopic station according to the corresponding relation between the dark field image and the element, and carrying out dust blowing and wiping treatment on the microscopic station, wherein if the target point disappears, the target point is a pollutant, and if the target point does not disappear, the target point is a defect. And completing the labeling through the steps, and expanding the labeled picture to obtain an expanded data set. The image preprocessing process comprises two parts of background information elimination and data set expansion. The specific process comprises the following steps:
step 2-1: in order to improve the accuracy of model identification, the image background information is removed, and the image background is filled with pure color, so that the background information is prevented from influencing the model identification process.
Step 2-2: the convolutional neural network needs a large amount of data in the model training process, and in order to expand a data set without substantially increasing pictures, data enhancement modes such as mirroring, rotation and the like are carried out on an original data set. Thereby improving the generalization ability of the model.
And 3, dividing the data set.
According to the embodiment of the invention, data sets in different size ranges are obtained by dividing according to the sizes of target points, the data sets are divided into a training set and a verification set, the training set is used for training the model, and the verification set is used for evaluating the performance of the model and adjusting the parameters of the model. The specific process of data set division is as follows:
step 3-1: in order to obtain the size threshold, the expanded data set is classified according to the size of a target point and divided into a data set with the size larger than 30um, a data set larger than 40um, a data set larger than 50um, a data set larger than 100um, a data set larger than 200um, a data set larger than 300um and a data set larger than 500 um.
Step 3-2: and (3) carrying out 1: and 9, dividing the test set into a verification set and a training set respectively. Note that all pictures after the same picture augmentation are placed in the same category.
And 4, training based on the ResNet classification model.
According to the embodiment of the invention, because the number of the images in the training set is less, in order to accelerate the training speed and improve the model identification effect, the ResNet network based on transfer learning is used for training, and the model is verified after each training is finished, so that the prediction effect of the model is evaluated. And comparing the migration learning results of different layers, and selecting a model with good prediction effect and less time consumption. The ResNet model comprises a residual structure as shown in FIG. 3, and the residual structure can prevent the gradient disappearance or gradient explosion of the network when the number of layers is large, so that the network model can be deep due to the residual structure. Through two specific residual error structures shown in fig. 4, the ResNet network can build a model with multiple layers, wherein the residual error structures shown in fig. 4(a) are applied to 18-layer and 34-layer networks, and the residual error structures shown in fig. 4(b) are applied to more than 50-layer networks.
Fig. 5 shows a specific process of building a ResNet network by using a residual structure, fig. 5(a) shows an 18-layer network and a 34-layer network, and fig. 5(b) shows a 50-layer, 101-layer and 152-layer network, in order to be suitable for recognition of a dark field image, the number of parameters of a model full-link layer is modified to be 2, and the two parameters are converted into prediction probabilities of damage and pollutants by a Sigmoid function, so that a result is output. In order to achieve better effect, the model with the best comprehensive effect is selected by training the layers 18, 34, 50, 101 and 152 of the ResNet model respectively. The specific training process of the network is as follows:
step 4-1: because the number of pictures in the dark field dataset is limited, transfer learning is introduced to improve model accuracy. The specific process of importing all node weights of the network model trained under the ImageNet data set is to download a pretrained file of a ResNet network model based on the ImageNet data set on the network, wherein the pretrained file contains the node weights of the ResNet model trained under the ImageNet data set, but the weights are suitable for recognizing images of the ImageNet data set, and when the pretrained network model is applied to dark-field images, the pretrained network model needs to be retrained to adjust part of the node weights so that the recognition effect of the node weights is more matched with the dark-field images.
The shallow layers in the convolutional neural network are mainly used for acquiring general features, and the deep layers are mainly used for acquiring semantic features. The difference between the ImageNet data set and the dark field data set is large, but the universal characteristics of the ImageNet data set are similar, so that the process of changing the weight of the characteristic layer by directly using the weight of a part of the universal layer can be selected during transfer learning, as shown in FIG. 7, the process is completed by partially freezing the pre-training file during the training process, firstly, the pre-training file is imported into the training model, and the weight of the relevant layer is not changed during the reverse transmission of the model through a freezing command, so that the aim of keeping the pre-training weight is fulfilled. Error back propagation is carried out on the unfrozen layers during training, and it can be seen from fig. 5 that ResNet network structures with different layers are obtained by splicing 4 basic modules with different numbers, pre-training weights of one, two and three residual blocks are respectively frozen from low to high during transfer learning, and the number of frozen layers for transfer learning is selected according to a training result. And finally, selecting the three residual blocks before freezing, and retraining the fourth residual block to obtain the weight parameters of the dark field image recognition model. Fig. 6 shows the structure of the convolutional neural network and the parameters of each layer, where w is a weight parameter, the purpose of the network training is to obtain an appropriate weight parameter through continuous iteration, and equation (1) illustrates the forward propagation and output process of the model.
Figure BDA0003379856120000081
In the formula, xiIs an input parameter;
Figure BDA0003379856120000082
the weight value between the ith node of the k-1 layer and the jth node of the k layer is obtained; sigma is an activation function;
Figure BDA0003379856120000083
bias for the ith neuron of layer j; y isiIs an output parameter.
Step 4-2: from the step 1, the dark field data sets have different exposure times, wherein the exposure time is longer and is closer to the binary image, the contained information is less, and the dark field data sets are not suitable for image identification; since there is much information in the image with shorter exposure time, several sets of data sets with shorter exposure time are used for model training. And 3, dividing the data set according to the size of the target point, and respectively importing the data sets in different size ranges into a model for training in the same exposure time in order to obtain an ideal size threshold. And loading different dark field data sets into ResNet models with different layers in sequence for training, and finally selecting a size threshold value of 50um under the condition of simultaneously considering the expansion of the identification range and the identification accuracy as much as possible, namely identifying all target points above 50um by the models. This process is illustrated in fig. 8.
Step 4-3: and evaluating the model by applying the verification set after the training set is trained once, evaluating the model by adopting the confusion matrix shown as the formula 2, and evaluating the effect of the model by the number of true positive, false positive, true negative and false negative in the model prediction result by adopting the confusion matrix method shown as the figure 9. The invention aims to eliminate pollutants as much as possible in a dark field stage, but the defects are not wanted to be judged as pollutants by mistake, and the defects are used as positive samples and the pollutants are used as negative samples in a model, so that the true positive property is more emphasized in the evaluation process, and the proportion of the true positive property is improved as much as possible.
Figure BDA0003379856120000091
In the formula, TP, FP, FN and TN represent the number of true positive, false positive, true negative and false negative in the prediction result respectively.
And calculating the difference between the predicted value and the true value of the model by the cross entropy loss function shown in the formula 3, and modifying the model parameters by reverse error transfer in a mode of firstly solving the derivative of the loss function on each weight parameter by a chain rule.
Figure BDA0003379856120000092
In the formula, yiRepresents a sample label, the positive sample is 1, and the negative sample is 0; p is a radical ofiRepresents the probability that sample i is predicted to be a positive sample; n represents the total number of samples.
And correcting the weight parameters in a mode shown in the formula 4, and repeating the steps to continuously reduce the model loss value until the model converges.
Figure BDA0003379856120000093
Wherein w is an optimized weight parameter; alpha is a momentum coefficient, and eta is a learning rate; l is a loss function; v. ofiIs the amount of update to the weight parameter.
And 5, applying the model.
According to the embodiment of the invention, the trained model is loaded into the whole optical element detection and repair system, and the target points in the dark field stage are identified and classified. The specific process of model application is to load the trained ResNet network model, input the obtained dark field target point image into the network for recognition, and the recognition result can play a decision-making role in the work of the microscopic station.
Another embodiment of the present invention provides an example analysis of a fast dark field detection method based on a convolutional neural network, which is illustrated by taking a training and application process of a certain dark field detection model as an example, and the specific process is as follows:
1. training of dark field inspection models
(1) And collecting dark field pictures under different exposure values. The method comprises the steps of firstly adjusting the exposure time of a high-resolution dark-field camera, respectively obtaining dark-field images at different exposure times, identifying the size and the position of a target point in a high-exposure-value dark-field image by using target identification, and cutting the image according to the size and the position of the target point to obtain target point images at different exposure values. The target recognition and the pictures at different exposure values are shown in fig. 10 and 11, respectively.
(2) The data set is expanded, for example, by using the target point ID-563 at a certain exposure value, and the data enhancement modes of reverse horizontal mirroring, vertical mirroring and random rotation are performed, as shown in FIG. 12.
(3) The data sets are divided by size into >30um, >40um, >50um, >100um, >200um, >300um and >500 um. Data sets were randomized as 1: and 9, dividing the ratio to obtain a verification set and a training set.
(4) And importing the training set and the verification set into ResNet network models with different layers for training, and selecting an exposure time of 15ms, a size threshold of 50um and a 50-layer ResNet network structure as a final dark field detection model according to a training result.
2. Application of dark field detection model
(1) And loading a trained dark field detection model.
(2) And adjusting the exposure time of the dark field camera to be the optimal exposure time.
(3) And carrying out target identification on the dark field picture, cutting to obtain a target point picture to be identified, and inputting the target point picture into a trained model for identification.
(4) And writing the model identification result into a dark field file, and if the target point is identified as a pollutant, identifying at the microscopic station.
The invention realizes the identification and classification of the target points in the dark field detection stage through the processes, greatly reduces the number of the target points to be identified in the microscopic station and effectively shortens the detection period of the whole optical element.
While the invention has been described with respect to a limited number of embodiments, those skilled in the art, having benefit of this description, will appreciate that other embodiments can be devised which do not depart from the scope of the invention as described herein. The present invention has been disclosed in an illustrative rather than a restrictive sense, and the scope of the present invention is defined by the appended claims.

Claims (9)

1. A fast dark field detection method for an optical element based on a convolutional neural network is characterized by comprising the following steps:
scanning and collecting the surface of an element in a dark field environment, and adjusting exposure values to obtain dark field image sets corresponding to different exposure values;
secondly, preprocessing the dark field image set;
inputting the preprocessed dark field image set into a recognition model based on a convolutional neural network for training to obtain a trained recognition model;
inputting the image to be recognized containing the element surface defect area into a trained recognition model to obtain a recognition result; the identification result includes whether the defective area is a pseudo-defective area.
2. The fast dark-field detection method for the optical element based on the convolutional neural network as claimed in claim 1, wherein the specific steps of the first step include:
step one, adopting an annular light source to irradiate the surface of an element at a low angle to form a dark field environment, carrying out row-by-row and column-by-column moving scanning on the surface of the element in the dark field environment, and acquiring a plurality of subgraphs of a plurality of preset photographing positions; the method comprises the following steps that each preset photographing position correspondingly collects a plurality of sub-images with different exposure values, and the plurality of different exposure values are divided into a high exposure value and a low exposure value;
step two, processing the multiple subgraphs to obtain multiple element surface defect area images;
and step three, carrying out dust blowing treatment on the element surface defect areas, and correspondingly labeling the images of the element surface defect areas according to the dust blowing treatment result so as to obtain image data of the positive sample and the negative sample.
3. The fast dark-field detection method for the optical element based on the convolutional neural network as claimed in claim 2, wherein the specific process of the second step comprises: and performing binarization processing on the subgraph corresponding to the high exposure value of each preset photographing position, extracting the outline of the defect region, calculating the minimum external square of the outline, and respectively capturing pictures in the subgraphs corresponding to the high exposure value and the low exposure value according to the central position and the size of the minimum external square to respectively obtain images of the defect region corresponding to the high exposure value and the low exposure value.
4. The fast dark-field detection method for optical elements based on the convolutional neural network as claimed in claim 3, wherein the preprocessing in step two includes data enhancement of flipping, rotating and noise disturbance to the image data.
5. The fast dark-field detection method for the optical element based on the convolutional neural network as claimed in claim 4, wherein the specific steps of step three include:
thirdly, dividing a preprocessed dark field image set according to a size range by taking the minimum circumscribed circle diameter enveloping the outline of the defect region after binarization processing as the pixel size of the defect region and based on the pixel size;
step two, dividing dark field image data corresponding to a plurality of size ranges into a training set and a verification set according to a proportion;
inputting the defect area image corresponding to the low exposure value in the training set into a recognition model based on a convolutional neural network ResNet for training;
and step three, inputting the verification set into the model after each training to adjust model parameters, and stopping training until the model identification accuracy is not improved any more, so as to obtain the trained identification model.
6. The fast dark field detection method for the optical element based on the convolutional neural network as claimed in claim 5, wherein the following cross entropy function is used as a loss function to calculate the error between the predicted value and the true value in the third step:
Figure FDA0003379856110000021
in the formula, yiRepresents a sample label, the positive sample is 1, and the negative sample is 0; p is a radical ofiRepresents the probability that sample i is predicted to be a positive sample; n represents the total number of samples.
7. The fast dark field detection method for optical elements based on the convolutional neural network as claimed in claim 6, wherein the model identification accuracy is calculated in step three or four by using the following formula:
Figure FDA0003379856110000022
in the formula, TP, FP, FN and TN respectively represent the number of true positive, false positive, true negative and false negative in the recognition result.
8. The fast dark field detection method for the optical element based on the convolutional neural network as claimed in claim 7, wherein in the third step, a ResNet network model is built by using a residual structure, and the node weights of the ResNet network model pre-trained under an ImageNet data set are migrated and loaded in the initial training stage.
9. The fast dark-field detection method for optical elements based on the convolutional neural network as claimed in claim 8, wherein in the third step, part of the convolutional layers corresponding to the node weights loaded by migration are frozen in the training process, and error back propagation is performed on the convolutional layers which are not frozen.
CN202111429842.5A 2021-11-29 2021-11-29 Optical element rapid dark field detection method based on convolutional neural network Active CN114119557B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111429842.5A CN114119557B (en) 2021-11-29 2021-11-29 Optical element rapid dark field detection method based on convolutional neural network

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111429842.5A CN114119557B (en) 2021-11-29 2021-11-29 Optical element rapid dark field detection method based on convolutional neural network

Publications (2)

Publication Number Publication Date
CN114119557A true CN114119557A (en) 2022-03-01
CN114119557B CN114119557B (en) 2024-04-26

Family

ID=80371132

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111429842.5A Active CN114119557B (en) 2021-11-29 2021-11-29 Optical element rapid dark field detection method based on convolutional neural network

Country Status (1)

Country Link
CN (1) CN114119557B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114782286A (en) * 2022-06-21 2022-07-22 奥蒂玛光学科技（深圳）有限公司 Defect repairing method, optical repairing device, electronic device and storage medium

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109934811A (en) * 2019-03-08 2019-06-25 中国科学院光电技术研究所 A kind of optical element surface defect inspection method based on deep learning
CN110031471A (en) * 2019-05-17 2019-07-19 中国科学院自动化研究所 Optical elements of large caliber surface defect Analysis in Growth method, system, device
WO2019191346A1 (en) * 2018-03-28 2019-10-03 Kla-Tencor Corporation Training a neural network for defect detection in low resolution images
CN110706224A (en) * 2019-10-09 2020-01-17 中国科学院自动化研究所 Optical element weak scratch detection method, system and device based on dark field image
CN111476283A (en) * 2020-03-31 2020-07-31 上海海事大学 Glaucoma fundus image identification method based on transfer learning
WO2021139069A1 (en) * 2020-01-09 2021-07-15 南京信息工程大学 General target detection method for adaptive attention guidance mechanism
CN113256576A (en) * 2021-05-18 2021-08-13 福州大学 Automatic optical element detection system and method based on polarization imaging and machine learning

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019191346A1 (en) * 2018-03-28 2019-10-03 Kla-Tencor Corporation Training a neural network for defect detection in low resolution images
CN109934811A (en) * 2019-03-08 2019-06-25 中国科学院光电技术研究所 A kind of optical element surface defect inspection method based on deep learning
CN110031471A (en) * 2019-05-17 2019-07-19 中国科学院自动化研究所 Optical elements of large caliber surface defect Analysis in Growth method, system, device
CN110706224A (en) * 2019-10-09 2020-01-17 中国科学院自动化研究所 Optical element weak scratch detection method, system and device based on dark field image
WO2021139069A1 (en) * 2020-01-09 2021-07-15 南京信息工程大学 General target detection method for adaptive attention guidance mechanism
CN111476283A (en) * 2020-03-31 2020-07-31 上海海事大学 Glaucoma fundus image identification method based on transfer learning
CN113256576A (en) * 2021-05-18 2021-08-13 福州大学 Automatic optical element detection system and method based on polarization imaging and machine learning

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
尹朝阳;张德志;赵林杰;陈明君;程健;蒋晓东;苗心向;牛龙飞: "大口径反射镜表面颗粒污染物暗场检测算法研究", 光学学报, no. 007, 31 December 2020 (2020-12-31) *
石祥滨;房雪键;张德园;郭忠强: "基于深度学习混合模型迁移学习的图像分类", 系统仿真学报, vol. 28, no. 1, 8 January 2016 (2016-01-08) *
范勇;陈念年;高玲玲;贾渊;王俊波;程晓锋;: "大口径精密表面疵病的数字化检测系统", 强激光与粒子束, no. 07, 15 July 2009 (2009-07-15) *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114782286A (en) * 2022-06-21 2022-07-22 奥蒂玛光学科技（深圳）有限公司 Defect repairing method, optical repairing device, electronic device and storage medium

Also Published As

Publication number Publication date
CN114119557B (en) 2024-04-26

Similar Documents

Publication Publication Date Title
CN111179251B (en) Defect detection system and method based on twin neural network and by utilizing template comparison
CN109724984B (en) Defect detection and identification device and method based on deep learning algorithm
CN111507990A (en) Tunnel surface defect segmentation method based on deep learning
CN112598672A (en) Pavement disease image segmentation method and system based on deep learning
CN114120317B (en) Optical element surface damage identification method based on deep learning and image processing
CN111667455A (en) AI detection method for various defects of brush
CN109509187A (en) A kind of efficient check algorithm for the nibs in big resolution ratio cloth image
CN111814850A (en) Defect detection model training method, defect detection method and related device
CN113239930B (en) Glass paper defect identification method, system, device and storage medium
CN110927171A (en) Bearing roller chamfer surface defect detection method based on machine vision
KR100457100B1 (en) Pattern inspection method and inspection apparatus
CN111507998A (en) Depth cascade-based multi-scale excitation mechanism tunnel surface defect segmentation method
NL2030315B1 (en) Computer-implemented ai method for detecting surface defects of electroluminescent semiconductor panel
CN114627383A (en) Small sample defect detection method based on metric learning
CN113077450A (en) Cherry grading detection method and system based on deep convolutional neural network
CN114359695A (en) Insulator breakage identification method based on uncertainty estimation
CN116843650A (en) SMT welding defect detection method and system integrating AOI detection and deep learning
CN114119557B (en) Optical element rapid dark field detection method based on convolutional neural network
CN114742818A (en) Visual defect detection system based on industrial area array camera and detection method thereof
CN111612787B (en) Concrete crack high-resolution image lossless semantic segmentation method and device and storage medium
CN114743102A (en) Furniture board oriented flaw detection method, system and device
CN117256514B (en) Method for removing unqualified hatching eggs for sorting system
CN114782410A (en) Insulator defect detection method and system based on lightweight model
CN111160100A (en) Lightweight depth model aerial photography vehicle detection method based on sample generation
CN113222901A (en) Method for detecting surface defects of steel ball based on single stage

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant