WO2023080587A1

WO2023080587A1 - Deep learning-based mlcc stacked alignment inspection system and method

Info

Publication number: WO2023080587A1
Application number: PCT/KR2022/016880
Authority: WO
Inventors: 오흥선; 손성빈; 박주찬; 이선훈; 정준욱; 박용준
Original assignee: 한국기술교육대학교 산학협력단
Priority date: 2021-11-05
Filing date: 2022-11-01
Publication date: 2023-05-11
Also published as: US20240103076A1; KR102602167B1; KR20230065558A

Abstract

A deep learning-based MLCC stacked alignment inspection system according to one embodiment of the present invention comprises an integrated defect detection unit which detects defects by detecting a key area requiring inspection in captured image data of a stacked structure of a semiconductor MLCC chip by using at least one deep learning-based key area detection model, segmenting the detected key area, and generating normal/defective data by determining whether or not a defect is present according to a standard margin rate range using the segmentation result; a result analysis unit which performs visualization of results from key area detection, segmentation, and defect detection of the integrated defect detection unit, and provides, through inspector's analysis of visualized analysis data for each step, an ability to determine whether to modify the data; and a data storage which stores the normal/defective data and the analysis data for each step.

Description

MLCC stacked alignment inspection system and method based on deep learning

The present invention relates to an MLCC stacked alignment inspection system and method, and more particularly, to a deep learning-based method for determining whether or not a defect is present by a margin ratio range using a key region detection and segmentation result of a deep learning-based model for a photographed image. It relates to an MLCC stack alignment inspection system and method.

In general, MLCC (Multi Layer Ceramic Capacitor) chips are used in various smart electronic device fields, including automobiles, as a key component that controls current to flow uniformly in circuits of electronic products.

The stacking alignment inspection of the MLCC chip may include an inspection process after the stacking process.

10 shows the entire process from lamination to inspection in the prior art. Lamination is a process of stacking several hundred layers of dielectric sheets printed with Ni electrodes in pattern units, and the laminated product is called a Green Bar, and trimming is a process of It is a process to prepare for vision inspection by dividing it into 4 equal parts and trimming the 4 sides additionally to expose the Ni electrode layer.

In addition, the inspection process is usually a process of taking images of a total of 8 points, 2 points each at the corner with a vision camera, and determining whether the laminated alignment is good or bad with the naked eye.

In addition, referring to FIG. 11, the MLCC inspection can generally focus on four types of stacked alignment defects: alignment slip, sheet folding, electrode skewing, and alignment misalignment.

However, in the existing inspection method, since the Green Bar is placed on the inspection stage and manually handled, WD (Working Distance) deviation occurs and DOF (Depth of Focus) Range difference due to the angle of the cutting surface occurs, making it difficult to secure a uniform image. However, it is necessary to improve the non-uniform shooting environment where cameras and inspection stages are not fixed.

In addition, the margin rate is judged with the captured image, and if it is out of the margin range, it is judged to be defective. In the existing inspection method, there is a deviation between inspectors when judging the margin rate. This is because the visual evaluation rather than the numerical evaluation is mainly used to measure the distance value between the electrodes by adjusting the threshold value based on the naked eye until it becomes clear and visually determining the highest point and the lowest point.

Therefore, in order to solve the above-mentioned problem, deep learning-based MLCC stack alignment inspection can remove noise through object recognition on images and improve the judgment of defects based on the margin rate range through learning on bad data and normal data. Research on systems and methods has become necessary.

[Prior art literature]

(Patent Document 1) Republic of Korea Patent Publication No. 10-2011-0004929 (published on January 17, 2011)

An object of the present invention is to remove noise through object recognition on an image and object recognition in a key area that requires inspection of image data taken from a semiconductor MLCC chip to improve defect determination by margin rate range through defect data learning. At least one deep learning-based core region detection model including an algorithm is used to detect in the image, segment the detected core region, and use the segmentation result to determine whether the defect is defective or not according to the standard margin rate range. It is to provide a learning-based MLCC stacked alignment inspection system and method.

For other purposes, visualization is performed for each result of core area detection, segmentation, and defect detection of the integrated defect detection unit, and through the analysis of the visualized step-by-step analysis data, the inspector is provided to determine whether to modify the data, thereby providing an error when detecting defects. It is to provide a deep learning-based MLCC stacked alignment inspection system and method that is robust to

A deep learning-based MLCC stack alignment inspection system according to an embodiment of the present invention utilizes at least one deep learning-based core area detection model for a core area requiring inspection of image data in which a stacked structure is photographed from a semiconductor MLCC chip. an integrated defect detection unit that detects defects, performs segmentation in the detected core regions, determines whether defects are present according to a standard margin rate range using the segmentation results, and generates normal/defective data to detect defects; A result analysis unit that performs visualization for each result of the core region detection, segmentation, and defect detection of the integrated defect detection unit, and provides the inspector to determine whether to modify the data through analysis of the visualized step-by-step analysis data; and a data storage for storing the normal/defective data and step-by-step analysis data.

In the system, the integrated defect detection unit performs boundary segmentation using a segmentation model that divides boundary surfaces for each class in a plurality of core regions detected by the core region detection model, and uses a segmentation result of the core region to perform electrode segmentation. The margin ratio for the straight line distance between the electrode and the electrode is measured, and if the measured margin ratio is within a preset reference margin ratio range, it is judged normal, and if it is outside the reference margin ratio range, it is characterized in that it can be judged to be defective.

The system further includes an annotation tool, which is a program that corrects labeling for step-by-step analysis data that requires modification obtained through a visualization process through the result analysis unit and result analysis.

The system further includes a self-learning unit that performs periodic learning on normal/defective data generated by the integrated defect detection unit to improve model performance.

The system may further include a bad data generation unit generating bad data necessary for learning through a model based on an adversarial generative neural network (GAN) in order to alleviate an imbalance between normal data and bad data.

In the deep learning-based MLCC stack alignment inspection method according to an embodiment of the present invention, the integrated defect detection unit provided in the deep learning-based MLCC stack alignment inspection system utilizes a key region extraction model pre-learned from semiconductor image data, extracting a region; classifying an inspection area by the integrated defect detection unit using segmentation within a core area; determining, by the integrated defect detection unit, whether the semiconductor chip is defective by comparing a margin rate in an inspection area classified by segmentation with a reference margin rate; The result analysis unit includes a step of analyzing and visualizing the result of each process when a defect is detected, generating analysis data, and providing it to determine whether or not to correct it.

In the above method, when the correction data is generated from the visualized step-by-step analysis data using the annotation tool, the result analysis unit stores the data storage together with normal data and defective data generated when determining whether or not the defect is defective for learning. contains more

In the method, the self-learning unit performs self-learning by utilizing normal data, defective data, and corrected data, improves the prediction accuracy of a model used when detecting defects, and periodically updates and reflects for optimization. include

The deep learning-based MLCC stacked alignment inspection system of the present invention removes noise through object recognition learning for images based on deep learning learning and improves defect judgment by margin rate range through defect data learning. The key area that needs to be inspected in the image data taken from the chip is detected in the image using at least one deep learning-based key area detection model including an object recognition algorithm, segmentation is performed in the detected key area, and segmentation is performed. Using the result, there is an advantage in that it is possible to accurately determine whether or not a defect is present according to the standard margin ratio range.

In addition, visualization is performed for each result of key area detection, segmentation, and defect detection, and the inspector analyzes the visualized step-by-step analysis data to determine whether to modify the data, so that it is robust against errors when detecting defects. there is.

In addition, since the existing model used for defect detection is a model learned with a small amount of data, when a certain amount of model results, generated defect data, and modified data are gathered, self-learning is performed periodically so that defect detection is better, It has the advantage of achieving performance improvement and variable optimization of prediction models (key region extraction model, segmentation model, defect detection model, etc.).

1 is a block diagram showing the configuration of a deep learning-based MLCC stack alignment inspection system according to an embodiment of the present invention.

FIG. 2 is a diagram showing detailed functions of the integrated defect detection unit of FIG. 1 .

3 is a view showing detailed functions of the result analysis unit of FIG. 1 .

4 is a diagram showing detailed functions of the annotation tool of FIG. 1 .

5 is a diagram showing detailed functions of the data storage of FIG. 1 .

6 is a diagram showing detailed functions of the self-learning unit of FIG. 1 .

FIG. 7 is a diagram showing detailed functions of the defective data generating unit of FIG. 1 .

8 is a flowchart of a deep learning-based MLCC stack alignment inspection method according to an embodiment of the present invention.

9 is a view showing the entire process from conventional MLCC lamination to inspection.

10 is a diagram showing general MLCC inspection types.

11 is a diagram for explaining an example of margin rate determination in the present invention.

Hereinafter, specific embodiments of the present invention will be described in detail with reference to the drawings. However, the spirit of the present invention is not limited to the presented embodiments, and those skilled in the art who understand the spirit of the present invention may add, change, delete, etc. other elements within the scope of the same spirit, through other degenerative inventions or the present invention. Other embodiments included within the scope of the inventive idea can be easily proposed, but it will also be said to be included within the scope of the inventive concept. In addition, components having the same function within the scope of the same idea appearing in the drawings of each embodiment are described using the same reference numerals.

The deep learning-based MLCC stacking alignment inspection system 1000 of the present invention is a system for inspecting MLCC stacking defects, and as shown in FIG. 1 to perform specific functions, an integrated defect detection unit 100, a result analysis unit ( 200), a data storage 300, an annotation tool 400, a self-learning unit 500, and a bad data generator 600.

As shown in FIG. 2 , the integrated defect detection unit 100 performs a key region detection function, a segmentation function, and a defect detection function that require inspection of image data captured from a semiconductor MLCC chip step by step to detect defects. can do.

First, the core region detection function detects a core region in an image by using at least one core region detection model capable of determining whether semiconductor image data is defective.

The model used here can be applied to any detection model based on deep learning, including object recognition algorithms, and the detection result of the model can be the box coordinates of the predicted core area. In addition, the key area may be an image area taken for the MLCC stacked structure that needs to be inspected.

Here, the object recognition algorithm or deep learning-based detection model may be CNN (Convolutional Neural Networks) and a Transformer algorithm, where the CNN algorithm is a neural network algorithm that recognizes an object through a convolution operation, and the Transformer algorithm converts an image into a sequence form. , and is an algorithm capable of detecting global importance using multi-head attention.

In addition, in the present invention, an object may be recognized using an R-CNN (Region-based CNN) algorithm or a DETR (Detection Transformer) using a deep learning object recognition method to excel in detecting an object position.

The R-CNN algorithm is a neural network algorithm that first creates a candidate region and trains CNN based on it to find the location of an object in an image. It includes the process of converting each candidate region to the same size, extracting features through CNN, and classifying the location and class of objects in the candidate region by inputting the extracted features as an input to the fully-connected layer.

At this time, since the position of the candidate region is not accurate, finally, the object region box position and class can be accurately corrected through the regression learning module and the class classification module.

In addition, the Detection Transformer algorithm first configures images in the form of a sequence and passes them through the Transformer Encoder to create an embedding that well contains the information of the image, additionally configures the Transformer Decoder, and converts the resulting embedding into the Transformer Encoder. It includes the process of accurately classifying the location of the actual object and the class of the object by reusing it in the decoder.

In addition, in the process of creating an embedding that well contains image information in the encoder, it is possible to capture high-quality features of an image by using multi-head attention, which can help learning by checking the importance between sequences. In the process of classifying the position and category of an object, unlike the existing Non-Maximum Suppression (NMS) method, it is possible to accurately correct the position and class of an object by minimizing duplication by using the Hungarian algorithm and binary matching. may be

In addition, other neural network algorithms include Fast R-CNN, R-FCN, YOLO (You only Look Once), and SSD (Single Shot MultiBox Detector). In addition, the object recognition speed may be improved by applying an algorithm corresponding to 1-stage among the above-mentioned neural network algorithms.

Furthermore, algorithms such as AdaBoost, Support Vector Machine (SVM), Linear Discriminant Analysis (LDA), and Principal Component Analysis (PCA) are added to increase the object recognition rate. All of these algorithm techniques identify the region to be recognized based on the appearance, and use a model trained by a set of video images to be used for training to object (e.g., MLCC stacked structure corresponding to the core region). Detects the surrounding area, and since various surrounding constraints are overcome through training, as a result, it is robust against image noise (defective pixels, etc.), and object recognition accuracy and reliability can be increased.

In addition, the segmentation function may perform boundary segmentation by utilizing a segmentation model that divides boundary surfaces for each class in a plurality of core regions detected by the core region detection model. That is, here, the boundary surface may correspond to one or more inspection areas requiring defect detection within the core area.

In addition, any algorithm-based or deep-learning-based segmentation model can be applied to the segmentation model, and the detection result of this segmentation model can be a class of each pixel existing in the predicted core region.

In addition, the integrated defect detection unit 100 may perform a defect detection function of measuring a margin rate for a straight line distance between electrodes using a segmentation result of a core region.

The reference margin ratio for comparing the measured margin ratio can be set individually for each semiconductor image data. Taking the set reference margin ratio as the reference point, if the measured margin ratio range is within the standard margin ratio range, it can be judged normal, and if it is outside the standard margin ratio range, it can be judged as defective. there will be Here, the reference margin ratio linear distance range may be, for example, the distance between each electrode pattern in the MLCC stacked structure.

More specifically, referring to FIG. 11 as an example, the margin ratio determination is performed by dividing the distance between the electrodes (margin section) in the printed pattern into 4 parts with the captured image data, and the criterion that the electrode corresponds to the margin part If it exceeds the margin rate of 1/4 (25%) (green circled area), it is judged to be defective, and if it is within 1/4, it is judged to be good.

Referring to FIG. 3 , the result analysis unit may perform a function of analyzing and converting data so that the result of the integrated defect detection unit 100 may be visually visualized after preprocessing.

In addition, the result analysis unit may select data (existing, normal, defective, and modified data) in the data storage 300 to perform result visualization.

In addition, the result analysis unit provides the result visualization data to the inspector so that the visualized step-by-step results can be used to determine whether or not the result predicted by the model is modified.

That is, the result analysis unit performs visualization for each result of the core region detection, segmentation, and defect detection, which are each functional step of the integrated defect detection unit 100, and the inspector directly checks the result, the analysis data for each stage, to determine whether correction is necessary. is to decide

Referring to FIG. 5 , the data storage 300 serves as a database in which normal data and defective data calculated through the integrated defect detection unit 100 are stored and managed along with existing data.

In addition, the data storage 300 may additionally store modified data modified through the annotation tool 400 to be described later.

In addition, here, various data stored in the data storage 300 is composed of an image file (eg, *.jpg, *.png, etc.) and a labeling file corresponding to the image, and the result analysis visualized by the result analysis unit 200 is analyzed. Data and bad data can be used for self-learning through the self-learning unit 500 to be described later.

In addition, the existing data is data pre-learned by the self-learning unit 500, and may be normal/defective data generated by the integrated defect detection unit 100 in the past, and when normal/defective data does not initially exist For prior learning, initial normal data of the MLCC chip model at the time of shipment and defective data randomly generated by the defective data generating unit 600 may be used.

Referring to FIG. 4, the annotation tool 400 performs re-labeling by correcting the labeling for step-by-step analysis data that requires correction obtained through the visualization process through the result analysis unit 200 and the result analysis. It is a tool (program tool).

At this time, labeling supports two functions: key area and segmentation, and through the web-based annotation tool for labeling correction (400), step-by-step analysis data that needs correction is corrected and relabeled, and the corrected data is stored in data storage (300). It can be saved and used for self-learning. In addition, such an annotation tool 400 may be installed in the form of a program in a manager terminal possessed by an inspector or in the result analysis unit 200 .

Since it is difficult for the self-learning unit 500 to collect semiconductor data, referring to FIG. 6, since the existing model is a model learned with little data, when a certain amount of model results, generated defective data, and corrected data are gathered, defect detection becomes more difficult. Learning may be periodically performed to improve the performance of the above-described predictive models (key region extraction model, segmentation model, defect detection model, etc.)

In addition, the self-learning unit 500 provides a web-based deep-learning-based learning service so that non-experts can use deep-learning algorithm-based self-learning, and values necessary for learning, such as selection of existing models and setting parameters, are stored on the web. It can be set through the base UI (User Interface) so that self-learning proceeds, the model is automatically updated when learning is finished, and it is set as the basic learning model.

In addition, the self-learning unit 500 may receive initial data for prior learning that is a standard for each model (core region extraction model, segmentation model, defect detection model, etc.) for each step used in the integrated defect detection unit and perform preliminary learning. As described above, the initial data may be initial normal data of the MLCC chip model at the time of shipment and defective data randomly generated by the defective data generator 600.

Referring to FIG. 7 , the defective data generating unit 600 may generate defective data in order to alleviate the imbalance since there is an imbalance between normal and defective data since the generation rate of defective data is very low in an actual semiconductor process.

Specifically, the bad data generator 600 may extract normal data from the data storage 300 and generate bad data through a model based on an adversarial generative neural network (GAN), and the generated bad data is stored in the data storage 300. It is stored as defective data and used for learning.

Here, the adversarial generative neural network (GAN) is a structure composed of several deep neural networks, unlike conventional deep learning networks, and requires dozens of times more computation than conventional deep neural network models to generate high-resolution images, but is excellent for image restoration and generation. performance can be provided.

In addition, the adversarial generative neural network is an unsupervised learning-based generative model that adversarially trains two networks with a generator and a discriminator. Input data is input to the generator and a fake image similar to the real image (in the present invention) of bad data images).

A noise value may be input as the input data. Noise values can follow any probability distribution. For example, it may be data generated with a zero-mean Gaussian.

The discriminator can be trained to discriminate between the real image and the fake image generated by the generator. More specifically, it is possible to learn to have a high probability when a real image is input, and to have a low probability when a fake image is input. That is, the discriminator can gradually learn to discriminate between real images and fake images.

First, the integrated defect detection unit 100 provided in the deep learning-based MLCC stack alignment inspection system 1000 may extract a core region by using a pre-learned core region extraction model from semiconductor image data (S100).

Next, the integrated defect detection unit 100 may classify the inspection area using segmentation within the core area (S102).

In addition, the integrated defect detection unit 100 may determine whether the semiconductor chip (MLCC chip) is defective by comparing the margin rate in the inspection area classified by the segmentation with the reference margin rate (S104).

In addition, the result analysis unit 200 analyzes and visualizes the results of each process when a defect is detected, generates analysis data, and may be provided to an inspector to determine whether or not to correct (S106).

In addition, the result analysis unit 200 uses the annotation tool 400 when correction data is generated from visualized step-by-step analysis data, normal data and defective data generated when determining whether or not the data storage 300 is defective for learning Save together (S108).

The self-learning unit 500 performs self-learning by utilizing normal, defective data, and corrected data, improves the prediction accuracy of a model used when detecting defects, and periodically updates and reflects for optimization (S110).

As a result, the more accumulated existing data (past data), normal/defective data, and correction data, the better the performance of the model through learning, thereby improving the prediction accuracy and optimizing the variables to ultimately improve the defect detection performance. .

Claims

At least one core region detection model based on deep learning is used to detect the core region requiring inspection of the image data of the stacked structure taken from the semiconductor MLCC chip, segmentation is performed in the detected core region, and the segmentation result is used. an integrated defect detection unit that determines whether or not a defect is present according to a standard margin rate range and generates normal/defective data so that defect detection is performed;

A result analysis unit that performs visualization for each result of the core region detection, segmentation, and defect detection of the integrated defect detection unit, and provides the inspector to determine whether to modify the data through analysis of the visualized step-by-step analysis data; and

Data storage that stores the normal/defective data and analysis data for each step

Deep learning-based MLCC stacked alignment inspection system including.
The method of claim 1,

The integrated defect detection unit

Performing boundary segmentation using a segmentation model that divides boundary surfaces for each class in a plurality of core regions detected by the core region detection model;

Using the segmentation result of the core area, the margin rate for the straight line distance between the electrode patterns is measured, and the measured margin rate is within the preset standard margin rate range. Deep learning-based MLCC stacked alignment inspection system, characterized in that.
The method of claim 2,

An annotation tool, which is a program that corrects the labeling of the analyzed data obtained through the result analysis through the visualization process through the result analysis unit and requires correction at each stage.

Deep learning-based MLCC stacked alignment inspection system further comprising a.
The method of claim 2,

A self-learning unit that performs periodic learning to improve model performance on the normal/defective data generated by the integrated defect detection unit.

Deep learning-based MLCC stacked alignment inspection system further comprising a.
The method of claim 4,

Bad data generation unit that generates bad data necessary for learning through a model based on adversarial generative neural network (GAN) to alleviate the imbalance between normal data and bad data.

Deep learning-based MLCC stacked alignment inspection system further comprising a.
Extracting a core region using a pre-learned core region extraction model from the semiconductor image data by an integrated defect detection unit provided in the deep learning-based MLCC stack alignment inspection system;

classifying an inspection area by the integrated defect detection unit using segmentation within a core area;

determining, by the integrated defect detection unit, whether the semiconductor chip is defective by comparing a margin rate in an inspection area classified by segmentation with a reference margin rate;

The result analysis unit analyzes and visualizes the results of each process when a defect is detected, generates analysis data, and provides it to determine whether or not to correct it.

Deep learning-based MLCC stacked alignment inspection method including.
The method of claim 6,

When the correction data is generated from the visualized step-by-step analysis data using the annotation tool, the result analysis unit stores the normal data and defective data generated when determining whether the data is defective or not in the data storage for learning.

Deep learning-based MLCC stacked alignment inspection method further comprising a.
The method of claim 7,

The self-learning unit performs self-learning using normal data, defective data, and corrected data, improves the prediction accuracy of the model used when detecting defects, and periodically updates and reflects them for optimization.

Deep learning-based MLCC stacked alignment inspection method further comprising a.