NL2030315B1

NL2030315B1 - Computer-implemented ai method for detecting surface defects of electroluminescent semiconductor panel

Info

Publication number: NL2030315B1
Application number: NL2030315A
Authority: NL
Inventors: Chen Boyuan
Original assignee: Chen Boyuan
Priority date: 2021-09-27
Filing date: 2021-12-28
Publication date: 2023-01-13
Also published as: NL2030315A; CN113822870A

Abstract

The present invention relates to a computer-implemented AI method for detecting surface defects of an electroluminescent semiconductor panel, comprising the steps of 5 ﬁrst time of image enhancement processing, marking and transformation of image defects, random distribution of training, veriﬁcation and testing of images, second time of image enhancement processing, establishment of a target detection neural network, data training and obtaining of optimal weight parameters, and AI detection of the panel images. The present invention aims to solve the technical problem that during defect 10 detection of the surface of the EL semiconductor panel, as limited by quality and amount of image datasets and restricted by a singular preprocessing method of image data and a small network scale of a pixel algorithm, the defects are recognized slowly in low precision, and the defects cannot be classiﬁed and positioned accurately. The method is applicable to detection of semiconductor chips and panels made of Si, Ge, AsGa, SiC 15 and the like.

Description

COMPUTER- IMPLEMENTED AI METHOD FOR DETECTING SURFACE DEFECTS OF ELECTROLUMINESCENT SEMICONDUCTOR PANEL TECHNICAL FIELD

[01] The present invention relates to an Al detection method for surface defects and in particular to a computer-implemented AI method for detecting surface defects of an electroluminescent semiconductor panel and belongs to the field of image recognition and nondestructive testing.

BACKGROUND ART

[02] Electroluminescent is called EL for short. EL detection is a testing technology that detects a panel surface detects using a near-infrared image shot by an HD infrared camera and plays a crucial role in fields such as operation and maintenance of photovoltaic power stations. Recognition of characteristic information of EL image defects is a core part in a detection process.

[03] Chinese patent 201810794758.5 discloses a method for detection and defect recognition of an EL image of a solar cell piece, comprising the following steps: obtaining an EL image of a solar cell piece to be detected, positioning a grid line and conducting regional division; deleting a grid line area and re-combining the image, calculating an image gray value and conducting 2D construction; calculating an interclass dispersion matrix of a particle swarm and determining a current optimal position; updating an optimal individual of a particle population and an optimal individual of a particle history; using a chaos model to generate a new chaos variable; updating positions and speeds of all particles in the particle swarm and conducting re- calculation till a number of iterations is reached; and obtaining a defect image of the cell piece by division according to the obtained optimal position and conducting defect recognition.

[04] Chinese patent 2019111876125 discloses an EL defect detection method applied to a site of a photovoltaic power station, comprising the following steps:

preprocessing an EL image obtained of an original photovoltaic module and obtaining a corrected EL image; cutting the corrected EL image to form a cell piece with a specification of m x n; processing the obtained cell piece by graying, corrosion, expansion, binarization and pixel calculation, finding a possible EL defect according to apixel projection and conducting defect analysis.

[05] The method can realize division, classification and combined processing of an EL image and determine types of surface defects of a panel through pixel calculation. However, as limited by quality and amount of image datasets and restricted by a singular preprocessing method of image data and a small network scale of a pixel algorithm, the panel defect displayed by the EL image is recognized slowly in low precision, and different types of defects on the surface of a same panel cannot be classified and positioned accurately.

SUMMARY

[06] Aiming at defects in the prior art, the present invention aims to provide a computer-im plemented AI method for detecting surface defects of an electroluminescent semiconductor panel and expects to solve technical problems in the prior art, such as singular image data, a small network scale of a pixel algorithm, a slow recognition process, low precision and lack of accurate defect positioning. The method is applicable to detection of semiconductor chips and panels made of Si, Ge, AsGa, SiC and the like.

[07] In order to solve above technical problems, the present invention discloses a computer-im plemented AI method for detecting surface defects of an electroluminescent semiconductor panel, comprising the following steps in sequence:

[08] 1. First time of image enhancement processing

[09] (1) Inputting EL images of a semiconductor panel containing defects, and using an image cutting function to adjust sizes of the input images into square images with a pixel of 640 x 640;

[10] (2) Writing an image random flipping algorithm, setting a flipping probability of the images to be 0.5 in a horizontal direction and a vertical direction, and flipping and saving the images;

[11] (3) Writing an image random rotating algorithm, setting a step length to be 2° to 10°, setting a probability of whether rotation takes place in each rotation direction to be

0.5 so as to ensure randomness of image changing, and rotating and saving the images; and

[12] (4) Obtaining different defect images of the semiconductor panel with a pixel of 640 x 640, of which the number is in total 4 times the original number of the input images, and conducting renaming in batch.

[13] 2. Marking and transformation of image defects

[14] (1) Marking defect positions of the semiconductor panel and obtaining image information corresponding to each defect;

[15] (2) Writing a format transformation algorithm, and transforming defect information of an XML format into a tensor format, wherein a coefficient before a tensor denotes a defect type and is corresponding to a format needed by an input end of a convolutional neural network; and

[16] (3) Normalizing labels of the defect images and tensor names and sequencing the labels and tensor names in a sequence under same label names through a sequencing algorithm.

[17] 3. Random distribution of training, verification and testing of images

[18] (1) Writing a random sampling algorithm, using a random.Sample function to conduct traversal on the images arranged in sequence and the corresponding labels, and conducting sampling at a random step length according to a sampling ratio of 5-8:2-4:1; and

[19] (2) Determining a file path and saving the defect information in corresponding datasets in a tensor form.

[20] 4. Second time of image enhancement processing

[21] (1) Setting a program to read data of 4-9 pictures while normalizing image parameters each time; and

[22] (2) Randomly selecting 4-9 pictures from a training set for processing of random distribution, random scaling and random splicing.

[23] 5. Establishment of target detection neural network

[24] (1) Using a self-adapted anchor frame calculation method to output a prediction frame based on an initial anchor frame, comparing the prediction frame with a true frame groundtruth, calculating a difference thereof, then conducting reverse updating and iteration, adjusting network weight parameters, and outputting an optimal anchor frame value in the training set in a self-adapted manner in each training;

[25] (2) Inputting a preprocessed image with specifications of 3 x 640 x 640, making 4 copies thereof and using a sectioning operation to cutting the copied images into 4 sections of 3 x 320 x 320;

[26] (3) Using a concat function to connect the 4 sections in an image depth direction to obtain an output image with specifications of 12 x 320 x 320;

[27] (4) Generating an image with specifications of 32 x 320 x 320 by a convolution layer with a convolution kernel number of 32, and using a batch _borm function and a leaky relu function to input an output result into a next convolution layer;

[28] (5) Using a Resnet residual network structure in a backbone network, adding the residual structure with initial input after convolution of the convolution layer with a convolution kernel number of 32, and using a depth multiple function to control the depth of the model;

[29] (6) Dividing the original input into two branches, conducting a convolution operation for each to halve a number of channels, conducting a bottleneck network multiplication operation in an input branch 1, and using the concat function to divide output into an output branch 1 and the output branch 2 so as to equalize characteristic sizes of input and output images of a cross-stage local network;

[30] (7) Conducting up-sampling processes of deconvolution and de-pooling of a high-level characteristic image, and using a direct addition method for characteristic fusion; and using a self-adapted characteristic pooling to recover damaged information paths between each candidate area and all characteristic layers, and aggregating each candidate area on each characteristic layer to avoid arbitrary distribution; and

[31] (8) Inputting the characteristic image on an spp layer, outputting after passing a convolution layer of 1x1, conducting down-sampling by three parallel Maxpools, adding a result with initial characteristics thereof and outputting a result.

[32] 6. Data training and obtaining of optimal weight parameters

[33] (1) Inputting the pre-processed EL images of the semiconductor panel for iteration trainings;

[34] (2) Calculating a loss function value, a recall (positive types found/all positive types that should be found) value and a map (Mean Average Precision) value in each iteration and drawing a broken line statistical diagram; and

[35] (3) Obtaining a weight document of optimal parameters, conducting coverage and saving of the optimal parameters after each iteration and obtaining an optimal parameter set document which is used for AI detection of the panel images.

[36] 7. Al detection of panel images

[37] Inputting non-preprocessed EL images of the semiconductor panel, which are directly shot by a near-infrared camera, and obtaining defect recognition results containing types and position information of the defects.

[38] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[39] (1) The AI detection technology adopted in the present invention can detect whether a panel has scratches and black spot abnormities based on EL images and can position and mark defect positions.

[40] (2) According to the present invention, under very limited image datasets of the EL semiconductor panel, the defect images are processed with two times of image enhancement, so that without influences on defect positioning precision, dataset richness and recognition precision are significantly increased.

[41] (3) In the present invention, the dataset random classification algorithm is used to transform and classify the data in the tensor form after two times of the image enhancement, so that exclusiveness of the data applied to training, verification and testing is ensured. In this way, stability of the detection model is enhanced, and generalization ability of target recognition is increased.

[42] (4) In the present invention, the target detection network is established through combination of the residual neural network, the characteristic fusion method and the cross-stage local network. Besides enhancement of model complexity and expansion of the network scale, recognition precision is increased, model robustness is increased, memory consumption is reduced and recognition is quicker.

[43] (5) In the present invention, optimal weight configuration is covered and saved, so that effectiveness of the training result is ensured, and recognition precision and speed are increased.

BRIEFT DESCRIPTION OF THE DRAWINGS

[44] FIG. 1 is a diagram showing means of a classification loss function;

[45] FIG. 2 shows recall;

[46] FIG. 3 shows map (Mean Average Precision) values;

[47] FIG. 4 shows a detection result of a photovoltaic plate with scratches;

[48] FIG. 5 shows a detection result of a photovoltaic plate with black spots; and

[49] FIG. 6 shows a detection result of a photovoltaic plate with both scratches and black spots.

DETAILED DESCRIPTION OF THE EMBODIMENTS [S0] In order to make purposes of the present invention, principles of technical solutions and advantages clearer, the present invention will be further described in detail in conjunction with accompanying drawings and specific implementations. In the implementation, the specific implementation schemes described are only for explaining the present invention, rather than set limits to the present invention. [S1] Embodiment

[52] An EL technology is used to shoot photos of a surface of a SiC semiconductor panel, and then the method of the present invention is used for defect detection. Detection steps are as follows:

[S3] An EL technology is used to shoot photos of a surface of a Si semiconductor photovoltaic module, and then the method of the present invention is used for defect detection. Detection steps are as follows:

[54] 1. First time of image enhancement processing

[55] (1) Inputting 500 EL images of a semiconductor panel containing defects of black spots and scratches, and using an image cutting function to adjust sizes of the input images into square images with a pixel of 640 x 640;

[56] (2) Designing an image random flipping algorithm, setting a flipping probability of the images to be 0.5 in a horizontal direction and a vertical direction, and flipping and saving the images;

[57] (3) Designing an image random rotating algorithm, setting a step length to be 5°, setting a probability of whether rotation takes place in each rotation direction to be

[58] (4) Obtaining 2000 different defect images of black spots and scratches of the semiconductor panel with a pixel of 640 x 640, and conducting renaming in batches of 0-2000. [S9] 2. Marking and transformation of image defects

[60] (1) Marking black spot positions of the semiconductor panel and obtaining image information corresponding to each defect;

[61] (2) Designing a format transformation algorithm, and transforming defect information of an XML format into a tensor format, wherein a coefficient before a tensor denotes a defect type and is corresponding to a format needed by an input end of a convolutional neural network; and

[62] (3) Normalizing labels of the defect images and tensor names and sequencing the labels and tensor names in a sequence under same label names through a sequencing algorithm.

[63] 3. Random distribution of training, verification and testing of images

[64] (1) Designing a random sampling algorithm, using a random. Sample function to conduct traversal on the images arranged in sequence and the corresponding labels, and conducting sampling at a random step length according to a sampling ratio of 6:3:1; and

[65] (2) Determining a file path and saving the defect information in corresponding datasets in a tensor form.

[66] 4. Second time of image enhancement processing

[67] (1) Setting a program to read data of 6 pictures while normalizing image parameters each time; and

[68] (2) Randomly selecting 6 pictures from a training set for processing of random distribution, random scaling and random splicing.

[69] 5. Establishment of target detection neural network

[70] (1) Using a self-adapted anchor frame calculation method to output a prediction frame based on an initial anchor frame, comparing the prediction frame with a true frame groundtruth, calculating a difference thereof, then conducting reverse updating and iteration, adjusting network weight parameters, and outputting an optimal anchor frame value in the training set in a self-adapted manner in each training;

[71] (2) Inputting a preprocessed image with specifications of 3 x 640 x 640, making 4 copies thereof and using a sectioning operation to cutting the 4 copied images into 4 sections of 3 x 320 x 320;

[72] (3) Using a concat function to connect the 4 sections in an image depth direction to obtain an output image with specifications of 12 x 320 x 320;

[73] (4) Generating an image with specifications of 32 x 320 x 320 by a convolution layer with a convolution kernel number of 32, and using a batch borm function and a leaky relu function to input an output result into a next convolution layer,

[74] (5) Using a Resnet residual network structure in a backbone network, adding the residual structure with initial input after convolution of the convolution layer with a convolution kernel number of 32, and using a depth multiple function to control the depth of the model;

[75] (6) Dividing the original input into two branches, conducting a convolution operation for each to halve a number of channels, conducting a bottleneck network multiplication operation in an input branch 1, and using the concat function to divide output into an output branch 1 and the output branch 2 so as to equalize characteristic sizes of input and output images of a cross-stage local network;

[76] (7) Conducting up-sampling processes of deconvolution and de-pooling of a high-level characteristic image, and using a direct addition method for characteristic fusion; and using a self-adapted characteristic pooling to recover damaged information paths between each candidate area and all characteristic layers, and aggregating each candidate area on each characteristic layer to avoid arbitrary distribution; and

[77] (8) Inputting the characteristic image on an spp layer, outputting after passing a convolution layer of 1x1, conducting down-sampling by three parallel Maxpools, adding a result with initial characteristics thereof and outputting a result.

[78] 6. Data training and obtaining of optimal weight parameters

[79] (1) Inputting the pre-processed 2000 EL images of the semiconductor panel for 100 iteration trainings;

[80] (2) Calculating a loss function value, a recall (positive types found/all positive types that should be found) value and a map (Mean Average Precision) value in each iteration and drawing a broken line statistical diagram; and

[81] (3) Obtaining a weight document of optimal parameters, conducting coverage and saving of the optimal parameters after each iteration and obtaining an optimal parameter set document which is used for Al detection of the panel images.

[82] 7. Al detection of panel images

[83] Inputting non-preprocessed EL images of the semiconductor panel, which are directly shot by a near-infrared camera, and obtaining defect recognition results containing types and position information of the defects. On average, recognition of each image with the pixel size of 640 x 640 spends 0.016s.

[84] Testing effects: As shown in Fig.l to Fig.6, when the number of iterations exceeds 70, the recall (positive types found/all positive types that should be found) value and the map value reach about 99.5% and tend to be stable. The loss function value decreases to 0.02 after 100 iterations. Obviously, the model has a very excellent ability in defect recognition.

Both photovoltaic panels with separate existence of scratches and black spots and those with coexistence of the scratches and black spots can be recognized and detected, while the defects can be marked with words, so that results can be analyzed easily.

Claims

_11- Conclusions

A computer-implemented Al method for detecting surface defects of an electroluminescent semiconductor panel, characterized in that it comprises the following steps in sequence: (1) a first time of image enhancement processing; (2) highlighting and transforming image defects; (3) random distribution of training, verification and testing images; (4) a second time of image enhancement processing; (5) establishment of a neural target detection network; (6) data training and acquisition of optimal weight parameters; and (7) Al detection of panel images.

The computer-implemented AI method according to claim 1, characterized in that the first time image enhancement processing comprises the following steps in sequence: (1) inputting EL images of a semiconductor panel containing defects, and using an image clipping function to reduce dimensions of adjust the input images to square images with a pixel size of 640 x 640; (2) writing an image random flip algorithm, setting a flip probability of the images to be 0.5 in a horizontal direction and a vertical direction, and flipping and storing the images; (3) writing an image random rotation algorithm, setting a step length to be 2° — 10°, setting a probability of rotation occurring in each direction of rotation to be 0.5 to ensure randomness of image change, and rotate and save the images; and (4) acquiring several defect images of the semiconductor panel having a pixel of 640 x 640, the number of which is a total of 4 times the original number of the input images, and performing batch renaming.

The computer-implemented Al method according to claim 1, characterized in that the marking and transformation of image defects comprises the following steps in sequence: (1) marking defect positions of the semiconductor panel and acquiring image information corresponding to each defect; (2) writing a format transformation algorithm, and transforming defect information from an XML format to a tensor format, where a

-12- coefficient preceding a tensor indicates a defect type and corresponds to a format required by an input end of a convolutional neural network; and (3) normalizing labels of the defect images and tensor names and sequencing the labels and tensor names into an order under the same label names by a sequencing algorithm.

The computer-implemented AI method according to claim 1, characterized in that the random distribution of training, verification and testing images comprises the following steps in order: (1) writing a random sampling algorithm, using a random sampling function to perform looping over the images arranged in order and the corresponding labels, and perform sampling at an arbitrary step length according to a sampling ratio of 5-8:2-4:1; and (2) determining a file path and storing the defect information in corresponding data sets in a tensor form.

The computer-implemented AI method according to claim 1, characterized in that the second time of image enhancement processing comprises the following steps in sequence: (1) setting a program to read data from 4 to 9 images, meanwhile adjusting image parameters each time to normalize; and (2) randomly selecting 4 — 9 images from a training set for random distribution processing, random scaling, and random splitting.

The computer-implemented AI method according to claim 1, characterized in that setting up the neural target detection network comprises the following steps in sequence (1) using a self-adapted anchor frame calculation method to output a prediction frame based on an initial anchor frame, comparing the prediction frame with an actual frame ground truth, calculating a difference thereof, then performing reverse updating and iteration, adjusting network weight parameters, and outputting an optimal anchor frame value in the training set in a self-adjusted manner in each training; (2) entering a pre-processed

S13 - image with specifications of 3 x 640 x 640, making 4 copies of them and using a divide operation to divide the copied images into 4 parts of 3 x 320 x 320; (3) using a concatenation function to connect the 4 parts in an image depth direction to acquire an output image with specifications of 12 x 320 x 320; (4) generating an image with specifications of 32 x 320 x 320 by a convolution layer with a convolution kernel number of 32, and using a bacth_borm function and a leaky relu function to input an output result into a subsequent convolution layer; (5) using a Resnet residue network structure as a backbone network, adding the residue structure with an initial input after convolution of the convolution layer with a convolution kernel number of 32, and using a multiple depth function to control the depth of the model; (6) dividing the original input into two branches, performing a convolution operation for each to halve a number of channels, performing a bottleneck network multiplication operation in an input branch-1, and using the concatenation function to convert output into an output branch- 1 and divide the output branch-2 so as to equalize characteristic sizes of input and output images of a cross-phase local network; (7) performing upsampling processes of deconvolution and depooling of high-level features mapping, and using a direct addition method for feature fusion; and using self-adapted feature pooling to repair damaged information paths between each candidate region and all feature layers, and aggregating each candidate region on each feature layer to avoid arbitrary distribution; and (8) inputting the characteristics map on an spp layer, outputting after passing a convolution layer of Ix1, performing downsampling through three parallel Maxpools, summing a result with initial characteristics thereof, and outputting a result.

The computer-implemented AI method according to claim 1, characterized in that the data training and obtaining optimal weight parameters comprises the following steps in sequence: (1) inputting pre-processed EL images of the semiconductor panel for iteration training; (2) calculating loss function value, a callback (positive types found/all positive types found

“14 - should be found) value and a map (Mean Average Precision)) value in each iteration and drawing a statistical broken line graph; and (3) acquiring a weight document of optimal parameters, performing coverage and storing the optimal parameters after each iteration, and acquiring an optimal parameter set document used for Al detection of panel images.

The computer-implemented Al method according to claim 1, characterized in that the Al detection of panel images comprises: inputting non-pre-processed EL images of the semiconductor panel taken directly by a near-infrared camera, and acquiring defect recognition results containing types and position information of the defects.