CN113920437B

CN113920437B - Conductive particle identification method, system, storage medium and computer equipment

Info

Publication number: CN113920437B
Application number: CN202111518830.XA
Authority: CN
Inventors: 雷雨; 贾可; 万迪文; 易国锋; 徐行; 沈复民
Original assignee: Chengdu Koala Youran Technology Co ltd
Current assignee: Chengdu Koala Youran Technology Co ltd
Priority date: 2021-12-14
Filing date: 2021-12-14
Publication date: 2022-04-12
Anticipated expiration: 2041-12-14
Also published as: CN113920437A

Abstract

The invention relates to the field of industrial intelligent quality inspection, in particular to a conductive particle identification method, a conductive particle identification system, a storage medium and computer equipment. The conductive particle identification method can efficiently detect and identify the conductive particles in the IC area; acquiring a conductive particle image of an IC area of a liquid crystal panel, and automatically calibrating the central position of a conductive particle point by a template matching method to obtain a first data set; and further preprocessing the data set by carrying out affine transformation and image scale reduction on the first data set to improve the calibration quality of the data set, designing a deep convolution model as a visual encoder, training a network model by a classification loss function and a correction loss function, and selecting the network model with the highest score as a test candidate model by index evaluation to realize accurate statistics of the number of conductive particles and accurate evaluation of the quality of the liquid crystal panel.

Description

Conductive particle identification method, system, storage medium and computer equipment

Technical Field

The invention relates to the field of industrial intelligent quality inspection, in particular to a conductive particle identification method, a conductive particle identification system, a storage medium and computer equipment.

Background

In the industrial environment, the detection to the liquid crystal display panel is mostly manual detection, the detection result has strong subjectivity and low detection efficiency, and generally only sampling detection can not meet the detection requirement of industrial production to the product quality of the liquid crystal display panel. In order to meet the requirement of quality detection in industrial production, the quality detection of industrial production based on deep learning also exists in the prior art. For the deep learning method, the calibration quality of the data and the data volume of the data set are important for the training of the model. The collected images of the IC area of the liquid crystal panel are generally low in visibility of conductive particles, irregular in shape and randomized in distribution; secondly, the amount of the acquired sample data is small, and the risk of model under-fitting exists. And finally, the hydration operation and the automatic quality inspection process in industrial production have high requirements on the real-time performance of model detection.

Disclosure of Invention

In order to solve the problems in the background art, the present invention provides a method for identifying conductive particles, including:

acquiring a data set, and calibrating the center position of the conductive particles by using a template matching algorithm to obtain a first data set;

performing first preprocessing on the first data set to obtain a second data set;

based on the second data set, carrying out visual coding on data in the second data set to obtain coding characteristics, and carrying out second preprocessing on the coding characteristics to obtain a prediction result;

training a network model through a classification loss function and a correction loss function respectively based on the prediction result;

and evaluating the network model, screening an optimal network model, and identifying the conductive particles based on the optimal network model.

Further, the acquiring the data set, calibrating the center position of the conductive particle by using a template matching algorithm, and obtaining the first data set specifically comprises:

acquiring a data set, wherein the data set comprises a plurality of original pictures;

acquiring a pixel value of an original picture, selecting a region where conductive particles are located in the original picture, and acquiring a mean value and a variance of the pixel value of the region;

and inputting the mean value and the variance of the pixel values of the region, and calibrating the central position of each conductive particle in the region based on a standard correlation coefficient matching method to obtain a first data set.

Further, the inputting of the mean and variance of the pixel values of the region, and based on a standard correlation coefficient matching method, calibrating the center position of each conductive particle in the region to obtain a first data set specifically includes:

inputting the mean value and the variance of the pixel values of the region, and inputting the pixel value of the original picture;

calculating a normalized value of the original picture in the region based on a standard correlation coefficient matching method;

calculating a normalized value of a sample to be transformed in the data set based on a standard correlation coefficient matching method;

calculating an affine transformation matrix based on the normalization value of the original picture in the region and the normalization value of a sample to be transformed in the data set;

and taking the coordinate position of the maximum similarity value in the affine matrix as the central position of the conductive particles in the area to obtain a first data set.

Further, the performing the first preprocessing on the first data set to obtain the second data set specifically includes:

and carrying out translation processing, rotation processing, miscut processing and scaling processing on the pictures of the first data set to obtain a second data set.

Further, the data in the second data set is visually encoded based on the second data set to obtain an encoding characteristic, and the encoding characteristic is subjected to second preprocessing to obtain a prediction result specifically;

inputting the pictures in the second data set into a visual encoder to obtain encoding characteristics;

and performing maximum pooling and 2D convolutional network processing on the coding characteristics to obtain a prediction result.

Further, the training of the network model by the classification loss function and the modification loss function based on the prediction result specifically includes:

disassembling and shunting the prediction result to obtain two parts of prediction results;

selecting a part of prediction results to carry out classification prediction to obtain classification prediction values, and training the classification prediction values to obtain a classification loss function;

selecting another part of prediction results to carry out regression prediction to obtain a regression prediction value, and training the regression prediction value to obtain a correction loss function;

the network model is trained based on the classification loss function and the modification loss function.

Further, the evaluating the network model, screening an optimal network model, and identifying the conductive particles based on the optimal network model specifically includes:

evaluating the training network model based on the accuracy prediction, the recall rate and the F1-score;

selecting the training network model with the highest F1-score as the optimal network model;

and identifying the conductive particles in the picture to be detected based on the optimal network model.

A conductive particle identification system, the system comprising:

a central position calibration module: the method comprises the steps of obtaining a data set, and calibrating the center position of conductive particles by using a template matching algorithm to obtain a first data set;

the first data preprocessing module: the first preprocessing is carried out on the first data set to obtain a second data set;

the second data preprocessing module: the data processing device is used for carrying out visual coding on the data in the second data set based on the second data set to obtain coding characteristics, and carrying out second preprocessing on the coding characteristics to obtain a prediction result;

a network model training module: the network model is trained through a classification loss function and a correction loss function respectively based on the prediction result;

a network model evaluation module: and the system is used for evaluating the network model, screening the optimal network model and identifying the conductive particles based on the optimal network model.

A storage medium having stored thereon a computer program which, when executed by a processor, implements the method of any of the above.

A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the method of any one of the above when executing the program.

Has the advantages that: the invention provides a conductive particle identification method, which can efficiently detect and identify conductive particles in an IC area; acquiring a conductive particle image of an IC area of a liquid crystal panel, and automatically calibrating the central position of a conductive particle point by a template matching method to obtain a first data set; and further preprocessing the data set by carrying out affine transformation and image scale reduction on the first data set to improve the calibration quality of the data set, designing a deep convolution model as a visual encoder, training a network model by a classification loss function and a correction loss function, and selecting the network model with the highest score as a test candidate model by index evaluation to realize accurate statistics of the number of conductive particles and accurate evaluation of the quality of the liquid crystal panel.

Drawings

In order to illustrate the embodiments of the present invention more clearly, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings can be obtained by those skilled in the art without inventive effort.

Fig. 1 is a schematic flow chart of a conductive particle identification method according to an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

According to the conductive particle identification method, the computer equipment comprises a processor, a memory, a network interface and a database which are connected through a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the computer device is used for storing data of the classification method. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a conductive particle identification method based on template matching and a deep convolutional network.

The embodiment of the application provides a conductive particle identification method, a conductive particle identification system, computer equipment and a storage medium, and aims to solve the problems that in the prior art, the calibration quality of data and the data volume of a data set are poor.

The following describes in detail the technical solutions of the present application and how the technical solutions of the present application solve the above technical problems by embodiments and with reference to the drawings. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments. It should be noted that, according to the conductive particle identification method provided in the embodiments of the present application, the execution subject may also be a classification apparatus, where the apparatus may be implemented as part of or all of the classification method through software, hardware, or a combination of software and hardware.

In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments.

In one embodiment, as shown in FIG. 1, step S1: acquiring a data set, and calibrating the center position of the conductive particles by using a template matching algorithm to obtain a first data set;

recording the acquired dataset as dataset D, wherein

Which represents the i-th original picture,

representing the area marked with the conductive particles in the ith original picture. F samples d are selected, and pictures are taken

The delimited region where the k relatively clear conductive particle points are located

J =1, …, k. The size of the defined area is preferably 5 x 5, and experiments prove that the F1-score test score is highest when the defined area is preferably 5 x 5, and the defined area can be adjusted according to actual conditions to the area

Performing maximum and minimum normalization processing to obtain the region

Mean value of pixel values

Sum variance

。

Inputting the mean value

And the variance

Pearson similarity calculation is performed based on open-cv, using the standard correlation coefficient matching (TM _ CCOEFF _ NORMED) method provided by it, using the pixel values of the original picture at coordinates (x, y)

Subtracting the input mean value

Then, the variance is divided by

Further obtain the picture

Of (2) a

Normalizing the result

. Specifically, the standard correlation coefficient matches the formula,

（1）

wherein the content of the first and second substances,

representing the result of normalization of a standard image taken from the region, w being the picture length, h being the picture width,

is the average of the pixel values of the original picture,

the standard deviation of the pixel value of the original picture is obtained;

（2）

wherein the content of the first and second substances,

representing samples of pictures to be transformed in a data set D

The result of the normalization process, w is the picture length, h is the picture width,

the pixel value of the picture sample to be transformed at (x, y),

is the average value of the picture samples to be changed,

the standard deviation of the picture sample to be changed is obtained;

（3）

is a matrix of radial transformations.

According to affine matrix

Taking the coordinate position of the maximum value of similarity in the affine matrix

These coordinates are taken as the true center position of each conductive particle point as the center position of the t-th position in the i-th sample. We denote the recalibrated data set as the first data set

Which is

。

Step S2: performing first preprocessing on the first data set to obtain a second data set;

for training set picture

Obtaining an enhanced data set by a data enhancement method

。

The method comprises the following specific steps: input gray scale picture is recorded as

For the gray scale picture

Respectively perform translation

Operation, as shown in equation (4):

（4）

wherein the content of the first and second substances,

in

Is the translational component of the x-axis in two dimensions;

the translation component of the y axis of the two-dimensional space is selected, and the value is determined by uniform sampling distributed according to self-defined 0-1;

rotate

Operation, as shown in equation (5):

（5）

wherein, in the matrix

Is the angle of rotation;

miscut cut

Operation, as shown in equation (6):

（6）

wherein the content of the first and second substances,

in

And

is the coordinate position of the source map rotation center,

is a miscut angle;

is a miscut control variable;

the above hyper-parameters

Etc. are all custom presets.

Step S21: and carrying out image-scale reduction preprocessing operation on the enhanced data set, and further representing the center position of the conductive particles through the region.

On the basis, in order to reduce the prediction difficulty, the regression position prediction problem is converted into a classification problem, and the original pixel level regression problem is converted into 0-1 classification prediction of the existence of conductive particles on an image region by reducing the scale of an original image. Specifically, for the region with conductive particles, we take

Otherwise

As a label; secondly, we calculate the offset error of the center position of the conductive particle point after scaling as the value to be corrected

. To summarize, we enhance the data set

For each picture size in the enhanced data set, scaling down

The scaling is performed to scale down the size to the original part size, preferably thirty-half, since in this size the F1-score is highest, and likewise a scaling to one sixteenth or one sixteenth of the original image size, etc. may be selected, by which the second data set is obtained

The definition is as follows:

step S3: based on the second data set, carrying out visual coding on data in the second data set to obtain coding characteristics, and carrying out second preprocessing on the coding characteristics to obtain a prediction result;

constructing a visual encoder

Inputting a second data set

To-be-detected picture in

Obtaining coding features

And respectively performing classification prediction and regression prediction by using the coding characteristics q output by the model F.

Designing a visual encoder

The visual encoder

Is composed of four layers of convolution modules, each layer of convolution module includes oneA multi-channel 2D convolution network, a BatchNorm regularization process and a LeakyReLU, the convolution module being defined as

I =1,2,3,4 represents the i-th layer module of the encoder F,

is an input feature of the ith layer;

the picture to be detected

And (4) coding in a visual coder of a 4-layer convolution module through 4 cycles, and outputting a final characteristic q.

And processing the coding characteristics q through maximum pooling and a 2D convolutional network, and outputting a prediction result H of a 3-channel.

Step S4: training a network model through a classification loss function and a correction loss function respectively based on the prediction result;

through the parameters of a loss function learning model, firstly, the prediction result H is disassembled and shunted to obtain two parts of prediction results, one part of the prediction results is classified and predicted to predict the picture to be detected

Which regions have a distribution of conductive particle centers

(ii) a The other part, performing regression prediction to predict the deviation value of the center position

The formula is as follows:

wherein the content of the first and second substances,

representing the prediction result of the ith sample of the training set,

representing a merge operation, where the split streams are broken down

The first channel represents

The last two channels represent

。

Step S41: selecting a part of prediction results to carry out classification prediction to obtain classification prediction values, training the classification prediction values to obtain a classification loss function

，

Classification loss function

And learning by judging whether the predicted area is the classification distribution of the center points of the conductive particles and then by a cross entropy loss function with weighted binomial distribution, as shown in formula (7):

（7）

wherein the content of the first and second substances,

is the true value between the output values 0-1, from

，

Is a prediction value for the classification of the model,

the table weights the positive and negative samples, by default 0.5.

Step S42: selecting another part of prediction results to carry out regression prediction to obtain a regression prediction value, and training the regression prediction value to obtain a correction loss function;

correcting loss function

Obtaining a predicted conductive particle to center shift error value

The predicted center position is compensated and corrected by a smooth L1 loss function, as shown in equation (8):

（8）

wherein the content of the first and second substances,

from

And n represents the number of all samples in the training set.

Step S43: computing a sample set

Classification loss function of

And correcting the loss function

；

，

In order to be a hyper-parameter,

as a function of total loss based on

And calculating gradient and updating the network model through gradient feedback.

Step S5: evaluating the network model, screening an optimal network model, and identifying the conductive particles based on the optimal network model;

to evaluate the performance of the entire network, we used the following indicators: precision prediction Precision, Recall and F1-score. Setting the number of training iterations (200 times) as a model training stop condition, and selecting a model with the highest F1-score as an optimal network model, wherein the optimal network model is specifically represented by formula (9):

（9）

wherein TP is predicted to be positive and actually is positive; FP is predicted to be positive and actually negative; FN is predicted negative, actually positive; TN is predicted to be negative and actually negative.

Step S51: in the testing stage, the screened optimal model is used for preprocessing the trial picture to be tested in the steps S1-S2, and the processed picture is directly input into the optimal model for testing and evaluation.

In one embodiment, there is provided a conductive particle identification system based on template matching and a deep convolutional network, the system comprising:

The implementation principle and technical effect of the conductive particle identification system based on template matching and deep convolutional network provided by the embodiment are similar to those of the method embodiment, and are not described herein again.

In one embodiment, a computer device is provided, comprising a memory and a processor, the memory having a computer program stored therein, the processor implementing the following steps when executing the computer program:

The implementation principle and technical effect of the computer device provided by the above embodiment are similar to those of the above method embodiment, and are not described herein again.

In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of:

The implementation principle and technical effect of the computer-readable storage medium provided by the above embodiments are similar to those of the above method embodiments, and are not described herein again.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent should be defined by the appended claims.

Claims

1. A method for identifying conductive particles based on template matching and a deep convolutional network, the method comprising:

performing first preprocessing on the first data set to obtain a second data set; the first pretreatment specifically comprises the following steps: performing data enhancement processing on the first data set to obtain an enhanced data set, performing image scale reduction preprocessing operation on the enhanced data set, performing classification prediction on whether conductive particles are contained in an image area after the image scale reduction, and representing the center position of the conductive particles through the image area containing the conductive particles;

2. The method for identifying conductive particles based on template matching and deep convolutional network as claimed in claim 1, wherein the obtaining of the data set, calibrating the center position of the conductive particles by using a template matching algorithm, and obtaining the first data set specifically comprises:

3. The method according to claim 2, wherein the method for identifying conductive particles based on template matching and deep convolutional network is characterized in that the mean and variance of the pixel values of the region are input, the center position of each conductive particle in the region is calibrated based on a standard correlation coefficient matching method, and the first data set is obtained specifically as follows:

4. The method for identifying conductive particles based on template matching and deep convolutional network as claimed in claim 1, wherein the first preprocessing is performed on the first data set to obtain a second data set specifically as follows:

5. The method for identifying conductive particles based on template matching and deep convolutional network as claimed in claim 1, wherein based on the second data set, the data in the second data set is visually encoded to obtain an encoded feature, and the encoded feature is subjected to second preprocessing to obtain a prediction result;

6. The method for identifying conductive particles based on template matching and deep convolutional network as claimed in claim 1, wherein the training of the network model by the classification loss function and the modification loss function respectively based on the prediction result specifically comprises:

7. The method for identifying conductive particles based on template matching and deep convolutional network as claimed in claim 1, wherein the evaluating the network model and screening the optimal network model, and the identifying conductive particles based on the optimal network model specifically comprises:

8. A conductive particle identification system based on template matching and deep convolutional networks, the system comprising:

the first data preprocessing module: the first preprocessing is carried out on the first data set to obtain a second data set; the first pretreatment specifically comprises the following steps: performing data enhancement processing on the first data set to obtain an enhanced data set, performing image scale reduction preprocessing operation on the enhanced data set, performing classification prediction on whether conductive particles are contained in an image area after the image scale reduction, and representing the center position of the conductive particles through the image area containing the conductive particles;

9. A storage medium on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-7.

10. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method of any of claims 1-7 when executing the program.