CN116705642A

CN116705642A - Method and system for detecting silver plating defect of semiconductor lead frame and electronic equipment

Info

Publication number: CN116705642A
Application number: CN202310961040.1A
Authority: CN
Inventors: 邓万宇; 陈琳; 介佳豪
Original assignee: Xian University of Posts and Telecommunications
Current assignee: Xian University of Posts and Telecommunications
Priority date: 2023-08-02
Filing date: 2023-08-02
Publication date: 2023-09-05
Anticipated expiration: 2043-08-02
Also published as: CN116705642B

Abstract

The invention discloses a method and a system for detecting silver plating defects of a semiconductor lead frame and electronic equipment, and relates to the technical field of data processing. According to the method for detecting the silver plating defect of the semiconductor lead frame, provided by the invention, after the source image of the semiconductor lead frame finished product is obtained, gray level image is obtained through gray level treatment, then the gray level image is input into the image reconstruction module for image restoration, so that the restored gray level image is obtained, then the gray level image and the restored gray level image are compared, a defect detection result is obtained, and further, the detection accuracy is improved, and meanwhile, the detection operation resource is reduced.

Description

Method and system for detecting silver plating defect of semiconductor lead frame and electronic equipment

Technical Field

The present invention relates to the field of data processing technologies, and in particular, to a method and a system for detecting silver plating defects of a semiconductor lead frame, and an electronic device.

Background

The lead frame is used as a carrier of an integrated circuit (which belongs to the field of semiconductors and is not repeated), and is a key structural member for realizing the electric connection between an internal circuit leading-out end of the chip and an external lead by means of bonding materials (gold wires, aluminum wires and copper wires) to form an electric loop, and plays a role of a bridge connected with an external lead, and most of semiconductor integrated blocks need to use the lead frame, so that the lead frame is an important basic material in the electronic information industry. Secondly, as a packaging base, its quality directly affects the life of the finished product, thereby affecting downstream fields such as electric appliances, automotive electronics, etc. Therefore, high quality lead frames are of great importance for advancing the development of integrated circuits and electronics technologies.

The existing main flow technical route for detecting the semiconductor lead frame is divided into two main types, namely a defect detection method based on morphology in the traditional sense and a defect detection method based on a convolutional neural network, which is hopeful to become the main flow technical route in the future industry. The detection method based on morphology is to search a corresponding template image without defects, and locate the defects in a comparison mode. The premise of this method is how to obtain a defect-free image. Such templates are usually drawn by hand by an operator, but are still feasible for simple textures, but often have undesirable effects for complex textures. Another method is to implement detection through a convolutional neural network, and generally reconstruct an image containing a defect into an image without the defect, so as to indirectly obtain a corresponding template image. In this way, a series of drawbacks of the conventional method can be solved, such as: the problem of template failure caused by rotation, the problem of influencing detection efficiency caused by drawing level, the problem of poor user experience caused by long drawing time, and the like. However, the convolutional neural network for detecting the semiconductor lead frame adopted at present has the defects of long training time, large occupied operation resource and the like.

Disclosure of Invention

The invention aims to provide a method and a system for detecting silver plating defects of a semiconductor lead frame and electronic equipment, which can improve detection accuracy and reduce detection operation resources.

In order to achieve the above object, the present invention provides the following solutions:

a silver plating defect detection method of a semiconductor lead frame comprises the following steps:

acquiring a source image of a semiconductor lead frame finished product;

graying the source image to obtain a gray image;

acquiring an image reconstruction model; the image reconstruction model is a trained neural network based on threshold convolution;

inputting the gray level image into the image reconstruction model to obtain a repaired gray level image;

and comparing the gray level image with the repaired gray level image to obtain a defect detection result.

Optionally, the method further comprises:

and generating a task list number by taking the timestamp as a main key, and storing the defect detection result, a storage path of a source image, a semiconductor lead frame type and operation information.

Optionally, the neural network based on the threshold convolution is a U-Net network in which the convolution mode is replaced by a dynamic threshold convolution mode.

Optionally, each layer of the encoder in the neural network based on threshold convolution comprises a convolution layer, a batch normalization layer and an activation layer; the first four layers of the decoder in the neural network based on the threshold convolution comprise a deconvolution layer, a batch normalization layer and an activation layer; the last layer of the decoder consists of a deconvolution layer and an activation function; the first layer of the encoder and the fourth layer of the decoder are provided with jump connections; the second layer of the encoder and the third layer of the decoder are provided with jump connection; the third layer of the encoder and the second layer of the decoder are provided with jump connections; the fourth layer of the encoder and the first layer of the decoder are provided with a jump connection.

Optionally, the loss function of the neural network based on threshold convolution is formed by an L1 loss function, a perceptual loss function, a style loss function and a variant loss function; the L1 loss function comprises a defect part loss function and a non-defect part loss function; the style loss function includes: a stylistic output loss function and a stylistic input loss function.

Optionally, the loss function of the neural network based on threshold convolution is:

the method comprises the steps of carrying out a first treatment on the surface of the In (1) the->For a neural network based on threshold convolution, a loss function +.>For non-defective part loss function, < >>For defective part loss function->For the perceptual loss function +.>For style output loss function, +.>Input loss function for style, +.>Is a variable loss function.

Optionally, training the neural network based on threshold convolution based on the defect data set using a static threshold defect detection method.

Optionally, the defect data set is generated based on an automatic defect generation mode of image morphology.

According to the specific embodiment provided by the invention, the invention discloses the following technical effects:

according to the method for detecting the silver plating defect of the semiconductor lead frame, provided by the invention, after the source image of the semiconductor lead frame finished product is obtained, gray level image is obtained through gray level treatment, then the gray level image is input into the image reconstruction module for image restoration, so that the restored gray level image is obtained, then the gray level image and the restored gray level image are compared, a defect detection result is obtained, and further, the detection accuracy is improved, and meanwhile, the detection operation resource is reduced.

Further, the invention also provides the following implementation structure:

a semiconductor leadframe silver plating defect detection system, comprising:

the image acquisition equipment is used for acquiring a source image of the semiconductor lead frame finished product;

the processing terminal is connected with the image acquisition equipment, is implanted with the semiconductor lead frame silver plating defect detection method, is used for obtaining a defect detection result based on the source image, is used for constructing a database, and is used for carrying out visual processing on the defect detection result and data in the database; the database is used for generating a task list number by taking the timestamp as a main key and storing the defect detection result, a storage path of a source image, the type of a semiconductor lead frame and operation information;

and the display terminal is connected with the processing terminal and used for displaying the data after the visualization processing.

An electronic device, comprising:

a memory for storing a computer program;

and the processor is connected with the memory and is used for calling and executing the computer program so as to implement the method for detecting the silver plating defect of the semiconductor lead frame.

The technical effects achieved by the two implementation structures provided by the invention are the same as those of the silver plating defect detection method for the semiconductor lead frame provided by the invention, so that the description is omitted here.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a flowchart of a method for detecting silver plating defects of a semiconductor lead frame provided by the invention;

fig. 2 is a schematic plan view of a lead frame according to the present invention;

fig. 3 is a schematic architecture diagram of an image capturing device according to the present invention;

FIG. 4 is a schematic diagram of a lightweight U-Net network model provided by the invention;

FIG. 5 is a schematic diagram of a conventional convolution scheme provided by the present invention;

FIG. 6 is a schematic diagram of an updated convolution pattern provided by the present invention;

FIG. 7 is a diagram of dynamic mask updating according to the present invention;

fig. 8 is a detection flow chart provided by the present invention.

Symbol description:

1-industrial camera mounting position, 2-electron microscope mounting position, 3-electron microscope lens, 4-telecentric lens, 5-marble Dan Longmen, 6-Z axis, 7-X axis, 8-Y axis and 9-objective table.

Detailed Description

The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

In order that the above-recited objects, features and advantages of the present invention will become more readily apparent, a more particular description of the invention will be rendered by reference to the appended drawings and appended detailed description.

Example 1

As shown in fig. 1, the method for detecting silver plating defects of a semiconductor lead frame provided in this embodiment includes:

step 100: a source image of a finished semiconductor leadframe is acquired.

Prior to this step, the semiconductor lead frame needs to be photographed. In the practical application process, firstly, a motion matrix is planned according to the input of the overall size of the material to be detected by a user, and a plurality of slightly overlapped sub-graphs which are not repeated are obtained. The resolution of these sub-images is 9344 by 7000 pixels, the actual size is about 1.96 by 1.49 cm, and the pixel size is about 2.1 microns. As shown in fig. 2, the semiconductor lead frame is formed by arranging a plurality of base island units according to a combination mode, and the size of each base island unit is absolutely smaller than half of the actual photographing size of the camera, so that planning can be performed according to the number of base island units and the whole width when planning a motion matrix. For example, assume that the overall width of the semiconductor lead frame is(unit: cm), the actual width of the industrial camera acquisition is +.>(unit: cm), the number of pictures to be photographed in the horizontal direction is +.>：. Let the height of the semiconductor lead frame be +.>(units: cm), industrial camera acquisitionThe actual height of the set is +.>(unit: cm), the number of pictures to be photographed in the vertical direction is +.>：. Therefore, the total number of pictures to be photographed by the camera is +.>：/>. When the image acquisition equipment shown in fig. 3 is adopted for photographing, the sliding table and the industrial camera lens generate relative displacement so as to achieve the purpose of moving photographing. In fig. 3, no. 1 is an industrial camera mounting position, no. 2 is an electron microscope mounting position, no. 3 is an electron microscope lens, no. 4 is a telecentric lens, no. 5 is a marble gantry, no. 6 is a Z axis, no. 7 is an X axis, no. 8 is a Y axis, and No. 9 is a stage.

Based on the above description, specific photographing steps are as follows:

(1) the method comprises the following steps Before starting collection, positioning the leftmost hollowed-out edge line through morphological corrosion expansion operation, and cutting out the protective edge of the base island unit. Then, the left side of the camera view is overlapped with the left side of the initial island unit through the sliding table, and the position at the moment is defined as a (0, 0) point.

(2) The method comprises the following steps Through planning of a motion matrix and conversion of a driver pulse signal. Setting a photographing position of the sliding table comparator. The X-axis is defined as negative to the left and positive to the right. The Y-axis is forward in a positive direction and backward in a negative direction.

(3) The method comprises the following steps The shooting adopts a fly shooting mode, the sliding table moves along the X-axis negative direction, and when the sliding table moves in place, the camera is triggered to shoot. At this time, the materials can generate relative motion on the object stage, so the image acquisition sequence is in the positive X direction. The acquired images are (0, 0), (0, 1) … in sequence…（0，）。

(4) The method comprises the following steps After the image of one row is collected, the sliding table moves to the next row along the positive direction of the Y axis, and at the moment, the sliding table moves along the positive direction of the X axis until the image of the current row is collected. The acquired images are in turn (1,）、（1，/>）……（1，0）。

(5) the method comprises the following steps And obtaining a source image of the finished semiconductor lead frame product according to the movement mode until the image acquisition is completed.

Step 101: and carrying out graying treatment on the source image to obtain a gray image.

Step 102: an image reconstruction model is acquired. The image reconstruction model is a trained neural network based on threshold convolution. The image reconstruction model adopts a lightweight U-Net network model, the size of the network model is only 0.323M, and the specific network structure is shown in figure 4. The encoder section comprises five layers, each layer of the encoder comprising three sub-layers, a convolutional layer, a batch normalization layer and an active layer, using convolutional kernels of sizes 7, 5, 3, respectively. The first four layers of the decoder comprise three sub-layers, namely a deconvolution layer, a batch normalization layer and an activation layer, and the last layer of the decoder consists of the deconvolution layer and the activation function. The first layer of the encoder and the fourth layer of the decoder, the second layer of the encoder and the third layer of the decoder, the third layer of the encoder and the second layer of the decoder, the fourth layer of the encoder and the first layer of the decoder are respectively provided with jump connection.

Further, the loss function of the neural network based on the threshold convolution is formed of an L1 loss function, a perceptual loss function, a style loss function, and a variant loss function. Wherein the L1 loss function includes a defective portion loss function and a non-defective portion loss function. The loss functions of the defective portion and the non-defective portion are as follows:

。

in the method, in the process of the invention,is a binary image with the same size as the network input image, and takes a value of 0 or 1 to respectively represent a defective area and a non-defective area. />Representing an output image of the network. />Representing an input image of the network. Since the fraction of defective portions is too small, the following mixing loss function will lose 8 as defective portions to non-defective portions: 1, and mixing the components in a proportion of 1.

The perceptual loss function is:

. In the formula, N represents the total number of pixels included in the image. The subscript n denotes the current pixel point. />Representing an output image of the network. />Input image representing network, ++>Representing an output image +.>For non-defective parts of (B)>Is replaced with a non-defective portion, the defective portion is maintainedIs unchanged. />Representing the intermediate feature layer in the pretrained VGG-16 model, where +.>、/>、/>Three layers. Thus, the per loss measures the L1 penalty of the feature layer. This loss appears visually as whether the reconstructed image of the defective area is significantly natural.

The style loss function includes: the style output loss function and the style input loss function are as follows:

。

in the method, in the process of the invention,representation->Matrix of->The result is a +.>Is mathematically called a Gram matrix (GramMatrix），/>For normalizing coefficient->This loss is visually represented as whether the reconstructed image contour perimeter is highlighted.

The loss of variability function is:。

the effect of the metamorphic loss function on the visual appearance is whether the image is smooth or not, which can be understood as a smoothing penalty loss. Where i represents the number of rows, j represents the number of columns, and P represents the current set of image pixels. Front item representationRight-shifted by one pixel, the latter term representing +.>Downshifting by one pixel and then combining the shifted output with + ->The loss of L1 is calculated, so that the restored image is ensured not to be too large in pixel value in the vertical direction, the visual effect is ensured, and the reconstructed texture characteristic information is ensured as large as possible.

Based on the above description, the final loss function of the neural network based on the threshold convolution is:

Further, after the model structure is constructed, a model of the structure needs to be trained.

The semiconductor lead frame image is generally composed of three parts of silver, copper, hollowed-out parts and the like. The brightest and whitish part is silver, and the pixel value corresponding to the pixel point is the largest. The darkest part is a hollowed-out part, and the reconstructed image is the part which is least concerned about whether the reconstructed image is correct or not. The remaining portion between the brightest and darkest is copper. The data is not continuous when image reconstruction is performed. The hollowed-out part is not concerned about whether defects exist or not in practice, and meanwhile, the hollowed-out part does not exist, so that a part of parameters are used for memorizing the pixel values of the hollowed-out part in network training, and the hollowed-out part is inappropriate, unreasonable and meaningless.

Based on the above, in order to improve the accuracy of the network and reduce the burden of the network, a static threshold defect detection method is provided, and a static Mask (static-Mask) with the same size and channel as the input image is introduced. The mask is like a canvas, the hollowed-out area is shielded, namely, invalidation operation is carried out, the pixel value of the covered area is set to be 0, and the other areas are not shielded, so that the original pixel value information is kept. Since the input Image (Masked Image) and the reconstructed Image (Reconstruction Image) are unique in that the reconstructed Image eliminates defects, restores the information that the real Image should have, and does not have any change. Because ofThe positional information contained in the introduced static mask is also applicable to the output image. Assume that the original image input into the network isThe image covered by the mask is +.>The coverage process has the following calculation formula:

。

in the method, in the process of the invention,representation dot->Masked pixel values. />Representation dot->The pixel values before masking are not present. />Representation dot->Gray values at that point. Since it is luminance information provided by the backlight, it is relatively stable, and a fixed constant can be used as a threshold.

Furthermore, the defect data is generated by adopting an automatic defect generation mode based on image morphology, so that the operations of manual addition, collection, management and the like are replaced, the degree of manual intervention is greatly reduced on the premise of ensuring the effect, precision and accuracy, and the automation level is improved.

Wherein OpenCV is an open-sourced third party visual library. Are now widely used in image processing. OpenCV offers a lot of convenience through autonomous packaging, plus its own continuous optimization perfection. The line function in the library is mainly used for line drawing operation, and the line drawing function can set the thickness degree of lines, so that scratch defects under different forces can be simulated. The circle function and the ellipse function in the library are used for drawing circles and ellipses respectively, can be used for simulating real hole defects, and can simulate defects such as silver deficiency, multi-silver deficiency, copper deficiency, multi-copper deficiency and the like according to different drawing positions. As for defects such as silver formation and silver staining, which often occur at the junction of copper and silver, the boundary area between copper and silver can be limited by corrosion expansion and non-operation, and irregular defects such as silver formation and silver staining can be simulated by means of opening holes in thick lines. Based on this, the specific steps for generating defect data are as follows:

(1) the method comprises the following steps Generating a defect label according to the method to simulate the information of defect positions, shape contours and the like, wherein the size and the thickness of the defect are required to meet randomness.

(2) The method comprises the following steps And generating a plurality of random defects according to a random generation mode, wherein the defects can cover most defects such as scratches, silver melting, oxidation, stains, silver leakage, silver deficiency and the like.

(3) The method comprises the following steps The image is roughly segmented, copper and silver are separated, and preparation is made for accurate addition of defects.

(4) The method comprises the following steps And adding defects according to the mapping of the label on the original image, and if the color of silver is added on copper, adding the color of copper on silver, and adding a mode of adding no color in a hollowed-out area.

Furthermore, the traditional convolution method is greatly influenced by the defect part, and the problems of inaccurate data characteristic extraction, inaccurate defect part reconstruction and the like in the traditional convolution method can be solved by introducing dynamic threshold convolution, so that feasibility support is provided for complete coverage and high accuracy of detection of defect type detection.

Specifically, the main calculation task is undertaken in the image coding stage, so that the main decision of the training time consumption and accuracy of the network is realized. Conventional convolution patterns fig. 5 shows the mathematical expression as follows:

。

in the method, in the process of the invention,representing the corresponding value of the convolution kernel (filter). />Feature information representing an input feature map. />Feature information representing the output feature map.

In fig. 5, a portion having a pixel value of 50 is a corresponding defective portion, and it is not difficult to see from surrounding pixel values, and the pixel value of the portion is likely to be 8. In a normal convolution process, if the area indicated by the grey-white box is not defective, the value obtained by the convolution should be 99, but the convolution is directly performed due to the defect, and the value of the portion is 225, so that the feature extraction is inaccurate. Although this problem is solved by normalization, it wastes a lot of time, so that the network convergence speed is greatly reduced.

As can be seen from fig. 4, the input of the lightweight U-Net network model has two parameters, the first parameter is the input image, which is a single-channel gray-scale image, and the size is 512×512. The second parameter is a dynamic mask, which is a single-channel binary image, and the size is 512×512. It should be noted that the input image is a gray image with defects, and the Dynamic Mask (Dynamic Mask) is a corresponding defect map, i.e. the Dynamic Mask is a single-channel binary map with 255 pixels except for the pixel value of the defect position. The dynamic threshold convolution is equivalent to a switch, the effective area corresponds to an "open" gate, i.e. the calculation is participated, and the ineffective area corresponds to an "close" gate, i.e. the calculation is not participated. Therefore, the input data is limited by a dynamic threshold, and the expression is as follows:

。

in the method, in the process of the invention,representing the original pixel point +.>Representing the covered pixel values, +.>Representing a dynamic mask binary map.

By setting the original defect location to 0 in the limited input feature, this will result in a smaller convolution result containing that portion, thus introducing a scale factorScale factor->Is defined as follows:

。

where 1 represents an all 1 matrix of the same size as the filter.Representing a dynamic mask sub-matrix with the same size as the filter by taking the current pixel point as the center, wherein the value of the dynamic mask sub-matrix is correspondingly derived from the dynamic mask matrix. By a scale factor->To adjust the problem of smaller convolution results near the current defect. The conventional convolution formula is updated as follows, and the updated convolution manner is shown in fig. 6.

。

In the method, in the process of the invention,representing the current pixel point +.>Is calculated by the computer. M, N represent the width and height of the convolution kernel, respectively. When s=0, y=0 represents the upper left-most corner position of the convolution kernel. />Representing the submatrices at the corresponding positions of the dynamic mask.

After the primary convolution is completed, the dynamic mask is updated, and the updated dynamic mask is shown in fig. 7, and the update expression is as follows:

。

where M, N represent the width and height of the convolution kernel, respectively.Representing the matrix to which the convolution kernel corresponds.Represents M _D (s, t) represents the submatrices at the corresponding positions of the dynamic mask.

Output of dynamic mask with increasing number of layersThe pixels with 0 in the method gradually become smaller, namely the effective area is more and more, the effective area is also more and more, the influence of the dynamic mask on the whole is less and less, and finally the original convolution mode is returned, however, the threshold convolution can also better extract the characteristics, so that the convergence speed of the network and the accuracy of reconstruction are improved.

Compared with the traditional reconstruction method based on pixel points, the reconstruction method based on the threshold convolution neural network provided by the invention solves the problem that the data of the reconstructed part is unreal, solves the problem that the convolution result deviates from the true result too much due to the defect part in the convolution process, and enhances the accuracy of forward transmission in network training, thereby improving the reconstruction effect on the irregular defect part and enabling the reconstruction effect to be closer to the true image.

In addition, compared with the traditional convolutional neural network, the neural network based on the threshold convolution has better adaptability to the defect type of irregular shapes. For the practical problem that most of defect types in industry are irregular shapes, the invention has stronger adaptability than the traditional method. Because the data set collected by the industrial camera mainly takes positive samples, and meanwhile, the operation flow is simplified as much as possible in practical application, and the frequent operation on the local data set is avoided, the complexity of the operation can be greatly reduced by adopting the method for automatically generating the mask, the label and the defects.

Notably, the input data of the dynamic threshold convolution network has a parameter of dynamic masking in addition to the feature image. The dynamic mask k indicates the location of the defect in a certain sense. However, in practical application, defect information of the object to be detected does not exist, and if so, the practical significance of detection is lost. In practice, the detection mode is that only a defect image is needed to reconstruct, and no label information is needed. To solve this problem, the dynamic mask with defect information and the dynamic mask without defect information (which is a full 1 matrix) should be combined in the experiment according to 1:2, so that the network can reconstruct correct images of defect parts on the premise of not knowing defect position information while improving the capability of repairing irregular defects on the premise of knowing defect information. In summary, only a defect image is required to be input during detection, and the tag information automatically generates a full 1 matrix to meet the input of a network, so that defect detection can be performed.

Step 103: and inputting the gray level image into an image reconstruction model to obtain a repaired gray level image.

Step 104: and comparing the gray level image with the repaired gray level image to obtain a defect detection result. If the defect detection result is that the defect exists, the accurate position of the defect can be correspondingly given out in the defect detection result.

In the practical application process, in order to prevent the influence of dust or ultrafine defects, the defects can be filtered through threshold segmentation and connected domain areas, wherein the threshold segmentation and the connected domain segmentation are respectively as follows:

。

in the method, in the process of the invention,and representing the residual image of the reconstructed image and the original image. At->Pixel values at the points. />Representing the residual threshold. />Representing connected domains found by findcounters functions in OpenCV.Representing the current connected domain area. />Representing the connected domain area threshold. />Indicating that the current connected domain does not account for the defect set. />Indicating that the current connected domain counts into the defect set. />Representing the current connected domain flag.

Further, after the defect detection result is obtained in step 104, the method further includes: and determining the timestamp as a main key, generating a unique task list number by the bit detection result, and storing the detection result, a storage path of a source image, an operator, a lead frame type and other information into a database so as to conveniently retrieve and review the historical defect detection result.

Example 2

This embodiment provides a semiconductor leadframe silver plating defect detection system. The system comprises:

and the image acquisition equipment is used for acquiring the source image of the semiconductor lead frame finished product.

The processing terminal is connected with the image acquisition equipment, and is implanted with the semiconductor lead frame silver plating defect detection method provided by the embodiment 1, and is used for obtaining a defect detection result based on a source image, constructing a database and carrying out visual processing on the defect detection result and data in the database. The database is used for generating a task list number by taking the timestamp as a main key and storing defect detection results, a storage path of a source image, a semiconductor lead frame type and operation information.

And the display terminal is connected with the processing terminal and used for displaying the visualized data.

Based on the above structure, the detection process of the silver plating defect detection system for the semiconductor lead frame provided by the embodiment is as follows:

step 1: the method comprises the steps of acquiring a source image of a semiconductor lead frame finished product by using an image acquisition device, acquiring the source image of the semiconductor lead frame finished product by using the image acquisition device, and transmitting the acquired source image to a terminal through a tera-network cable.

Step 2: the detection software of the processing terminal reads the source image and starts to execute the detection flow.

Step 3: and entering an image preprocessing stage of the detection flow, and automatically carrying out graying processing on the acquired image by the processing terminal to obtain a gray image.

Step 4: and (3) entering an image reconstruction stage of the detection flow, and sending the gray level image containing the defects in the step (3) into a trained network to obtain a repaired gray level image.

Step 5: and (3) entering a defect positioning stage of the detection flow, and comparing the gray level image obtained in the step (3) with the gray level image obtained in the step (4) to obtain the accurate position of the defect.

Step 6: the detection result is visually displayed through a multifunctional detection result interface of the detection software by adopting a display, so that a user can intuitively and rapidly know the detection result, and the user is supported to carry out local secondary judgment.

Step 7: after the defect detection result is obtained in the step 6, the processing terminal determines the timestamp as a primary key, the bit detection result generates a unique task list number, and information such as the defect detection result, a storage path of a source image, an operator, a lead frame type and the like is stored in a database.

Step 8: the user can query the history detection task stored in the database according to the condition in the "history task" function module of the software, thereby reviewing the detection result of the history.

Example 3

As shown in fig. 8, the following steps are performed in one detection flow:

step 1: the detection is initiated by clicking on the device software "detect" button. The image acquisition platform acquires the material source image, and the acquired source image has the characteristic of slight overlapping but non-repeated. The image acquisition architecture shown in fig. 3 is equipped with X, Y, Z triaxial motion to meet the requirements of horizontal photographing and material focusing. The main parameter table of the apparatus is shown in table 1. The main technical parameters are shown in table 2.

Step 2: the collected data is transmitted into a central processing unit through a tera-mega network cable, and is changed into a gray image through graying processing. And (5) locally searching whether the material with the model is the material with the model appearing for the first time or not by taking the name of the material model as a main key. If so, a short training is performed, the process usually takes 10 minutes, and a trained network is obtained after the training is finished. If not, the network model parameters are directly read from the local.

Step 3: since the images acquired by the camera are not 512×512, the images need to be cut into a predetermined size in a certain order and then sent to a network for image reconstruction, and the defect part is automatically repaired into a defect-free image after the reconstruction. This process typically takes 11ms to process a 512 x 512 size image.

Step 4: and (3) comparing the reconstructed image obtained in the step (3) with the source image, obtaining residual scores, performing first filtering through threshold segmentation, and performing secondary judgment on the defects through manually set defect sizes if the residual scores pass, and finally displaying the defects on a software interface. If the user has objection to the defect, the user can conduct the re-judgment through a microscope carried by the equipment on site, the process is extremely convenient, and the re-judgment can be achieved only by clicking the defect image.

TABLE 1

Article name	Brand model	Model number	Remarks
				Intelligent vision detection system	Custom-made type	Custom-made type
GPU industrial control computer	Custom-made type	Custom-made type	Parallel image high-performance processing apparatus
				Display device	Dell/philips/samsung/others	24 inches	Double screen
Transport control	SOLID/Lei Sai		0.508
				Multidimensional exercise combination	The XY precision of the linear motor is 0.003mm. Z precision 0.01mm		Precisely ground marble
Equipment system control software	Custom-made type	Custom-made type
				Servo system	Pine/upper silver/high trauma
Sensor for detecting a position of a body	pine/Adenoki/other
				Electrical appliance	Schneider/OMRON
Cylinder	SMC/AirTAC
				Sliding rail	HTWIN/THK
Clamp/stage	Custom-made type

TABLE 2

Parameter name	Specification of specification	Unit (B)	Remarks
				External dimension of the device	1066×1466×1780	mm	(Length. Times. Width. Times. Height) reference only
Total mass of	900	KG	For reference only
				Capacity of equipment	240-360	s	Equipment operating time
Maximum size of product inspection	500×650	mm
				Motor speed	1-100	mm/s
Pixel size	5	um (micron)
				Leak rate	<1%
False detection rate	<1%
				Air pressure requirement	4-6	Kg/cm ²
Power supply	AC220V 50/60Hz
				Power of	1.5	KW
Device development cycle	90-120	Tiantian (Chinese character of 'Tian')

Further, to test the performance of the system, 8 different models of lead frames are tested. Let it now be assumed that the lead frames are numbered 1, 2, 3, 4, 5, 6, 7, 8. Firstly, the number of defects contained in the current model is obtained manually, the number is sent to a manual detection center for detection to obtain detection time and accuracy, and then the number is detected by equipment to obtain the detection time and accuracy. The above procedure was repeated until all inspection of 8 different models of lead frames was completed. The results are shown in Table 3.

Furthermore, since the defect detection of the present invention is essentially image reconstruction, the quality of image reconstruction directly affects the result of defect detection. Therefore, the invention is evaluated from two indexes of structural similarity and peak signal-to-noise ratio with the traditional method, the result is shown in table 4, the original network is UNet, and the network of the invention is named SM-UNet.

TABLE 3 Table 3

Lead frame numbering	Number of actual defects	The device detects the number of defects	The number of defects is detected manually	Equipment time-consuming(s)	Time-consuming manual work(s)	Accuracy of equipment	Manual accuracy rate
								1	17	17	16	184	201	100%	94.12%
2	21	20	18	188	205	95.23%	85.71%
								3	8	8	8	180	198	100%	100%
4	13	13	12	182	197	100%	92.30%
								5	18	18	16	195	202	100%	88.89%
6	22	22	21	191	200	100%	95.45%
								7	15	15	12	202	198	100%	80%
8	5	5	5	192	204	100%	100%

TABLE 4 Table 4

Lead frame numbering	Network name	PSNR（dB）↑	SSIM(%)↑	Model size (MB) ≡	Training time(s) ∈
						1	UNet	31.2473	96.24	0.323	535.9556
1	SM-UNet	31.8223	98.87	0.323	541.2448
						2	UNet	31.2473	96.24	0.323	555.9556
2	SM-UNet	31.8223	98.87	0.323	561.2448
						3	UNet	28.4659	91.26	0.323	545.9556
3	SM-UNet	29.8916	96.35	0.323	549.2448
						4	UNet	27.9765	93.27	0.323	562.4721
4	SM-UNet	29.2272	95.49	0.323	567.0915
						5	UNet	27.7108	92.18	0.323	544.4185
5	SM-UNet	30.4189	94.18	0.323	549.8529
						6	UNet	29.4074	88.48	0.323	537.3208
6	SM-UNet	30.5106	93.15	0.323	539.7391
						7	UNet	27.0922	89.75	0.323	557.2085
7	SM-UNet	29.3104	94.31	0.323	561.4766
						8	UNet	28.8245	92.85	0.323	571.0185
8	SM-UNet	29.7098	93.08	0.323	573.7419

Wherein, the larger the ∈value, the better. The smaller the value is, the better is. PSNR is inferior below 20dB, 20-30dB is generally good (difference is perceived by vision) of 30-40dB, 40dB-50dB is excellent (difference is hardly perceived by vision), and more than 50dB is excellent. The SSIM values range from 0% to 100% to describe the degree of structural similarity, 0% indicating no structural similarity, 100% indicating identical images.

Based on the above description, the present invention has the following advantages over the prior art:

1) The object-oriented defect detection method is used for detecting defects in industry, and can complete training and participate in production in a short time by means of a small parameter quantity.

Experimental results show that for one type of material, only 10 minutes of training time is needed to participate in production. In addition, only 11.5ms is needed for reconstructing an image with the size of 512×512, so that the requirements of high real-time performance and high accuracy in industry are greatly met. The invention supports dual GPU detection, and further greatly enhances the real-time performance.

2) The invention omits manual update of the defect library by the user, and solves the problem that no related defects exist during initial training. The user only needs click detection, so that the automation level of the equipment is greatly improved.

3) The method solves the problem of poor use effect caused by inaccurate feature extraction from the root of the algorithm. By adopting the method of convolution of the static threshold and the dynamic threshold and combining the loss function, the network convergence speed is greatly improved, and the reconstructed image is more accurate, so that the characteristic of high accuracy of detection is realized.

4) After the detection is finished, the specific position of the defect can be positioned through the left index, and the secondary judgment of the local microscope is carried out by combining the right defect image operation equipment, so that the transfer of a sample is omitted, and convenience is provided for the repeated judgment.

5) The invention can reproduce the historical task through the historical task module, so that the invention has better memory effect on the new defect type and can facilitate the debugging of the assembly line.

6) The invention supports the filtering of defect results, and the concerned defects can be screened out according to the attention degree of the user to the defects, thereby greatly improving the detection efficiency.

Example 4

This embodiment provides an electronic device. The electronic device includes:

and a memory for storing a computer program.

And the processor is connected with the memory and is used for retrieving and executing a computer program to implement the method for detecting the silver plating defect of the semiconductor lead frame.

Furthermore, the computer program in the above-described memory may be stored in a computer-readable storage medium when it is implemented in the form of a software functional unit and sold or used as a separate product. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method of the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a U disk, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk.

In the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, and identical and similar parts between the embodiments are all enough to refer to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section.

The principles and embodiments of the present invention have been described herein with reference to specific examples, the description of which is intended only to assist in understanding the methods of the present invention and the core ideas thereof; also, it is within the scope of the present invention to be modified by those of ordinary skill in the art in light of the present teachings. In view of the foregoing, this description should not be construed as limiting the invention.

Claims

1. A method for detecting silver plating defects of a semiconductor lead frame, comprising:

acquiring a source image of a semiconductor lead frame finished product;

graying the source image to obtain a gray image;

2. The method for detecting silver plating defects of a semiconductor lead frame according to claim 1, further comprising:

3. The method for detecting silver plating defects of a semiconductor lead frame according to claim 1, wherein the neural network based on threshold convolution is a U-Net network in which a convolution mode is replaced by a dynamic threshold convolution.

4. A semiconductor lead frame silver plating defect detection method according to claim 3, wherein each layer of the encoder in the threshold convolution based neural network comprises a convolution layer, a batch normalization layer and an activation layer; the first four layers of the decoder in the neural network based on the threshold convolution comprise a deconvolution layer, a batch normalization layer and an activation layer; the last layer of the decoder consists of a deconvolution layer and an activation function; the first layer of the encoder and the fourth layer of the decoder are provided with jump connections; the second layer of the encoder and the third layer of the decoder are provided with jump connection; the third layer of the encoder and the second layer of the decoder are provided with jump connections; the fourth layer of the encoder and the first layer of the decoder are provided with a jump connection.

5. The method for detecting silver plating defects of a semiconductor lead frame according to claim 4, wherein the loss function of the neural network based on threshold convolution is formed of an L1 loss function, a perceptual loss function, a style loss function, and a variant loss function; the L1 loss function comprises a defect part loss function and a non-defect part loss function; the style loss function includes: a stylistic output loss function and a stylistic input loss function.

6. The method for detecting silver plating defects of a semiconductor lead frame according to claim 5, wherein the loss function of the neural network based on threshold convolution is:

the method comprises the steps of carrying out a first treatment on the surface of the In (1) the->For nerves based on threshold convolutionLoss function of network, ">For non-defective part loss function, < >>For defective part loss function->For the perceptual loss function +.>For style output loss function, +.>Input loss function for style, +.>Is a variable loss function.

7. The method for detecting silver plating defects on a semiconductor leadframe according to claim 6, wherein the threshold convolution based neural network is trained based on a defect dataset using a static threshold defect detection method.

8. The method for detecting silver plating defects on a semiconductor lead frame according to claim 7, wherein the defect data set is generated based on an automatic defect generation method of image morphology.

9. A semiconductor leadframe silver plating defect detection system, comprising:

a processing terminal connected with the image acquisition equipment and implanted with the method for detecting the silver plating defect of the semiconductor lead frame according to any one of claims 1-8, wherein the processing terminal is used for obtaining a defect detection result based on the source image, constructing a database and carrying out visual processing on the defect detection result and data in the database; the database is used for generating a task list number by taking the timestamp as a main key and storing the defect detection result, a storage path of a source image, the type of a semiconductor lead frame and operation information;

10. An electronic device, comprising:

a memory for storing a computer program;

a processor, coupled to the memory, for retrieving and executing the computer program to implement the semiconductor leadframe silver plating defect detection method as recited in any one of claims 1-8.