CN117809115A

CN117809115A - SMT (surface mounted technology) patch wrong part detection method, SMT patch wrong part detection device and SMT patch wrong part detection visual detection system

Info

Publication number: CN117809115A
Application number: CN202311870517.1A
Authority: CN
Inventors: 刘梦舒; 唐永亮
Original assignee: Shenzhen Lingyun Shixun Technology Co ltd
Current assignee: Shenzhen Lingyun Shixun Technology Co ltd
Priority date: 2023-12-29
Filing date: 2023-12-29
Publication date: 2024-04-02

Abstract

The application discloses a SMT patch wrong part detection method, device and visual detection system, and belongs to the technical field of image processing. The method comprises the following steps: acquiring an SMT patch image to be detected and an SMT patch template image of a circuit board; inputting the SMT patch image to be detected and the SMT patch template image into an countermeasure network classification model to obtain a wrong piece detection result of the circuit board output by the countermeasure network classification model; the antagonism network classification model is obtained based on sample image set training, the sample image set comprises a plurality of first sample image pairs and a plurality of second sample image pairs, the first sample image pairs comprise first wrong-piece defect sample images and first template sample images, the second sample image pairs comprise second wrong-piece defect sample images and second template sample images, the first wrong-piece defect sample images are acquired by an image acquisition device, and the second wrong-piece defect sample images are generated based on patch element images.

Description

SMT (surface mounted technology) patch wrong part detection method, SMT patch wrong part detection device and SMT patch wrong part detection visual detection system

Technical Field

The application belongs to the technical field of image processing, and particularly relates to a SMT patch wrong piece detection method, device and visual detection system.

Background

Surface mount technology (Surface Mounted Technology, SMT) is one of the most popular technologies and techniques in the electronics assembly industry, and SMT has the advantages of high density packaging, high quality connection, and the like. The circuit board assembled based on the SMT technology has the defects of wrong parts, namely, the components are installed at wrong positions on the circuit board, the probability of the occurrence of the defects of wrong parts is low, however, once the defects of wrong parts can seriously affect the quality of products, and therefore, the defects of wrong parts of the circuit board need to be detected before the circuit board is applied to the products.

At present, two types of commonly used circuit board wrong part defect methods exist, namely, a manual visual detection method is adopted, namely, human eyes are utilized to detect the wrong part defect, and the method depends on the familiarity degree of each person on a detection object, so that the detection accuracy is low and the efficiency is low; another category is Automated Optical Inspection (AOI), which typically uses conventional computer vision techniques such as image processing and feature extraction, often requires a manually designed feature extractor to extract features, and is relatively poorly adapted in scenes where the component types are complex.

Disclosure of Invention

The present application aims to solve at least one of the technical problems existing in the prior art. Therefore, the application provides an SMT patch wrong part detection method, device and visual detection system, which can obtain more accurate wrong part detection results, and has the advantages of high detection efficiency, simple operation and low cost.

In a first aspect, the present application provides a method for detecting an SMT patch error component, where the method includes:

acquiring an SMT patch image to be detected and an SMT patch template image of a circuit board;

inputting the SMT patch image to be detected and the SMT patch template image to an countermeasure network classification model to obtain a wrong piece detection result of the circuit board output by the countermeasure network classification model;

the countermeasure network classification model is trained based on a sample image set, the sample image set comprises a plurality of first sample image pairs and a plurality of second sample image pairs, the first sample image pairs comprise first wrong-piece defect sample images and first template sample images corresponding to the first wrong-piece defect sample images, the second sample image pairs comprise second wrong-piece defect sample images and second template sample images corresponding to the second wrong-piece defect sample images, the first wrong-piece defect sample images are acquired by an image acquisition device, and the second wrong-piece defect sample images are generated based on patch element images.

According to the SMT patch error detection method, the countermeasure network classification model is trained through the sample image set, the sample image set comprises a first error defect sample image acquired by equipment and a second error defect sample image generated based on patch element images, sample data are rich, SMT patch error detection is carried out based on the trained countermeasure network classification model, the obtained detection result is more accurate, the detection efficiency is high, the whole process is realized based on image processing, the detection result is obtained based on the difference between the SMT patch image to be detected and the SMT patch template image, a configuration file is not required to be set, the operation is simple, and the cost can be reduced.

According to one embodiment of the application, the second sample image pair is obtained by:

acquiring a patch element image set, wherein the patch element image set comprises a plurality of patch element images, and the element types of the patch element images are different;

replacing element areas in the second template sample image with patch element images with different element types based on the patch element image set, and generating a plurality of second error piece defect sample images;

and obtaining a plurality of second sample image pairs based on the second error piece defect sample images and the second template sample images.

According to an embodiment of the present application, before the replacing the component area in the second template sample image with the patch component image having a different component type, the method further includes:

and performing size transformation on the patch element image based on the size of the element region in the second template sample image.

According to an embodiment of the present application, the countermeasure network classification model includes a feature extraction layer, a feature fusion layer and a classifier that are sequentially connected, the feature extraction layer includes a first feature extraction module and a second feature extraction module, the SMT patch image to be detected and the SMT patch template image are input to the countermeasure network classification model, and a false part detection result of the circuit board output by the countermeasure network classification model is obtained, including:

Inputting the SMT patch image to be detected to the first feature extraction module, and inputting the SMT patch template image to the second feature extraction module to obtain a first image feature output by the first feature extraction module and a second image feature output by the second feature extraction module;

inputting the first image features and the second image features into the feature fusion layer to obtain fusion image features output by the feature fusion layer;

and inputting the fused image features into the classifier to obtain a wrong detection result of the circuit board output by the classifier.

According to one embodiment of the application, the first feature extraction module and the second feature extraction module are constructed based on a MobileNetV2 network.

According to one embodiment of the application, the challenge network classification model is trained by:

acquiring the sample image set;

performing data enhancement processing on the sample image set;

training and updating the countermeasure network classification model to be trained according to the sample image set after data enhancement until iteration is stopped, and obtaining the trained countermeasure network classification model.

According to one embodiment of the present application, the performing data enhancement processing on the sample image set includes:

at least one of turning, rotating, scaling and deforming the false part defect sample image of the sample image set;

and/or performing color conversion processing on the error piece defect sample image of the sample image set.

According to one embodiment of the application, the training and updating the challenge network classification model to be trained according to the data-enhanced sample image set includes:

inputting the sample image set with the enhanced data to the countermeasure network classification model to be trained;

calculating a cross entropy loss function and a dynamic scaling cross entropy loss function based on the error defect labels of the sample image set and the error defect information predicted by the countermeasure network classification model;

and updating model parameters of the countermeasure network classification model based on the cross entropy loss function and the dynamic scaling cross entropy loss function.

In a second aspect, the present application provides an SMT patch error detection device, the device comprising:

the acquisition module is used for acquiring an SMT patch image to be detected and an SMT patch template image of the circuit board;

The processing module is used for inputting the SMT patch image to be detected and the SMT patch template image to an countermeasure network classification model to obtain a wrong piece detection result of the circuit board, which is output by the countermeasure network classification model;

According to the SMT patch wrong part detection device, training is carried out on the countermeasure network classification model through the sample image set, the sample image set comprises a first wrong part defect sample image acquired by equipment and a second wrong part defect sample image generated based on patch element images, sample data are rich, SMT patch wrong part detection is carried out on the basis of the countermeasure network classification model after training, the obtained detection result is more accurate, the detection efficiency is high, the whole process is realized on the basis of image processing, the detection result is obtained on the basis of the difference between the SMT patch image to be detected and the SMT patch template image, a configuration file is not required to be set, the operation is simple, and the cost can be reduced.

In a third aspect, the present application provides a visual inspection system comprising:

the image acquisition device is used for acquiring an SMT patch image to be detected and an SMT patch template image;

the data processing device is electrically connected with the image acquisition device and is used for executing the SMT patch error detection method according to the first aspect.

According to the visual detection system, the countermeasure network classification model is trained through the sample image set, the sample image set comprises a first misplaced piece defect sample image acquired by equipment and a second misplaced piece defect sample image generated based on the patch element image, sample data are rich, SMT patch misplaced piece detection is carried out based on the trained countermeasure network classification model, the obtained detection result is more accurate, the detection efficiency is high, the whole process is realized based on image processing, the detection result is obtained based on the difference between the SMT patch image to be detected and the SMT patch template image, a configuration file is not required to be set, the operation is simple, and the cost can be reduced.

In a fourth aspect, the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, where the processor implements the SMT patch error component detection method according to the first aspect when executing the computer program.

In a fifth aspect, the present application provides a non-transitory computer readable storage medium, on which a computer program is stored, which when executed by a processor, implements the SMT patch error detection method according to the first aspect described above.

In a sixth aspect, the present application provides a computer program product comprising a computer program which, when executed by a processor, implements the SMT patch error component detection method as described in the first aspect above.

Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the application.

Drawings

The foregoing and/or additional aspects and advantages of the present application will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings, wherein:

fig. 1 is a schematic flow chart of an SMT patch error detection method according to an embodiment of the present application;

FIG. 2 is a second flow chart of the SMT chip error detection method according to the embodiment of the present application;

FIG. 3 is a third flow chart of the SMT patch error detection method according to the embodiment of the present application;

fig. 4 is a schematic structural diagram of an SMT patch error detection device according to an embodiment of the present application;

Fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

Technical solutions in the embodiments of the present application will be clearly described below with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application are within the scope of the protection of the present application.

The terms first, second and the like in the description and in the claims, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged, as appropriate, such that embodiments of the present application may be implemented in sequences other than those illustrated or described herein, and that the objects identified by "first," "second," etc. are generally of a type and not limited to the number of objects, e.g., the first object may be one or more. Furthermore, in the description and claims, "and/or" means at least one of the connected objects, and the character "/", generally means that the associated object is an "or" relationship.

The following describes in detail the SMT patch error detection method, the SMT patch error detection device, the visual detection system, the electronic device and the readable storage medium provided in the embodiment of the present application through specific embodiments and application scenarios thereof with reference to the accompanying drawings.

The SMT patch error detection method can be applied to a terminal, and can be specifically executed by hardware or software in the terminal.

The terminal includes, but is not limited to, a portable communication device such as a mobile phone or tablet having a touch sensitive surface (e.g., a touch screen display and/or a touch pad). It should also be appreciated that in some embodiments, the terminal may not be a portable communication device, but rather a desktop computer having a touch-sensitive surface (e.g., a touch screen display and/or a touch pad).

In the following various embodiments, a terminal including a display and a touch sensitive surface is described. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, mouse, and joystick.

According to the SMT patch error detection method, an execution main body of the SMT patch error detection method can be electronic equipment or a functional module or a functional entity capable of realizing the SMT patch error detection method in the electronic equipment, the electronic equipment comprises, but is not limited to, a mobile phone, a tablet personal computer, a camera, a wearable device and the like, and the SMT patch error detection method provided by the embodiment of the application is described below by taking the electronic equipment as the execution main body.

As shown in FIG. 1, the SMT patch error detection method comprises the following steps: step 110 and step 120.

And 110, acquiring an SMT patch image to be detected and an SMT patch template image of the circuit board.

Among them, the surface mount technology (Surface Mounted Technology, SMT) is a technology of mounting and attaching electronic components on the surface of a printed circuit board.

In this embodiment, the circuit board is a circuit board on which electronic components are mounted based on SMT technology, and the SMT patch image to be detected of the circuit board is an image obtained by photographing the surface of the side of the circuit board on which the electronic components are mounted to be detected.

In this embodiment, a circuit board, e.g., a cell phone, a computer, an automobile, etc., may be included in the electronic product.

In this embodiment, the circuit board may include a plurality of elements thereon, such as resistors, capacitors, diodes, etc., which may be soldered to the circuit board.

In this embodiment, the individual components on the same circuit board are normally soldered in a fixed position.

In this embodiment, the SMT patch image to be detected may be an image in which each component on the circuit board is in a correct position, or may be an image in which there is a component on the circuit board that is not in a correct position.

In this embodiment, the SMT patch template image is an image of the circuit board corresponding to the SMT patch image to be detected when each component is in the correct position.

In this embodiment, the same image capturing device may be used to capture the circuit board to be detected under the same light source, obtain the SMT patch image of the circuit board to be detected, and capture the circuit board of the same kind and with correct positions of the respective components, so as to obtain the SMT patch template image.

And 120, inputting the SMT patch image to be detected and the SMT patch template image to the countermeasure network classification model to obtain a wrong part detection result of the circuit board output by the countermeasure network classification model.

In this embodiment, the countermeasure network classification model is based on generating a classification model of the countermeasure network, and the countermeasure network classification model may detect the SMT patch image to be detected based on a difference between the SMT patch image to be detected and the SMT patch template image, and output a wrong detection result of the circuit board.

For example, as shown in fig. 2, the detection image, that is, the SMT patch image to be detected and the template image, that is, the SMT patch template image are simultaneously input to the countermeasure network classification model, and the countermeasure network classification model outputs a defect classification, that is, a wrong piece detection result, based on the difference between the SMT patch image to be detected and the SMT patch template image.

In this embodiment, it may be determined whether components in the circuit board corresponding to the SMT patch image to be detected are all in the correct positions based on the wrong piece detection result.

In this embodiment, each first pair of sample images includes a first mispiece defect sample image and a first template sample image corresponding to the first mispiece defect sample image.

The first wrong-part defect sample image is an image of a circuit board with the component not at the correct position, and is acquired by an image acquisition device.

In this embodiment, the first template sample image corresponding to the first error defect sample image is an image with a circuit board type consistent with that of the first error defect sample image and a correct component position, and the first template sample image may be acquired by an image acquisition device.

In this embodiment, when the first wrong-piece defect sample image and the first template sample image corresponding to the first wrong-piece defect sample image are acquired, the image acquisition equipment used is the same, and the illumination when the pictures are acquired is the same.

In this embodiment, each second sample image pair includes a second mispiece defect sample image and a second template sample image corresponding to the second mispiece defect sample image.

The second wrong-piece defect sample image is an image of the circuit board with the component not in the correct position, and is generated based on the patch component image.

In this embodiment, the patch element image is an image corresponding to each element on the circuit board, for example, the patch element image is a resistance image, a capacitance image, or the like.

In this embodiment, the second template sample image corresponding to the second wrong-part defect sample image is an image with a circuit board type consistent with a circuit board type corresponding to the second wrong-part defect sample image and a correct component position, and the second template sample image may be generated based on the patch component image.

In this embodiment, the patch element images may be placed in the blank circuit board image at the wrong locations to obtain a second wrong-piece defect sample image, and each patch element image may be placed in the blank circuit board image at the correct locations to obtain a second template sample image.

In the related art, two types of commonly used circuit board wrong part defect methods are available, namely, a manual visual detection method is adopted, namely, the wrong part defect is detected by utilizing human eyes, and the method depends on the familiarity degree of each person on a detection object and has low efficiency; another category is Automated Optical Inspection (AOI), which typically uses conventional computer vision techniques such as image processing and feature extraction, often requires a manually designed feature extractor to extract features, and is relatively poorly adapted in complex and diverse scenes.

According to the method and the device, an countermeasure network classification model is obtained based on training of a sample image set, the sample image set comprises a plurality of first sample image pairs and a plurality of second sample image pairs, the first sample image pairs comprise first error piece defect sample images and first template sample images corresponding to the first error piece defect sample images, the second sample image pairs comprise second error piece defect sample images and second template sample images corresponding to the second error piece defect sample images, the first error piece defect sample images are acquired by image acquisition equipment, the second error piece defect sample images are generated based on patch element images, the error piece defect sample images in the sample image set are rich in variety, the countermeasure network classification model obtained based on training of the sample image set is high in detection precision and high in detection efficiency, and can be suitable for complex and high-diversity scenes.

In the related art, a correct template image is configured for installing a component corresponding to a circuit board to be detected by setting a configuration file, the configuration file has a large influence on a detection result, the configuration file is easy to make mistakes, and in addition, different configuration files are required to be set for different circuit boards, so that the operation is complex and the cost is high.

In this embodiment of the application, obtain SMT paster template image, obtain the testing result through detecting the difference of waiting to detect SMT paster image and SMT paster template image, need not set up configuration file, easy operation, and can reduce cost.

According to the SMT patch wrong part detection method provided by the embodiment of the application, the countermeasure network classification model is trained through the sample image set, the sample image set comprises a first wrong part defect sample image acquired by equipment and a second wrong part defect sample image generated based on patch element images, the sample data are rich, the SMT patch wrong part detection is carried out based on the trained countermeasure network classification model, the obtained detection result is more accurate, the detection efficiency is high, the whole process is realized based on image processing, the detection result is obtained based on the difference between the SMT patch image to be detected and the SMT patch template image, a configuration file is not required to be set, the operation is simple, and the cost can be reduced.

In some embodiments, the second sample image pair is obtained by:

based on the patch element image set, replacing element areas in the second template sample image with patch element images with different element types, and generating a plurality of second error piece defect sample images;

and obtaining a plurality of second sample image pairs based on the plurality of second error piece defect sample images and the second template sample image.

In this embodiment, the patch element image set includes patch element images of all element types that should be mounted on the corresponding type of circuit board.

In this embodiment, the element region is a region where the element is mounted, replacing the element region in the second template sample image with a patch element image having a different element type is deleting or covering the element of the element region in the second template sample image, and placing the patch element image in the element region.

In this embodiment, the patch element image is placed in the element region of the corresponding element type, and a second error defect sample image may be obtained, for example, a resistive element image is placed in the capacitor region.

In this embodiment, the patch element images may be randomly matched in pairs to form a wrong-piece image pair, where two patch element images of the wrong-piece image pair are respectively placed in element areas corresponding to the element types of the other pair, so as to obtain a second wrong-piece defect sample image.

For example, the element type of the patch element image a is a1, the element type of the patch element image B is B1 when the patch element image a is placed in the element region corresponding to a1, the patch element image a is placed in the element region corresponding to B1, and the patch element image B is placed in the element region corresponding to a1, so that the second error defect sample image can be obtained.

In this embodiment, the patch element images may be randomly matched in pairs multiple times to form a plurality of error image pairs, so as to obtain a plurality of second error defect sample images.

For example, with patch element image A, B, C, D, E, F, first a and B, C, D, E, F are randomly matched 3-5 times to obtain 3-5 wrong-piece image pairs containing a, then B and A, C, D, E, F are randomly matched 3-5 times to obtain 3-5 wrong-piece image pairs containing B, and so on, multiple wrong-piece image pairs can be obtained, so as to obtain multiple second wrong-piece defect sample images.

In this embodiment, the second template sample image and each of the correspondingly generated second error-prone sample images may comprise a second sample image pair.

In some embodiments, prior to replacing the component areas in the second template sample image with patch component images of different component types, the method further comprises:

the patch element image is size transformed based on the size of the element region in the second template sample image.

In this embodiment, the patch element image may be subjected to a size conversion to convert its size to a size in conformity with the element region to be placed.

In actual implementation, the size transformation of the patch element image may be achieved by adjusting the size of each pixel in the patch element image.

In some embodiments, the countermeasure network classification model includes a feature extraction layer, a feature fusion layer, and a classifier connected in sequence, the feature extraction layer includes a first feature extraction module and a second feature extraction module, the SMT patch image to be detected and the SMT patch template image are input to the countermeasure network classification model, and a wrong component detection result of the circuit board output by the countermeasure network classification model is obtained, including:

inputting the SMT patch image to be detected into a first feature extraction module, and inputting the SMT patch template image into a second feature extraction module to obtain a first image feature output by the first feature extraction module and a second image feature output by the second feature extraction module;

Inputting the first image features and the second image features into a feature fusion layer to obtain fusion image features output by the feature fusion layer;

and inputting the fused image features into a classifier to obtain a wrong detection result of the circuit board output by the classifier.

In this embodiment, the feature extraction layer may perform image feature extraction, the first feature extraction module may extract a first image feature of the SMT patch image to be detected, and the second feature extraction module may extract a second image feature of the SMT patch template image.

In this embodiment, the first image feature may represent feature information of texture, edge, etc. of the SMT patch image to be detected, and the first image feature may represent feature information of texture, edge, etc. of the SMT patch template image.

In this embodiment, the feature fusion layer may fuse the first image feature and the second image feature to obtain a fused image feature, for example, the first image feature and the second image feature may be added to obtain the fused image feature.

In this embodiment, the classifier may classify the SMT patch image to be detected based on the fused image features, so as to obtain a wrong component detection result of the circuit board, that is, whether the circuit board has a wrong component defect.

In this embodiment, a linear (linear) layer and a softmax function may be included in the classifier, and the SMT patch image to be detected may be classified by the linear (linear) layer and the softmax function.

In some embodiments, the first feature extraction module and the second feature extraction module are constructed based on a MobileNetV2 network.

The mobile network V2 (MobileNetV 2) is a deep learning model, and can be used for feature extraction, image classification, and the like.

In this embodiment, the mobilenet v2 network has the advantage of high efficiency and rapidness, and the training speed can be improved by constructing the first feature extraction module and the second feature extraction module based on the mobilenet v2 network.

In some embodiments, the challenge network classification model is trained by:

acquiring a sample image set;

carrying out data enhancement processing on the sample image set;

training and updating the countermeasure network classification model to be trained according to the sample image set after data enhancement until iteration is stopped, and obtaining a trained countermeasure network classification model.

In this embodiment, the data enhancement processing is a process of performing different kinds of transformations on each of the first misinformation defect sample image and the second misinformation defect sample image in the sample image set, and obtaining a plurality of transformed misinformation defect sample images on the basis of the misinformation defect sample images.

In this embodiment, the sample image set after the data enhancement processing may include a lot of kinds of mispiece defect samples, that is, an original mispiece defect sample image and a transformed mispiece defect sample image.

In this embodiment, a loss function may be constructed, and the countermeasure network classification model is updated based on the loss function, so that the detection accuracy of the countermeasure network classification model is continuously improved, and a trained countermeasure network classification model is obtained.

In some embodiments, performing data enhancement processing on a sample image set includes:

performing at least one of flipping, rotating, scaling and deforming operations on the mismatching defective sample image of the sample image set;

The wrong-piece defect sample image comprises a first wrong-piece defect sample image and a second wrong-piece defect sample image.

In this embodiment, the flipping is performed such that the wrong-piece defect sample image is flipped according to the set axis, for example, a vertical axis is set, the wrong-piece defect sample image is flipped left and right, and a horizontal axis is set, so that the wrong-piece defect sample image is flipped up and down.

In this embodiment, the rotation is performed by rotating the error defect sample image according to a set reference point, which is a stationary point in the error defect sample image, for example, a center point of the error defect sample image is set as a reference point, and the error defect sample image is rotated clockwise with respect to the center point.

In this embodiment, scaling is performed to scale up or down the mispiece defect sample image to a certain scale, and scaling may be performed for each pixel of the mispiece defect sample image to a specified scale.

In this embodiment, the deformation is performed by performing operations such as twisting, stretching, compressing, etc. on the error piece defect sample image, for example, setting a deformation mapping relationship, substituting coordinates corresponding to pixels where elements are located in the error piece defect sample image into the deformation mapping relationship, obtaining new pixel coordinates, and moving the pixels to the new pixel coordinates.

In this embodiment, the color transformation process may be a random adjustment of the color values of individual pixels in the error-prone sample image.

For example, 20% of pixels in the wrong sample image are randomly selected, and the color value of the 20% of pixels is multiplied by a random factor to obtain a new color value to replace the original color value.

In some embodiments, training and updating the challenge network classification model to be trained from the data-enhanced sample image set includes:

inputting the sample image set after data enhancement into an countermeasure network classification model to be trained;

calculating a cross entropy loss function and a dynamic scaling cross entropy loss function based on the error defect labels of the sample image set and error defect information predicted by the countermeasure network classification model;

model parameters of the countermeasure network classification model are updated based on the cross entropy loss function and the dynamically scaled cross entropy loss function.

In this embodiment, the missing piece defect label may be an actual missing piece defect type corresponding to the missing piece defect sample image in the sample image set, and the missing piece defect information may be a missing piece defect type predicted by the countermeasure network classification model.

In this embodiment, by the loss function, a gap between the predicted result and the actual result of the countermeasure network classification model can be calculated, so that the model parameters of the countermeasure network classification model are updated according to the gap.

In this embodiment, a Cross-entropy loss function (CE loss) and a dynamic scaling Cross-entropy loss function (Focal loss) may be used, optimized for prediction accuracy and difficulty samples, respectively, and the values of the two are added, and model parameters of the countermeasure network classification model are updated based on the added values.

CE loss can be calculated by the following formula:

in the formula, x represents a sample, y represents an actual label corresponding to the sample, p represents a predicted output result, and n represents a total sample amount.

Focal Loss can be calculated by the following formula:

in the formula, alpha is the proportion between positive and negative sample losses, lambda is the loss contribution of a reduced-probability sample, and the larger p is the foreground type or background type, the easier the sample is distinguished, and the smaller the modulation factor is.

In this embodiment, α may be set to 0.25 and λ may be set to 2.

In this embodiment, all the error defect sample images in the sample image set after data enhancement may be divided into a training set, a verification set and a test set according to a certain proportion, for example, according to 8:1: the scale of 1 is divided into a training set, a validation set and a test set.

The training set is data for training the countermeasure network classification model, the verification set is data for verifying the detection effect of the countermeasure network classification model in the training process, and the test set is data for testing the detection effect of the countermeasure network classification model after the countermeasure network classification model is trained.

An embodiment of the SMT patch error detection method is as follows.

As shown in fig. 3, step one, collecting the images of the wrong-part defect and the corresponding template images to form a wrong-part image pair, i.e. obtaining a plurality of first sample image pairs.

And step two, collecting images of different elements, namely acquiring an image set of the patch element.

Step three, generating error piece pairs by data enhancement, namely replacing element areas in the second template sample images with patch element images with different element types based on a patch element image set, generating a plurality of second error piece defect sample images, and obtaining a plurality of second sample image pairs based on the plurality of second error piece defect sample images and the second template sample images, wherein the sample image set comprises a plurality of first sample image pairs and a plurality of second sample image pairs.

And fourthly, preprocessing the image pair, namely performing at least one of turning, rotating, zooming and deforming operation on the wrong piece defect sample image of the sample image set, and performing color conversion processing on the wrong piece defect sample image of the sample image set to obtain a sample image set with enhanced data.

Training the model, and continuously performing training effect test in the training process, namely training according to the sample image set after data enhancement, and updating the countermeasure network classification model to be trained until iteration is stopped, so as to obtain the trained countermeasure network classification model.

Step six, acquiring an SMT patch image and an SMT patch template image to be detected of the circuit board, inputting the SMT patch image and the SMT patch template image to be detected into a trained countermeasure network classification model, and obtaining classification results output by the countermeasure network classification model, namely false part detection results of the circuit board.

According to the SMT patch wrong piece detection method, the execution main body can be the SMT patch wrong piece detection device. In this embodiment of the application, an SMT patch error detection device executes an SMT patch error detection method as an example, which illustrates the SMT patch error detection device provided in this embodiment of the application.

The embodiment of the application also provides a SMT patch wrong piece detection device.

As shown in fig. 4, the SMT patch error detection device includes:

an acquiring module 410, configured to acquire an SMT patch image and an SMT patch template image to be detected of a circuit board;

the processing module 420 is configured to input the SMT patch image to be detected and the SMT patch template image to the countermeasure network classification model, and obtain a wrong component detection result of the circuit board output by the countermeasure network classification model;

According to the SMT patch wrong piece detection device provided by the embodiment of the application, the countermeasure network classification model is trained through the sample image set, the sample image set comprises a first wrong piece defect sample image acquired by equipment and a second wrong piece defect sample image generated based on a patch element image, sample data are rich, SMT patch wrong piece detection is carried out based on the trained countermeasure network classification model, the obtained detection result is more accurate, the detection efficiency is high, the whole process is realized based on image processing, the detection result is obtained based on the difference between the SMT patch image to be detected and the SMT patch template image, a configuration file is not required to be set, the operation is simple, and the cost can be reduced.

In some embodiments, the second sample image pair is obtained by:

In some embodiments, the processing module 420 is configured to size transform the patch element image based on the size of the element region in the second template sample image.

In some embodiments, the countermeasure network classification model includes a feature extraction layer, a feature fusion layer and a classifier that are sequentially connected, where the feature extraction layer includes a first feature extraction module and a second feature extraction module, and the processing module 420 is configured to input an SMT patch image to be detected to the first feature extraction module, and input an SMT patch template image to the second feature extraction module, so as to obtain a first image feature output by the first feature extraction module and a second image feature output by the second feature extraction module;

In some embodiments, the challenge network classification model is trained by:

Acquiring a sample image set;

carrying out data enhancement processing on the sample image set;

In some embodiments, the processing module 420 is configured to perform at least one of a flipping, rotating, scaling, and deforming operation on the mispiece defect sample image of the sample image set;

In some embodiments, the processing module 420 is configured to input the data-enhanced sample image set into an countermeasure network classification model to be trained;

The SMT patch error detection device in the embodiment of the present application may be an electronic device, or may be a component in the electronic device, for example, an integrated circuit or a chip. The electronic device may be a terminal, or may be other devices than a terminal. By way of example, the electronic device may be a mobile phone, tablet computer, notebook computer, palm computer, vehicle-mounted electronic device, mobile internet appliance (Mobile Internet Device, MID), augmented reality (augmented reality, AR)/Virtual Reality (VR) device, robot, wearable device, ultra-mobile personal computer, UMPC, netbook or personal digital assistant (personal digital assistant, PDA), etc., but may also be a server, network attached storage (Network Attached Storage, NAS), personal computer (personal computer, PC), television (TV), teller machine or self-service machine, etc., and the embodiments of the present application are not limited in particular.

The SMT patch error detection device in the embodiment of the present application may be a device with an operating system. The operating system may be an Android operating system, an IOS operating system, or other possible operating systems, which is not specifically limited in the embodiments of the present application.

The SMT patch error detection device provided in this embodiment of the present application may implement each process implemented by the method embodiments of fig. 1 to 3, and in order to avoid repetition, a description is omitted here.

The embodiment of the application also provides a visual detection system.

The vision detection system comprises an image acquisition device and a data processing device.

the data processing device is electrically connected with the image acquisition device and is used for executing the SMT patch error detection method.

In this embodiment, the data processing device may be a chip, a CPU, or the like, which writes the above-described SMT patch error detection method.

According to the visual detection system provided by the embodiment of the application, the countermeasure network classification model is trained through the sample image set, the sample image set comprises a first misplaced piece defect sample image acquired by equipment and a second misplaced piece defect sample image generated based on the patch element image, the sample data are rich, SMT patch misplaced piece detection is carried out based on the trained countermeasure network classification model, the obtained detection result is more accurate, the detection efficiency is high, the whole process is realized based on image processing, the detection result is obtained based on the difference between the SMT patch image to be detected and the SMT patch template image, a configuration file is not required to be set, the operation is simple, and the cost can be reduced.

In some embodiments, as shown in fig. 5, the embodiment of the present application further provides an electronic device 500, including a processor 501, a memory 502, and a computer program stored in the memory 502 and capable of running on the processor 501, where the program when executed by the processor 501 implements each process of the above-mentioned SMT patch error component detection method embodiment, and the same technical effects can be achieved, and for avoiding repetition, a detailed description is omitted herein.

The electronic device in the embodiment of the application includes the mobile electronic device and the non-mobile electronic device described above.

The embodiment of the application further provides a non-transitory computer readable storage medium, on which a computer program is stored, which when executed by a processor, implements each process of the above embodiment of the SMT patch error detection method, and can achieve the same technical effect, so that repetition is avoided, and no further description is given here.

Wherein the processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes computer readable storage medium such as computer readable memory ROM, random access memory RAM, magnetic or optical disk, etc.

The embodiment of the application also provides a computer program product, which comprises a computer program, and the computer program realizes the SMT patch error detection method when being executed by a processor.

The embodiment of the application further provides a chip, the chip includes a processor and a communication interface, the communication interface is coupled with the processor, the processor is used for running a program or an instruction, implementing each process of the above embodiment of the SMT patch error detection method, and achieving the same technical effect, so as to avoid repetition, and not repeated here.

It should be understood that the chips referred to in the embodiments of the present application may also be referred to as system-on-chip chips, chip systems, or system-on-chip chips, etc.

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element. Furthermore, it should be noted that the scope of the methods and apparatus in the embodiments of the present application is not limited to performing the functions in the order shown or discussed, but may also include performing the functions in a substantially simultaneous manner or in an opposite order depending on the functions involved, e.g., the described methods may be performed in an order different from that described, and various steps may also be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solutions of the present application may be embodied essentially or in a part contributing to the prior art in the form of a computer software product stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk), comprising several instructions for causing a terminal (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the methods described in the embodiments of the present application.

The embodiments of the present application have been described above with reference to the accompanying drawings, but the present application is not limited to the above-described embodiments, which are merely illustrative and not restrictive, and many forms may be made by those of ordinary skill in the art without departing from the spirit of the present application and the scope of the claims, which are also within the protection of the present application.

In the description of the present specification, reference to the terms "one embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present application. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

While embodiments of the present application have been shown and described, it will be understood by those of ordinary skill in the art that: many changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the application, the scope of which is defined by the claims and their equivalents.

Claims

1. The SMT patch wrong part detection method is characterized by comprising the following steps of:

2. The SMT patch error detection method of claim 1, wherein said second sample image pair is obtained by:

3. The SMT patch error detection method according to claim 2, wherein before said replacing a component area in said second template sample image with said patch component image having a component type different, said method further comprises:

4. The SMT patch wrong component detection method according to claim 1, wherein said countermeasure network classification model includes a feature extraction layer, a feature fusion layer and a classifier connected in sequence, said feature extraction layer includes a first feature extraction module and a second feature extraction module, said inputting said SMT patch image to be detected and said SMT patch template image to the countermeasure network classification model, obtaining a wrong component detection result of said circuit board output by said countermeasure network classification model includes:

5. The SMT patch error detection method of claim 4, wherein said first feature extraction module and said second feature extraction module are configured based on a MobileNetV2 network.

6. An SMT patch error detection method according to any of claims 1-5, wherein said challenge network classification model is trained by:

acquiring the sample image set;

performing data enhancement processing on the sample image set;

7. The SMT patch error detection method of claim 6, wherein said performing data enhancement processing on said sample image set comprises:

8. The SMT patch error detection method of claim 6, wherein said training and updating said challenge network classification model to be trained from said data enhanced sample image set comprises:

9. SMT paster wrong piece detection device, its characterized in that includes:

10. A visual inspection system, comprising:

a data processing device electrically connected to the image acquisition device, the data processing device being configured to perform the SMT patch error detection method according to any one of claims 1-8.