CN109741295B

CN109741295B - Product quality detection method and device

Info

Publication number: CN109741295B
Application number: CN201811434979.8A
Authority: CN
Inventors: 刘宏坤; 陈晓康; 王涌霖; 高巍; 张亮; 张向东
Original assignee: Goertek Inc
Current assignee: Goertek Inc
Priority date: 2018-11-28
Filing date: 2018-11-28
Publication date: 2021-04-20
Anticipated expiration: 2038-11-28
Also published as: CN109741295A

Abstract

The embodiment of the invention provides a product quality detection method and a device, wherein the method comprises the following steps: the detection equipment firstly obtains an image to be identified, which is obtained by shooting a product to be detected after the product to be detected is subjected to the spot welding process again, wherein the product to be detected is a product which is subjected to the spot welding process for the first time and has a preset defect type. Then, the image to be identified is input into a first classification module, so that the first classification module performs classification identification on the image to obtain at least one defect type corresponding to the product to be detected. Finally, the inspection equipment can determine the target defect type corresponding to the product to be inspected according to the confirmation processing logic corresponding to the identified at least one defect type. Therefore, the product quality detection method provided by the invention executes the confirmation processing logic corresponding to the determined defect type according to the determined defect type, so as to finally determine the target defect type of the product to be detected. The confirmation processing logic is targeted and there is no human intervention in the whole processing process.

Description

Product quality detection method and device

Technical Field

The invention relates to the technical field of automatic detection, in particular to a product quality detection method and device.

Background

Soldering is a common and important process in the manufacture of electronic devices. Due to the influence of various factors such as production environment, production equipment and production process, various defects inevitably occur in the welding process, and further problems may occur in the product quality. Common defects include lead lifting, no lead, cold joint, lead offset, and the like.

On the one hand, along with the strict requirements of the processing technology on the production line, higher requirements are also put on the detection capability of the fine defects of the electronic devices. On the other hand, in practical applications, for electronic products with different defects, different processing methods are often required, such as direct discarding or continuous use after processing. Therefore, it is important to accurately identify the defect type of the electronic device.

Disclosure of Invention

In view of this, embodiments of the present invention provide a method and an apparatus for detecting product quality, so as to improve accuracy of determining a defect type.

In a first aspect, an embodiment of the present invention provides a product quality detection method, including:

acquiring an image to be identified, which is obtained by shooting a product to be detected after the product to be detected is subjected to a spot welding process again, wherein the detected product is a product with a preset defect type after the product to be detected is subjected to the spot welding process for the first time, and the image to be identified comprises a spot welding area of the product to be detected after the product to be detected is subjected to the spot welding process again;

classifying and identifying the image to be identified according to a first classification model so as to identify at least one defect type corresponding to the product to be detected;

and determining the target defect type corresponding to the product to be detected according to the confirmation processing logic corresponding to the at least one defect type.

In a second aspect, an embodiment of the present invention provides a product quality detection apparatus, including:

the device comprises an acquisition module, a detection module and a processing module, wherein the acquisition module is used for acquiring an image to be identified, which is obtained by shooting a product to be detected after the product to be detected is subjected to a spot welding process again, the detected product is a product which is subjected to a preset defect type after the product to be detected is subjected to the spot welding process for the first time, and the image to be identified comprises a spot welding area of the product to be detected after the;

the classification module is used for classifying and identifying the image to be identified according to a first classification model so as to identify at least one defect type corresponding to the product to be detected;

and the type determining module is used for determining the target defect type corresponding to the product to be detected according to the confirmation processing logic corresponding to the at least one defect type.

According to the product quality detection method provided by the embodiment of the invention, the detection equipment firstly obtains the image to be identified, which is obtained by shooting the product to be detected after the spot welding process is carried out again, the product to be detected is the product with the preset defect type after the spot welding process is carried out for the first time, and the image to be identified comprises the spot welding area formed after the product to be detected is subjected to the spot welding process again. Then, the image to be recognized is input to the first classification module, so that the first classification module performs classification recognition on the image, the obtained classification result may include at least one defect type corresponding to the product to be detected, and the classification result may specifically be a probability value that the product to be detected has various defect types. Since different defect types have different confirmation processing logics, the detection equipment can determine the target defect type corresponding to the product to be detected according to the confirmation processing logic corresponding to the identified at least one defect type. Therefore, the product quality detection method provided by the invention executes the confirmation processing logic corresponding to the determined defect type according to the determined defect type, so as to finally determine the target defect type of the product to be detected which is subjected to the spot welding process again. The validation processing logic used is targeted and there is no human intervention throughout the process.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.

Fig. 1 is a flowchart of a product quality detection method according to an embodiment of the present invention;

FIG. 2 is a logic diagram of an optional target defect type validation process according to an embodiment of the present invention;

FIG. 3 is a logic diagram of another alternative target defect type validation process provided by an embodiment of the present invention;

FIG. 4 is another alternative target defect type validation processing logic provided in accordance with an embodiment of the present invention;

FIG. 5 is a block diagram illustrating yet another alternative target defect type validation processing logic according to an embodiment of the present invention;

FIG. 6 is a block diagram illustrating yet another alternative target defect type validation processing logic according to an embodiment of the present invention;

FIG. 7 is an alternative determination of training data corresponding to a second classification model provided by embodiments of the present invention;

fig. 8 is a schematic structural diagram of a product quality detection apparatus according to an embodiment of the present invention;

fig. 9 is a schematic structural diagram of an electronic device according to a first embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

The terminology used in the embodiments of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the examples of the present invention and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, and "a" and "an" generally include at least two, but do not exclude at least one, unless the context clearly dictates otherwise.

It should be understood that the term "and/or" as used herein is merely one type of association that describes an associated object, meaning that three relationships may exist, e.g., a and/or B may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship.

It should be understood that although the terms first, second, third, etc. may be used to describe XXX in embodiments of the present invention, these XXX should not be limited to these terms. These terms are only used to distinguish XXX from each other. For example, a first XXX may also be referred to as a second XXX, and similarly, a second XXX may also be referred to as a first XXX, without departing from the scope of embodiments of the present invention.

The words "if", as used herein, may be interpreted as "at … …" or "at … …" or "in response to a determination" or "in response to a detection", depending on the context. Similarly, the phrases "if determined" or "if detected (a stated condition or event)" may be interpreted as "when determined" or "in response to a determination" or "when detected (a stated condition or event)" or "in response to a detection (a stated condition or event)", depending on the context.

It is also noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a good or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such good or system. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a commodity or system that includes the element.

Fig. 1 is a flowchart of a first embodiment of a product quality detection method according to an embodiment of the present invention, where an execution main body of the product quality detection method according to this embodiment may be a detection device, as shown in fig. 1, the method includes the following steps:

s101, acquiring an image to be identified, which is obtained by shooting a product to be detected after the product to be detected is subjected to the spot welding process again, wherein the detected product is a product which is subjected to the spot welding process for the first time and has a preset defect type, and the image to be identified comprises a spot welding area of the product to be detected after the product is subjected to the spot welding process again.

After the products are processed by each process on the production line, the products can be shot by an industrial camera, so that whether the products processed by each process are qualified or not is determined according to shot images. Meanwhile, in consideration of reducing waste, after the product to be detected is subjected to the spot welding process for the first time, for the product with the preset defect type, the mechanical arm on the production line can clamp the product out, then the lead on the pad can be torn off manually, and the rest pad is put into a spot welding machine for spot welding again. The preset defect type may be any one of the following defect types, namely a heavy defect type.

In the spot welding scenes provided by the embodiment and the following embodiments, the product to be detected may be a product which has a preset defect type after the first spot welding process and is subjected to the second spot welding. The industrial camera shoots the product to be detected after the spot welding process is carried out again so as to obtain an image to be identified corresponding to the product to be detected. The image to be identified includes a spot welding area, and optionally, the spot welding area may include a lead, a welding point and a welding pad.

In an actual spot welding scenario, the welding device may spot weld the sound-producing component, i.e., the voice coil, in the microphone to the pad via the lead wire. Then, the industrial camera shoots the voice coil and the bonding pad which are integrated after spot welding, so as to obtain the image to be identified.

In the above mentioned spot welding scenario, there may be a plurality of possible defect types of the product to be detected, for example, there are no defects, lead lines are tilted, no lead line is on the pad to which the product to be detected is welded, a cold solder joint, a spot welding, a lead line has a deviation (specifically including lead line inward deviation and lead line outward deviation), a redundant lead line (specifically including single stub, multiple stub and stub head) exists on the pad to which the product to be detected is welded, a welding spot has impurities, or a shell of the product to be detected has a damage.

In the actual production, the defects of lead tilting, no lead, insufficient soldering, dotting, lead external deviation and lead internal deviation can be considered as severe defects. The products to be detected with the serious defects can be clamped out by a mechanical hand on a production line, and the spot welding process can be carried out again after the products to be detected with the serious defects are manually treated. That is, when the product is determined to have any one of the predetermined defect types, i.e., the severe defects, after the initial spot welding, the product is spot-welded again, thereby forming a product to be detected. And after the industrial camera shoots the product, obtaining an image to be identified corresponding to the product to be detected. The defects of single broken wire, double broken wires and broken wire heads can be considered as medium defects, and the medium defects are usually discharged in a whole edition and put into production again after being manually repaired. Defects such as the presence of impurities in the weld or damage to the outer shell of the product to be inspected can be considered mild defects. These minor defects are usually caused by spot welding equipment, which does not affect the product properties, and products with such defects continue to remain on the production line for further processing. When the number of products with such light defects in a certain period of time is greater than a threshold value, the worker may re-debug the spot welding equipment.

S102, classifying and identifying the image to be identified according to the first classification model so as to identify at least one defect type corresponding to the product to be detected.

The detection device can directly receive the image to be recognized sent by the industrial camera, and then input the image to be recognized into the first classification model, so that the first classification model carries out classification recognition on the image and outputs a classification result. Optionally, the classification model may output at least one defect type corresponding to the product to be detected, that is, the output classification result may include probability values that the product to be detected has various defect types. An optional classification result form: defect type I: 95%, defect type II: 87%, defect type III: 40%, defect type IV: 15 percent. The classification result shows that the probability of the products to be detected having the defect type I, the defect type II, the defect type III and the defect type IV is 95%, 87%, 40% and 15% respectively.

S103, determining a target defect type corresponding to the product to be detected according to the confirmation processing logic corresponding to at least one defect type.

After at least one defect type corresponding to the product to be detected is obtained, the similarity of the target defects of the product to be detected can be further determined according to the specific content of the at least one defect type and the corresponding confirmation processing logic. For a brief description of this embodiment and the following embodiments, a defect type with the highest probability among at least one defect type corresponding to a product to be detected may be referred to as a first candidate defect type, and a defect type with the second highest probability among at least one defect type corresponding to a product to be detected may be referred to as a second candidate defect type.

Based on this, optionally, in the simplest manner, if the first candidate defect type is the preset defect type, it is determined that the first candidate defect type is the target defect type. And if the first candidate defect type is a non-preset defect type, determining that the second candidate defect type is a target defect type.

In order to distinguish from the preset defect type involved in step 101, the preset defect type mentioned in step 101 may be referred to as a first preset defect type, and the preset defect type mentioned in step 103 may be referred to as a second preset defect type. And this second predetermined defect type may typically be defect free or the presence of excess wire on the pads of the product to be inspected, based on the plurality of defect types exemplified in step 101.

In this embodiment, the detection device first obtains an image to be identified, which is obtained by shooting a product to be detected after the product to be detected is subjected to the spot welding process again, the product to be detected is a product having a preset defect type after the product to be detected is subjected to the spot welding process for the first time, and the image to be identified includes a spot welding area formed after the product to be detected is subjected to the spot welding process again. Then, the image to be recognized is input to a first classification module, so that the first classification module performs classification recognition on the image, the obtained classification result may include at least one defect type corresponding to the product to be detected, and the classification result may specifically be a probability value that the product to be detected has various defect types. Since different defect types have different confirmation processing logics, the detection equipment can determine the target defect type corresponding to the product to be detected according to the confirmation processing logic corresponding to the identified at least one defect type. Therefore, the product quality detection method provided by the invention executes the confirmation processing logic corresponding to the determined defect type according to the determined defect type, so as to finally determine the target defect type of the product to be detected which is subjected to the spot welding process again. The confirmation processing logic is targeted and there is no human intervention in the whole processing process. In addition, as for the first classification model in step 102, it may be a classification model trained in a deep learning manner. Alternatively, the training data used for training the first classification model may be obtained according to the following procedure: the method comprises the steps of firstly acquiring images of products with various types of defects shot by an industrial camera, wherein the defect type corresponding to each image is known. The images are then converted to a format supported by the model training, such as to a tf-record data file, and each piece of binary data in the tf-record data file corresponds to a recognized image. After conversion, the binary data corresponding to each identified image is then associated with the defect type of the detected product. This association process may also be understood as setting a defect type tag for each binary datum in the tf-record data file. And finally, taking the binary data with the defect type labels in the tf-record data file as training data, thereby training to obtain a first classification model. Alternatively, for the obtained image, images belonging to the same defect type may be saved into one folder according to the defect type, and the format of the image therein may be converted in units of folders.

In addition, in practical applications, the training process of the first classification model is usually performed by a processing device, which may be built in the detection device, and the processing capabilities of different processing devices are also uneven. Therefore, before the image is subjected to format conversion, the image size is also adjusted according to the processing capacity of the device, so that the image size meets the processing capacity of the processing device while the image information is ensured not to be lost. Moreover, for processing equipment with low processing capacity, because the processing equipment cannot process a large amount of training data at one time, after files in the tf-record format are obtained, the files can be grouped according to a preset number to obtain at least one group of files, each group of files are sequentially used as training data, and finally a first classification model is obtained through training. Alternatively, each set of files may correspond to all or part of the defect types.

There is also a need forIn the first classification model training process, in order to improve the classification accuracy of the first classification model, the following Loss function Loss may be used: loss ═ alpha (1-p)^γlog (p). Wherein alpha is a preset coefficient, p is a probability value that a product to be detected has a certain defect type, and log (p) is a cross entropy.

In the above description of the embodiment shown in fig. 1, a plurality of defect levels, and a plurality of defect types included in each defect level, have been mentioned. And the defect type with the highest probability value, namely the first candidate defect type, in the at least one defect type determined by the first classification model may also be any one of the defect types. The detection device may execute different validation processing logic for different first candidate defect types. The following describes the confirmation processing logic corresponding to each of the different first candidate defect types.

For such a product subjected to the spot welding process again, since it is subjected to the initial spot welding, there is a possibility that an impression of the lead at the time of the initial spot welding may be left on the pad. Therefore, in order to avoid the inspection apparatus directly determining this wire indentation as the presence of an unnecessary wire on the pad, resulting in the generation of a wire detected as defective, alternatively, as shown in fig. 2, one case may be:

and if the defect type with the highest probability value in the at least one defect type, namely the first candidate defect type, is the first defect type, analyzing the image to be identified to determine whether lead indentation exists on the bonding pad welded with the product to be detected. The first defect type can be that redundant leads exist on the welding pad on which the product to be detected is welded.

And if the lead indentation does not exist, determining that the target defect type corresponding to the product to be detected is the first defect type. And if the lead indentation exists, inputting the image to be recognized into the second classification model, and further determining the target defect type corresponding to the product to be detected according to the result output by the second classification model. The second classification model can be a small welding spot model used for determining whether the product to be detected has cold solder or not, and the cold solder can be expressed as small welding spot.

Specifically, if the second classification model determines that the product to be detected has the cold joint, the target defect type is determined to be the cold joint, and the manipulator on the production line clamps the product to be detected. And if the second classification model determines that the product to be detected does not have the cold joint, determining that the product to be detected is normal or has no defects.

For determining whether the product to be detected has lead indentation, an optional determination method comprises the following steps:

a semantic segmentation map corresponding to the image to be recognized is generated, and for the generation of the semantic segmentation map, any one of a Thresholding method (Thresholding method), a pixel-Clustering-based segmentation method (Clustering-based segmentation method), a Graph-partitioning segmentation method (Graph segmentation method), or Deep learning (Deep learning) may be generally employed. The semantic segmentation graph can comprise a lead of a product to be detected after a spot welding process. Optionally, pads and pads may be included in the semantic segmentation map in addition to the leads. And the lead, the pad and the pad are distinguished and indicated by different colors, respectively. The detection equipment can accurately determine the area corresponding to the lead in the semantic segmentation graph according to different colors. If the number of the areas corresponding to the leads in the semantic segmentation graph is zero, determining that the product to be detected has an indentation; and if the number of the areas corresponding to the lead lines in the semantic segmentation graph is not zero, determining that the target defect type is the first defect type.

Alternatively, yet another case may be:

and if the defect type with the highest probability value in the at least one defect type, namely the first candidate defect type, is the second defect type, directly determining that the target defect type corresponding to the product to be detected is the second defect type, and clamping the product to be detected by a manipulator on the production line. The second defect type can be lead tilting or no lead on a bonding pad welded with a product to be detected. In actual production, the above-mentioned several second defect types are obvious defects, and the possibility of judgment errors is low, so the logic for confirming the second defect type is simple.

Alternatively, as shown in fig. 3, yet another case may be:

and if the defect type with the highest probability in the at least one defect type, namely the first candidate defect type, is the third defect type, and the probability value corresponding to the third defect type is greater than or equal to the first preset value, directly determining that the target defect type of the product to be detected is the third defect type. Wherein the third defect type may be non-defective, i.e., qualified product, and the first preset value is generally set to 55%.

And if the first candidate defect type is a third defect type and the probability value corresponding to the third defect type is smaller than the first preset value, determining a target defect type corresponding to the product to be detected according to the defect type with the second highest probability value in the at least one defect type, namely the second candidate defect type.

Specifically, if the second candidate defect type is a fourth defect type and the probability value corresponding to the fourth defect type is smaller than the second preset value, the target defect type corresponding to the product to be detected is directly determined to be the third defect type. The fourth selected defect type may be a cold joint or a lead offset, the cold joint may be specifically represented as a small joint, and the second preset value is usually set to 25%. The second classification model may also be a small classification model of the solder joint for determining whether the product to be detected has a cold solder joint.

And if the second candidate defect type is a fourth defect type and the probability value corresponding to the fourth defect type is greater than or equal to a second preset value, inputting the image to be identified corresponding to the product to be detected into a second classification model, and determining the target defect type corresponding to the product to be detected according to the classification result output by the second classification model.

Specifically, if the second classification model determines that the product to be detected has the insufficient solder, the insufficient solder is used as the target defect type of the product to be detected, and the manipulator on the production line clamps the product out. And if the second classification model determines that the product to be detected does not have the insufficient solder, determining the third defect type of the product to be detected as the target defect type.

And the corresponding confirmation processing logic is used for confirming that the first candidate defect type of the product to be detected is non-defective.

Alternatively, when the degree of defect is mild defect, as shown in fig. 4, another case may be:

and if the defect type with the highest probability value in the at least one defect type, namely the first candidate defect type, is the fifth defect type, and the probability value corresponding to the fifth defect type is greater than or equal to the third preset value, directly determining the target defect type corresponding to the product to be detected as the fifth defect type. The fifth defect type may be any one of impurities existing in a welding spot or slight damage existing in a shell of a product to be detected, and the third preset value may be generally set to 80%.

And if the first candidate defect type is a fifth defect type and the probability value corresponding to the fifth defect type is smaller than a third preset value, determining a target defect type corresponding to the product to be detected according to the defect type with the second highest probability value in the at least one defect type, namely the second candidate defect type.

And if the second candidate defect type is not the sixth defect type, directly determining the target defect type corresponding to the product to be detected as the second candidate defect type. The sixth defect type may be defect-free, insufficient solder, and offset of lead.

And if the second candidate defect type is the sixth defect type, inputting the image to be identified corresponding to the product to be detected into the second classification model, and further determining the target defect type corresponding to the product to be detected according to the classification result output by the second classification model.

Specifically, similarly to the case shown in fig. 3, the second classification model may also be a small classification model of the solder joint for determining whether or not the solder joint is faulty. And if the second classification model determines that the false solder exists, the false solder is used as the target defect type of the product to be detected, and the mechanical arm on the production line clamps the product out. And if the second classification model determines that the cold joint does not exist, determining the fifth defect type of the product to be detected as the target defect type, and enabling the product to be continuously circulated on the production line.

And correspondingly confirming the processing logic when the first alternative defect type of the product to be detected is that impurities exist in the welding spot or the shell of the product to be detected is damaged.

Alternatively, when the defect level is a heavy defect, as shown in fig. 5, still another case may be:

and if the defect type with the highest probability value in the at least one defect type, namely the first candidate defect type, is the lead deviation, analyzing the image to be identified, and determining whether the product to be detected has lead deviation or lead deviation.

As to an optional way for determining the lead internal deviation or external deviation, after the image to be recognized in step 101 is obtained, a semantic segmentation map corresponding to the image to be recognized is further generated according to the image. Different colors are used to represent different types of objects in the semantic segmentation map. Taking a spot welding scene as an example, the semantic segmentation map may represent the solder joint, the lead and the pad with different colors, such as pink for the pad, gray for the solder joint and orange for the lead. Then, in the semantic segmentation map, the minimum bounding rectangle and the center of the minimum bounding rectangle corresponding to the lead and the pad respectively are determined. And then setting a horizontal line, taking the horizontal line as a standard, rotating the horizontal line anticlockwise to obtain a first edge which is intersected with the minimum external rectangle corresponding to the lead wire firstly, and determining an included angle between the horizontal line and the first edge. And finally, determining whether the product to be detected has lead outward deviation or lead inward deviation according to the included angle and the position relation between the centers of the minimum external rectangles corresponding to the lead and the bonding pad respectively.

For the sake of simple subsequent description, in the semantic segmentation map, a pixel point corresponding to the center of the minimum circumscribed rectangle corresponding to the lead is called a first center pixel point, a pixel point corresponding to the center of the minimum circumscribed rectangle corresponding to the pad is called a second center pixel point, and the number of columns of the two pixel points in the semantic segmentation map are x respectively₁And x₂。

Specifically, if the included angle is smaller than the preset angle and x₁＜x₂And determining that the lead wire of the product to be detected is deviated.

If the included angle is greater than or equal to the preset angle and x₁≥x₂And determining that the lead wire of the product to be detected is deviated.

If the included angle is greater than or equal to the preset angle and x₁＜x₂And determining that the lead wire of the product to be detected is internally deviated.

If the included angle is smaller than the preset angle and x₁≥x₂And determining that the lead wire of the product to be detected is internally deviated.

Wherein the predetermined angle may be set to 45 °, and the angle between the horizontal and the first side is typically in the range of [ -90 °, 90 ° ]]. An angle less than the preset angle indicates a left deviation of the pad, and an angle greater than or equal to the preset angle indicates a right deviation of the pad. x is the number of₁＜x₂Denotes that the center of the minimum circumscribed rectangle corresponding to the lead is left of the center of the minimum circumscribed rectangle corresponding to the pad, x₁≥x₂The representation shows that the center of the minimum circumscribed rectangle corresponding to the lead is to the right of the center of the minimum circumscribed rectangle corresponding to the pad.

In order to further improve the accuracy of determining whether the lead has the deviation, the following processes can be further performed after the steps: determining the center of the minimum external rectangle corresponding to the lead in the semantic segmentation graph, fitting the minimum external rectangle corresponding to the lead into a first straight line, determining the intersection point of the first straight line and the lower bottom edge of the minimum external rectangle corresponding to the welding point, and determining a second straight line by the center of the minimum external rectangle corresponding to the lead and the intersection point. And further calculating the distance from the center of the minimum bounding rectangle corresponding to the bonding pad to the second straight line. And if the distance is greater than or equal to the preset distance, finally obtaining whether the lead has inward deviation or outward deviation according to the determination result. And if the distance is smaller than the preset distance, determining that the lead wire deviation does not exist in the product to be detected.

Through the process, the detection equipment can determine whether the product to be detected has lead external deviation or lead internal deviation. And if the product to be detected is determined to have lead wire outward deviation, directly determining that the target defect type corresponding to the product to be detected is the lead wire outward deviation, and clamping the product by a manipulator on the production line.

And if the product to be detected has lead internal deviation, inputting the image to be identified into a second classification model, namely a small welding spot model, and further determining the target defect type corresponding to the product to be detected according to a classification result output by the second classification model.

Specifically, if the second classification model determines that the cold joint exists, the cold joint is used as a target defect type of the product to be detected, and the manipulator on the production line is used for clamping the product to be detected. And if the second classification model determines that the cold solder joint does not exist, determining the product to be detected as being defect-free, and continuously circulating on the production line.

And the corresponding confirmation processing logic is used when the first candidate defect type of the product to be detected is the lead deviation.

Alternatively, when the defect degree is a heavy defect, as shown in fig. 6, still another case may be:

if the defect type with the highest probability value in the at least one defect type, namely the first candidate defect type, is the seventh defect type, and the probability value corresponding to the seventh defect type is greater than or equal to the fourth preset value, the target defect type corresponding to the product to be detected is directly determined to be the seventh defect type, and the manipulator on the production line clamps the product. Wherein the seventh defect type may be a cold solder joint or a spot welding, and the fourth preset value is usually set to 90%.

And if the first candidate defect type is a seventh defect type and the probability value corresponding to the seventh defect type is smaller than a fourth preset value, inputting the image to be identified into a second classification model, and determining the target defect type corresponding to the product to be detected according to the classification result output by the second classification model. Wherein, the second classification model can also be a small model of the welding spot.

Specifically, if the second classification model determines that the cold joint exists, the cold joint is used as a target defect type of the product to be detected, and the manipulator on the production line clamps the product out. And if the second classification model determines that the cold solder joint does not exist, determining the product to be detected as being defect-free, and enabling the product to continue to circulate on the production line.

And correspondingly confirming and processing logic when the first candidate defect type of the product to be detected is cold joint or spot welding.

In summary, the embodiments shown in fig. 2 to 6 are respectively the confirmation processing logic that should be executed when the first candidate defect type of the product to be detected is different. Therefore, each defect type has corresponding confirmation processing logic, so that the detection equipment can accurately determine the defects of the products to be detected, and further adopt different processing modes, such as direct discarding or reprocessing, for the products with different defect types, thereby avoiding the waste of the electronic devices in the production process.

For the above-mentioned embodiments shown in fig. 2 to 6, the second classification model, i.e. the small weld point classification model, is mentioned. The training process of the model is the same as that of the first classification model, and a deep learning mode can be adopted. The training data of the second classification model may be obtained by the following method:

and acquiring the identified images of the detected products with the defect types of the faulty solder and the non-defective solder from the identified images corresponding to the identified products with the defect types, wherein the non-defective solder and the non-defective solder respectively correspond to the positive and negative training data of the training. Then, a semantic segmentation map corresponding to the recognized image is generated, and different colors are also used in the semantic segmentation map to respectively represent the lead and the welding point of the detected product after the spot welding process. And further acquiring the minimum circumscribed rectangle corresponding to the lead and the minimum circumscribed rectangle corresponding to the welding point in the semantic segmentation graph. Then, the effective part in the recognized image is determined according to the minimum bounding rectangles corresponding to the lead and the welding point in the semantic segmentation graph. And finally, training by taking the effective part as training data to obtain a second classification model.

An alternative way to determine the effective part of the identified image is to fit the minimum bounding rectangle P corresponding to the lead line to a straight line L1, and determine the intersection point a of the straight line L1 and the lower bottom edge of the minimum bounding rectangle N corresponding to the welding point, as shown in fig. 7. And determining a line segment AB with the preset length L by taking the point A as the lower vertex of the central axis, and making a rectangle M by taking a straight line L1 as a symmetry axis, wherein the length of the rectangle is the same as that of the line segment AB, and the width of the rectangle M is also the preset value. Finally, an effective part corresponding to the rectangle M can be intercepted from the recognized image, and the effective part is used as training data to train and obtain a second classification model. Wherein, the position of the intercepted effective part in the recognized image is completely consistent with the position of the rectangle M in the semantic segmentation map.

In addition, the embodiments shown in fig. 2 to 6 are actually used to determine what type of defect the product after the spot welding process is performed again. However, in practical applications, the detection device may also determine the defect types of the products subjected to the initial spot welding process according to the corresponding confirmation processing logic of each defect type.

Fig. 8 is a schematic structural diagram of a first embodiment of a product quality detection apparatus according to an embodiment of the present invention, and as shown in fig. 8, the product quality detection apparatus includes: an acquisition module 11, a classification module 12 and a type determination module 13.

The acquisition module 11 is configured to acquire an image to be identified, which is obtained by shooting a product to be detected after the product is subjected to the spot welding process again, wherein the detected product is a product having a preset defect type after the product is subjected to the spot welding process for the first time, and the image to be identified includes a spot welding area of the product to be detected after the product is subjected to the spot welding process again.

And the classification module 12 is configured to perform classification and identification on the image to be identified according to a first classification model, so as to identify at least one defect type corresponding to the product to be detected.

And a type determining module 13, configured to determine, according to the confirmation processing logic corresponding to the at least one defect type, a target defect type corresponding to the product to be detected.

Optionally, the type determining module 13 in the product quality detecting apparatus is specifically configured to:

if the defect type with the highest probability value in the at least one defect type is the first defect type, analyzing the image to be identified to determine whether lead indentation exists on the bonding pad welded with the product to be detected;

if no lead indentation exists, determining that the target defect type corresponding to the product to be detected is the first defect type;

if the lead indentation exists, inputting the image to be recognized into a second classification model; and determining the target defect type corresponding to the product to be detected according to the result output by the second classification model.

Optionally, the type determining module 13 in the product quality detecting apparatus includes: a generating unit 131, and a type determining unit 132.

The generating unit 131 is configured to generate a semantic segmentation map corresponding to the image to be recognized, where the semantic segmentation map includes a lead of the product to be detected after the spot welding process.

The type determining unit 132 is configured to determine that an indentation exists in the product to be detected if the number of the areas corresponding to the lead lines in the semantic segmentation map is zero; and if the number of the areas corresponding to the lead lines in the semantic segmentation graph is not zero, determining the target defect type as the first defect type.

Optionally, the first defect type includes the presence of an excess lead on the pad to which the product to be detected is soldered.

and if the defect type with the highest probability value in the at least one defect type is the second defect type, determining that the target defect type corresponding to the product to be detected is the second defect type.

Optionally, the second defect type includes any one of the following types: the lead is tilted, and no lead is welded on the bonding pad of the product to be detected.

and if the defect type with the highest probability value in the at least one defect type is a third defect type, and the probability value corresponding to the third defect type is greater than or equal to a first preset value, determining that the target defect type corresponding to the product to be detected is the third defect type.

and if the defect type with the highest probability value in the at least one defect type is a third defect type, and the probability value corresponding to the third defect type is smaller than a first preset value, determining a target defect type corresponding to the product to be detected according to the defect type with the second highest probability value in the at least one defect type.

and if the defect type with the second highest probability value in the at least one defect type is a fourth defect type, and the probability value corresponding to the fourth defect type is smaller than a second preset value, determining that the target defect type corresponding to the product to be detected is the third defect type.

if the defect type with the highest probability value in the at least one defect type is a fourth defect type, and the probability value corresponding to the fourth defect type is greater than or equal to a second preset value, inputting the image to be recognized into a second classification model; and determining the target defect type corresponding to the product to be detected according to the classification result output by the second classification model.

Optionally, the third defect type is defect-free, and the fourth defect type is a faulty solder or a lead offset.

and if the defect type with the highest probability value in the at least one defect type is a fifth defect type, and the probability value corresponding to the fifth defect type is greater than or equal to a third preset value, determining that the target defect type corresponding to the product to be detected is the fifth defect type.

and if the defect type with the highest probability value in the at least one defect type is a fifth defect type and the probability value corresponding to the fourth defect type is smaller than a third preset value, determining a target defect type corresponding to the product to be detected according to the defect type with the second highest probability value in the at least one defect type.

if the defect type with the highest probability value in the at least one defect type is a sixth defect type, inputting the image to be recognized into a second classification model; and determining the target defect type corresponding to the product to be detected according to the classification result output by the second classification model.

and if the defect type with the second highest probability value in the at least one defect type is not the sixth defect type, determining that the target defect type corresponding to the product to be detected is the defect type with the second highest probability value.

Optionally, the fifth defect type includes any one of the following types: impurities exist in welding spots, and the shell of a product to be detected is damaged; the sixth defect type is any one of the following types: no defect, cold joint, and offset of lead.

if the defect type with the highest probability value in the at least one defect type is that a lead has deviation, analyzing the image to be identified to determine whether the product to be detected has lead outward deviation or lead inward deviation;

if the lead wire is out of line, determining that the target defect type corresponding to the product to be detected is the lead wire out of line;

if the lead internal deviation exists, inputting the image to be recognized into a second classification model; and determining the target defect type corresponding to the product to be detected according to the classification result output by the second classification model.

Optionally, the type determination model in the product quality detection apparatus includes: a generation unit 131, a center determination unit 133, an intersection determination unit 134, and a type determination unit 132.

The generating unit 131 is configured to generate a semantic segmentation map corresponding to the image to be identified, where the semantic segmentation map includes a lead and a pad of the product to be detected after the spot welding process.

A center determining unit 133, configured to determine, in the semantic segmentation map, minimum circumscribed rectangles corresponding to the leads and the pads, respectively, and respective centers of the minimum circumscribed rectangles.

And an intersection point determining unit 134, configured to determine an included angle between a horizontal line and a first side of the minimum circumscribed rectangle corresponding to the lead line, where the horizontal line intersects the first side first when rotating counterclockwise.

And the type determining unit 132 is configured to determine whether the product to be detected has lead outward deviation or lead inward deviation according to the included angle and a position relationship between centers of the minimum circumscribed rectangles corresponding to the lead and the pad respectively.

Optionally, the type determining unit 134 in the product quality detecting apparatus is specifically configured to:

if the included angle is smaller than a preset angle, the number of the lines of the first center pixel points in the semantic segmentation map is smaller than the number of the lines of the second center pixel points in the semantic segmentation map, or the included angle is larger than or equal to the preset angle, the number of the lines of the first center pixel points in the semantic segmentation map is larger than or equal to the number of the lines of the second center pixel points in the semantic segmentation map, determining that the product to be detected has lead wire offside, wherein the center of the minimum circumscribed rectangle corresponding to the lead wire is the first center pixel point, and the center of the minimum circumscribed rectangle corresponding to the pad is the second center pixel point;

and if the included angle is larger than or equal to the preset angle and the number of the lines of the first central pixel points in the semantic segmentation map is smaller than the number of the lines of the second central pixel points in the semantic segmentation map, or the included angle is smaller than the preset angle and the number of the lines of the first central pixel points in the semantic segmentation map is larger than or equal to the number of the lines of the second central pixel points in the semantic segmentation map, determining that the lead internal deviation exists in the product to be detected.

and if the defect type with the highest probability value in the at least one defect type is a seventh defect type, and the probability value corresponding to the seventh defect type is greater than or equal to a fourth preset value, determining that the target defect type corresponding to the product to be detected is the seventh defect type.

if the defect type with the highest probability value in the at least one defect type is a seventh defect type, and the probability value corresponding to the seventh defect type is smaller than a fourth preset value, inputting the image to be recognized into a second classification model; and determining the target defect type corresponding to the product to be detected according to the classification result output by the second classification model.

Optionally, the seventh defect type includes any one of the following types: and (5) cold joint and spot welding.

Optionally, the apparatus for detecting quality of product further comprises: a generation module 21, an image determination module 22 and a training module 23.

An acquiring module 11, configured to acquire an identified image of the detected product with the target defect type of cold joint and no defect.

And the generating module 21 is configured to generate a semantic segmentation map corresponding to the identified image, where the semantic segmentation map includes leads and welding points of the detected product after the spot welding process.

And the image determining module 22 is configured to determine an effective portion in the identified image according to the minimum circumscribed rectangles corresponding to the lead and the solder joint in the semantic segmentation map.

And the training module 23 is configured to train to obtain the second classification model by using the valid portion as training data.

The apparatus shown in fig. 8 can perform the method of the embodiment shown in fig. 1 to 7, and the related description of the embodiment shown in fig. 1 to 7 can be referred to for the part not described in detail in this embodiment. The implementation process and technical effect of the technical solution refer to the descriptions in the embodiments shown in fig. 1 to 7, and are not described herein again.

The internal functions and structure of the product quality detection apparatus are described above, and in one possible design, the structure of the product quality detection apparatus may be implemented as an electronic device. Fig. 9 is a schematic structural diagram of an electronic device according to a first embodiment of the present invention, and as shown in fig. 9, the electronic device includes: a memory 31, and a processor 32 connected to the memory, the memory 31 being used for storing a program for the electronic device to execute the product quality detection method provided in any of the above embodiments, the processor 32 being configured to execute the program stored in the memory 31.

The program comprises one or more computer instructions which, when executed by the processor 32, are capable of performing the steps of:

Optionally, processor 32 is also configured to perform all or some of the method steps described above.

The electronic device may further include a communication interface 33 for communicating with other devices or a communication network.

The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and the parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.

Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by adding a necessary general hardware platform, and of course, can also be implemented by a combination of hardware and software. With this understanding in mind, the above technical solutions may be embodied in the form of a computer-readable storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., which includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods according to the various embodiments or some parts of the embodiments.

Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims

1. A product quality detection method is characterized by comprising the following steps:

determining a target defect type corresponding to the product to be detected according to the confirmation processing logic corresponding to the at least one defect type;

determining the target defect type corresponding to the product to be detected according to the confirmation processing logic corresponding to the at least one defect type, including:

if the lead indentation exists, inputting the image to be recognized into a second classification model;

and determining the target defect type corresponding to the product to be detected according to the result output by the second classification model.

2. The method of claim 1, wherein said analyzing said image to be identified to determine whether said product to be detected has an indentation comprises:

generating a semantic segmentation map corresponding to the image to be recognized, wherein the semantic segmentation map comprises a lead of the product to be detected after a spot welding process;

if the number of the areas corresponding to the leads in the semantic segmentation graph is zero, determining that the product to be detected has an indentation;

and if the number of the areas corresponding to the lead lines in the semantic segmentation graph is not zero, determining that the target defect type is the first defect type.

3. Method according to claim 1 or 2, characterized in that said first type of defect comprises the presence of excess leads on the pads on which the products to be inspected are soldered.

4. The method according to claim 1, wherein the determining the target defect type corresponding to the product to be detected according to the confirmation processing logic corresponding to the at least one defect type comprises:

5. The method of claim 4, wherein the second defect type comprises any one of the following types: the lead is tilted, and no lead is welded on the bonding pad of the product to be detected.

6. The method according to claim 1, wherein the determining the target defect type corresponding to the product to be detected according to the confirmation processing logic corresponding to the at least one defect type comprises:

7. The method according to claim 1, wherein the determining the target defect type corresponding to the product to be detected according to the confirmation processing logic corresponding to the at least one defect type comprises:

8. The method according to claim 7, wherein the determining the target defect type corresponding to the product to be detected according to the defect type with the second highest probability value in the at least one defect type comprises:

9. The method according to claim 7, wherein the determining the target defect type corresponding to the product to be detected according to the defect type with the second highest probability value in the at least one defect type comprises:

if the defect type with the highest probability value in the at least one defect type is a fourth defect type, and the probability value corresponding to the fourth defect type is greater than or equal to a second preset value, inputting the image to be recognized into a second classification model;

and determining the target defect type corresponding to the product to be detected according to the classification result output by the second classification model.

10. The method of claim 8 or 9, wherein the third defect type is defect-free and the fourth defect type is a faulty solder or a wire offset.

11. The method according to claim 1, wherein the determining the target defect type corresponding to the product to be detected according to the confirmation processing logic corresponding to the at least one defect type comprises:

12. The method according to claim 1, wherein the determining the target defect type corresponding to the product to be detected according to the confirmation processing logic corresponding to the at least one defect type comprises:

and if the defect type with the highest probability value in the at least one defect type is a fifth defect type, and the probability value corresponding to the fifth defect type is smaller than a third preset value, determining a target defect type corresponding to the product to be detected according to the defect type with the second highest probability value in the at least one defect type.

13. The method according to claim 12, wherein the determining the target defect type corresponding to the product to be detected according to the defect type with the second highest probability value in the at least one defect type comprises:

if the defect type with the highest probability value in the at least one defect type is a sixth defect type, inputting the image to be recognized into a second classification model;

14. The method according to claim 13, wherein the determining the target defect type corresponding to the product to be detected according to the defect type with the second highest probability value in the at least one defect type comprises:

15. The method of claim 13, wherein the fifth defect type comprises any one of the following types: impurities exist in welding spots, and the shell of a product to be detected is damaged; the sixth defect type is any one of the following types: no defect, cold joint, and offset of lead.

16. The method according to claim 1, wherein the determining the target defect type corresponding to the product to be detected according to the confirmation processing logic corresponding to the at least one defect type comprises:

if the lead internal deviation exists, inputting the image to be recognized into a second classification model;

17. The method of claim 16, wherein analyzing the image to be identified to determine whether the product to be detected has lead-out or lead-in bias comprises:

generating a semantic segmentation map corresponding to the image to be recognized, wherein the semantic segmentation map comprises a lead and a bonding pad of the product to be detected after a spot welding process;

determining minimum circumscribed rectangles corresponding to the lead and the bonding pad respectively and respective centers of the minimum circumscribed rectangles in the semantic segmentation graph;

determining an included angle between a horizontal line and a first edge of a minimum circumscribed rectangle corresponding to the lead, wherein the horizontal line is intersected with the first edge firstly when rotating anticlockwise;

and determining whether the product to be detected has lead outward deviation or lead inward deviation according to the included angle and the position relationship between the centers of the minimum external rectangles corresponding to the lead and the bonding pad respectively.

18. The method according to claim 17, wherein the determining whether the product to be detected has lead out deviation or lead in deviation according to the included angle and the position relationship between the centers of the minimum circumscribed rectangles corresponding to the lead and the pad respectively comprises:

19. The method according to claim 1, wherein the determining the target defect type corresponding to the product to be detected according to the confirmation processing logic corresponding to the at least one defect type comprises:

20. The method according to claim 1, wherein the determining the target defect type corresponding to the product to be detected according to the confirmation processing logic corresponding to the at least one defect type comprises:

if the defect type with the highest probability value in the at least one defect type is a seventh defect type, and the probability value corresponding to the seventh defect type is smaller than a fourth preset value, inputting the image to be recognized into a second classification model;

21. The method of claim 19, wherein the seventh defect type comprises any one of the following types: and (5) cold joint and spot welding.

22. The method of claim 1 or 9 or 13 or 16 or 20, further comprising:

acquiring an identified image of a detected product with a target defect type of cold joint and no defect;

generating a semantic segmentation map corresponding to the identified image, wherein the semantic segmentation map comprises a lead and a welding spot of the product to be detected after a spot welding process;

determining an effective part in the identified image according to the minimum circumscribed rectangles corresponding to the lead and the welding points in the semantic segmentation graph;

and training to obtain the second classification model by taking the effective part as training data.

23. A product quality detection apparatus, comprising:

the type determining module is used for determining a target defect type corresponding to the product to be detected according to the confirmation processing logic corresponding to the at least one defect type;

the type determination module is to: if the defect type with the highest probability value in the at least one defect type is the first defect type, analyzing the image to be identified to determine whether lead indentation exists on the bonding pad welded with the product to be detected; if no lead indentation exists, determining that the target defect type corresponding to the product to be detected is the first defect type; if the lead indentation exists, inputting the image to be recognized into a second classification model; and determining the target defect type corresponding to the product to be detected according to the result output by the second classification model.