CN116559170A

CN116559170A - Product quality detection method and related system

Info

Publication number: CN116559170A
Application number: CN202210101996.XA
Authority: CN
Inventors: 王乾人; 钱颖; 葛江帆
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2022-01-27
Filing date: 2022-01-27
Publication date: 2023-08-08

Abstract

The application provides a product quality detection method, which is applied to the technical field of Artificial Intelligence (AI), and comprises the following steps: the method comprises the steps of obtaining an image to be detected, identifying the image to be detected, obtaining at least one first detected target and a first reference target in the image to be detected, determining a first relative position of the at least one first detected target relative to the first reference target, obtaining a second relative position of at least one second detected target in a template image relative to a second reference target in the template image, and carrying out quality detection on a product to be detected according to the first relative position and the second relative position to obtain a quality detection result. According to the method, the reference targets are respectively introduced into the image to be detected and the template image, and the relative positions of the detected target in the image to be detected and the detected target in the template image relative to the reference targets in the respective images are compared, so that quality detection is realized, image registration is avoided, false detection rate is reduced, and robustness is improved.

Description

Product quality detection method and related system

Technical Field

The present application relates to the field of artificial intelligence (artificial intelligence, AI) technology, and in particular, to a product quality detection method, a product quality detection system, and a computing device cluster, a computer-readable storage medium, and a computer program product.

Background

With the continuous development of artificial intelligence (artificial intelligence, AI) technology, particularly Deep Learning (DL) technology, quality detection of products based on deep learning is a trend in the field of industrial manufacturing. The quality detection of products based on deep learning refers to that images are acquired by a camera instead of human eyes, and quality detection standards for specific products are compiled into image processing rules by a deep learning algorithm so as to replace manual experience to realize automatic quality detection.

Quality testing of products based on deep learning typically involves two parts, offline processing and online processing. The off-line processing refers to selecting a qualified product as a template, generating a template image through camera shooting, then adopting a deep learning algorithm to identify a detected target in the template image, obtaining the outline of the detected target and the position of the detected target in a reference coordinate system of the template image, and recording the number of the detected targets in the template image and the positions of the detected targets in the reference coordinate system of the template image. The online processing means that a product to be detected is shot through a camera to generate an image to be detected, then a deep learning algorithm is adopted to identify the outline of a detected target in the image to be detected and the position of the detected target in a reference coordinate system of the image to be detected, affine transformation is carried out on the image to be detected, so that the reference coordinate system of the image to be detected is overlapped with the reference coordinate system of a template image, registration of the image to be detected and the template image is realized, and finally the detected target in the image to be detected and the detected target in the template image are matched. If the matching is successful, the product is qualified, and if the matching is unsuccessful, the product is unqualified.

However, in some scenarios, for example: under the scenes of product deviation, rotation and the like, the false detection rate of quality detection of products by the scheme in the prior art is higher.

Disclosure of Invention

The method comprises the steps of respectively introducing reference targets into an image to be detected and a template image, comparing the relative positions of the detected targets in the image to be detected and the template image relative to the reference targets in the respective images, and realizing quality detection, thereby reducing registration, further solving the problem of inaccurate affine transformation caused by product deviation or rotation, and reducing false detection rate. The application also provides a product quality detection system, a computing device cluster, a computer readable storage medium and a computer program product corresponding to the method.

In a first aspect, the present application provides a method for product quality detection. The method may be performed by a product quality detection system. In some embodiments, the product quality detection system may be a software system, the computing device or cluster of computing devices executing program code of the software system to perform the product quality detection method. In other embodiments, the product quality detection system may also be a hardware system for quality detection of a product, such as: industrial personal computers, servers, etc. The embodiment of the application uses a product quality detection system as a software system for illustration.

Specifically, the product quality detecting system acquires an image to be detected, which is an image obtained by photographing a product to be detected, then the product quality detecting system recognizes the image to be detected, obtains at least one detected target (in this application, the detected target in the image to be detected is also referred to as a first detected target) and a reference target (in some cases, also referred to as a first reference target) in the image to be detected, determines a relative position (for convenience of description, also referred to as a first relative position) of the at least one first detected target with respect to the first reference target, and further acquires a relative position (also referred to as a second relative position) of the at least one detected target (also referred to as a second detected target) in the template image with respect to the reference target (i.e., a second reference target) in the template image, and performs quality detection on the product to be detected based on the first relative position and the second relative position, thereby obtaining a quality detection result.

According to the method, the reference targets are respectively introduced into the image to be detected and the template image, the relative positions of the detected targets in the image to be detected and the template image relative to the reference targets in the respective images are compared, so that quality detection is realized, template registration is not required, affine transformation is not required, the problem that affine transformation is inaccurate due to the fact that a product is offset or a rotation angle is slightly large in the visual field of a camera is avoided, the false detection rate is reduced, and good robustness is achieved. In addition, affine transformation is not needed, so that the calculated amount is greatly reduced, the detection time is shortened, and the production requirement can be met.

In some possible implementations, when the image to be detected includes a plurality of first detected targets, the product quality detection system may perform quality detection on the product to be detected according to the number of the first detected targets in the image to be detected, the number of the second detected targets in the template image, the first relative position, and the second relative position, to obtain a quality detection result.

Specifically, the product quality detection system may first determine whether the number of first detected objects in the image to be detected is consistent with the number of second detected objects in the template image. If the first relative position and the second relative position are consistent, the first relative position and the second relative position are continuously matched, and a quality detection result is obtained according to the matching result, wherein the matching result represents that the first relative position and the second relative position are matched, the quality detection result is qualified for a product to be detected, the matching result represents that the first relative position and the second relative position are not matched, and the quality detection result is unqualified for the product to be detected; if the products are inconsistent, the products to be detected can be directly determined to be unqualified products.

Therefore, partial unqualified products (such as products with inconsistent numbers of the first detected targets and the second detected targets) can be screened out in advance, the quality detection efficiency is improved, and the production requirement is met.

In some possible implementations, the product quality detection system may match the first relative position with the second relative position to obtain a matching result. And when the matching result represents that the first relative position is not matched with the second relative position, the product quality detection system determines that the product to be detected is a disqualified product.

According to the method, the quality detection of the product can be realized by carrying out simple matching operation on the relative positions, and the quality detection efficiency is improved without carrying out a large amount of operation, so that the production requirement can be met, the quality detection can be realized without configuring high-performance hardware, and the detection cost is reduced.

In some possible implementations, the product quality detection system may determine a coordinate difference of the first relative position and the second relative position. And when the coordinate difference value is larger than a preset threshold value, the product quality detection system determines that the first relative position and the second relative position are not matched.

The coordinate difference may include a horizontal coordinate difference value and a vertical coordinate difference value. The product quality detection system can perform subtraction operation on the abscissa of the first relative position and the second relative position, and then take the absolute value, so as to obtain a horizontal coordinate difference value. Similarly, the product quality detection system may subtract the ordinate of the first relative position and the second relative position and then take the absolute value to obtain the ordinate difference. In some embodiments, the preset thresholds may include a first preset threshold and a second preset threshold. When the horizontal coordinate difference value is larger than a first preset threshold value or the vertical coordinate difference value is larger than a second preset threshold value, the product quality detection system can determine that the product to be detected is a disqualified product.

Compared with the scheme that the image to be detected and the template image are registered first and then target matching is carried out, the method is less influenced by product offset or rotation, so that the false detection rate is lower and the robustness is better.

In some possible implementations, the product quality detection system may identify the image to be detected through a deep learning algorithm, generate a candidate frame, classify and regress the candidate frame, and obtain a position of at least one first detected target in the image to be detected in a reference coordinate system of the image to be detected and a position of the first reference target in the image to be detected in the reference coordinate system of the image to be detected. Accordingly, the product quality detection system may obtain the first relative position based on the position of the at least one first detected object in the reference coordinate system of the image to be detected and the position of the first reference object in the reference coordinate system of the image to be detected.

When the product quality detection system identifies an image to be detected, features can be extracted through a deep learning algorithm to obtain a feature map, a plurality of candidate frames with different scales are generated for each feature point in the feature map, and then classification is carried out on whether the candidate frames comprise targets (such as a first detected target and a first reference target). Further, the product quality detection system may perform regression on the candidate frame including the target as the classification result, so as to obtain the position of the target in the reference coordinate system of the image to be detected.

According to the method, the positions of the first detected target and the first reference target in the reference coordinate system of the image to be detected are determined through a deep learning algorithm, then the first relative position is determined based on the positions of the first detected target and the first reference target in the reference coordinate system of the image to be detected, and quality detection is performed based on the first relative position and the second relative position, so that the method has high availability.

In some possible implementations, the product quality detection system may also present the quality detection result to the user, so as to prompt the user for information, so that the user can process the unqualified product in time, and meet the production requirement.

In some possible implementations, when the quality detection result characterizes the product to be detected as a defective product, the product quality detection system may further receive a processing instruction of the user on the product to be detected through a result display interface, so as to process the product to be detected according to the processing instruction. Wherein the process indication may be used to indicate that the reject is to be discarded or reworked. Therefore, unqualified products can be prevented from being mixed into qualified products, and the qualification rate of batch products is improved.

In some possible implementations, the first reference target is a boundary of the product to be detected and the second reference target is a boundary of the acceptable product. Taking the boundary of the product as a reference target can provide a reference for determining the relative position of the detected target.

In some possible implementations, the second relative position of the at least one second detected object in the template image with respect to the second reference object in the template image may be pre-identified and calculated. Specifically, the product quality detection system may acquire the template image in advance, then determine a second reference target in the template image, for example, determine a target specified by the user through the parameter configuration interface as the second reference target, then identify the template image through the deep learning algorithm, obtain at least one second detected target in the template image, and determine a second relative position of the at least one second detected target with respect to the second reference target. The product quality detection system may store the second relative position in the storage device, so that the second relative position is obtained from the storage device during subsequent quality detection, and quality detection is performed according to the first relative position and the second relative position.

In the method, the product quality detection system determines the second relative position of at least one second detected target relative to the second reference target in the template image by pre-identifying the template image, so that the calculated amount of subsequent quality detection can be reduced, and the quality detection efficiency is improved.

In some possible implementations, the product quality detection system may also identify the template image in real-time, obtain at least one second detected object and a second reference object (e.g., a user-specified object of the objects identified by the product quality detection system), and determine a second relative position of the at least one second detected object with respect to the second reference object. Therefore, the quality detection can be carried out on the product based on the first relative position and the second relative position, so that the false detection rate can be reduced on the one hand, and the quality detection efficiency can be improved on the other hand.

In a second aspect, the present application provides a product quality inspection system. The system comprises:

the interaction module is used for acquiring an image to be detected, wherein the image to be detected is an image obtained by shooting a product to be detected;

the identification module is used for identifying the image to be detected and obtaining at least one first detected target and a first reference target in the image to be detected;

The identification module is further used for determining a first relative position of the at least one first detected target relative to the first reference target;

the detection module is used for acquiring a second relative position of at least one second detected target in the template image relative to a second reference target in the template image, wherein the template image is an image obtained by shooting a qualified product, and quality detection is carried out on the product to be detected according to the first relative position and the second relative position to obtain a quality detection result.

In some possible implementations, the detection module is specifically configured to:

when the image to be detected comprises a plurality of first detected targets, quality detection is carried out on the product to be detected according to the number of the first detected targets in the image to be detected, the number of the second detected targets in the template image, the first relative position and the second relative position, and a quality detection result is obtained.

matching the first relative position with the second relative position to obtain a matching result;

And when the matching result represents that the first relative position is not matched with the second relative position, determining that the product to be detected is a disqualified product.

determining a coordinate difference of the first relative position and the second relative position;

and when the coordinate difference value is larger than a preset threshold value, determining that the first relative position and the second relative position are not matched.

In some possible implementations, the identification module is specifically configured to:

identifying the image to be detected through a deep learning algorithm, generating a candidate frame, classifying and regressing the candidate frame, and obtaining the position of at least one first detected target in the image to be detected in a reference coordinate system of the image to be detected and the position of the first reference target in the image to be detected in the reference coordinate system of the image to be detected;

and obtaining the first relative position according to the position of the at least one first detected object in the reference coordinate system of the image to be detected and the position of the first reference object in the reference coordinate system of the image to be detected.

In some possible implementations, the interaction module is further configured to:

and presenting the quality detection result to a user.

and when the quality detection result represents that the product to be detected is a disqualified product, receiving a processing instruction of the user on the product to be detected through a result display interface so as to process the product to be detected according to the processing instruction.

In some possible implementations, the first reference target is a boundary of the product to be detected and the second reference target is a boundary of the acceptable product.

In some possible implementations, the second relative position of the at least one second detected object in the template image with respect to the second reference object in the template image is pre-identified and calculated.

In a third aspect, the present application provides a cluster of computing devices. The cluster of computing devices includes at least one computing device including at least one processor and at least one memory. The at least one processor and the at least one memory are in communication with each other. The at least one processor is configured to execute instructions stored in the at least one memory to cause the computing device or cluster of computing devices to perform the product quality detection method as in the first aspect or any implementation of the first aspect.

In a fourth aspect, the present application provides a computer readable storage medium having stored therein instructions for instructing a computing device or a cluster of computing devices to perform the product quality detection method according to the first aspect or any implementation manner of the first aspect.

In a fifth aspect, the present application provides a computer program product comprising instructions which, when run on a computing device or cluster of computing devices, cause the computing device or cluster of computing devices to perform the product quality detection method of the first aspect or any implementation of the first aspect.

Further combinations of the present application may be made to provide further implementations based on the implementations provided in the above aspects.

Drawings

In order to more clearly illustrate the technical method of the embodiments of the present application, the drawings that are required to be used in the embodiments will be briefly described below.

Fig. 1 is a schematic architecture diagram of a product quality detection system according to an embodiment of the present application;

FIG. 2 is an interface schematic diagram of a parameter configuration interface according to an embodiment of the present disclosure;

FIG. 3 is an interface schematic diagram of a result display interface according to an embodiment of the present disclosure;

FIG. 4 is a flowchart of a method for detecting product quality according to an embodiment of the present disclosure;

FIG. 5 is a schematic diagram of identifying a template image to determine a second relative position according to an embodiment of the present application;

fig. 6 is a schematic flow chart of target matching according to an embodiment of the present application;

fig. 7 is a schematic flow chart of target matching according to an embodiment of the present application;

fig. 8 is a schematic structural diagram of a computing device cluster according to an embodiment of the present application.

Detailed Description

The terms "first", "second" in the embodiments of the present application are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include one or more such feature.

Some technical terms related to the embodiments of the present application will be first described.

Quality inspection, also known as quality inspection, refers to the detection of whether the inherent characteristics of a product meet the quality requirements (e.g., applicability requirements, safety requirements) specified by the relevant regulations. Wherein the relevant regulations may be one or more of legal regulations, industry regulations, customer regulations, etc.

Machine Learning (ML), a branch of artificial intelligence (artificial intelligence, AI), mainly uses automatic analysis from data to obtain rules and uses the rules to predict unknown data. Machine learning can be used to detect quality of corresponding products in the fields of electronics, semiconductors, home appliances, printing, automobiles, food packaging, etc., and this quality detection mode is also called machine vision-based quality detection.

Different machine learning algorithms are adopted, and different implementations are possible for quality detection of products based on machine vision. One typical example is quality detection of a product based on deep learning. Quality detection of products based on deep learning generally requires registration of the image to be detected with the template image by affine transformation after regression of the position of the object to be detected in the reference coordinate system of the image to be detected by deep learning. However, affine transformations are relatively sensitive to product offset and rotation. When the product is offset or the rotation angle is slightly large in the field of view of the camera, affine transformation is inaccurate, so that the false detection rate is increased, and the robustness of the quality detection method based on deep learning is affected. In addition, affine change also involves matrix calculation, which is time-consuming and difficult to meet production requirements.

In view of this, the embodiment of the application provides a product quality detection method. The method may be performed by a product quality detection system (sometimes referred to herein simply as a detection system for simplicity of expression). In some embodiments, the product quality detection system may be a software system, the computing device or cluster of computing devices executing program code of the software system to perform the product quality detection method. In other embodiments, the product quality detection system may also be a hardware system for quality detection of a product, such as: industrial personal computers (Industrial Personal Computer, IPC), servers, etc. The embodiment of the application uses a product quality detection system as a software system for illustration.

According to the method, the reference targets are respectively introduced into the image to be detected and the template image, the quality detection is realized by comparing the relative positions of the detected targets in the image to be detected and the template image relative to the reference targets in the respective images, template registration is not required, affine transformation is not required, and therefore the problem that affine transformation is inaccurate due to the fact that a product is offset or a rotation angle is slightly large in the visual field of a camera is avoided, the false detection rate is reduced, and the method has good robustness. In addition, affine transformation is not needed, so that the calculated amount is greatly reduced, the detection time is shortened, and the production requirement can be met.

The product quality detection method of the embodiment of the application can be suitable for different industries to detect the quality of different products. For example, the product quality detection method of the embodiment of the application can be applied to the field of semiconductors to detect quality of products such as display panels. For another example, the product quality detection method of the embodiment of the application can be applied to the field of food packaging and is used for quality detection of food packaging bags or food packaging boxes and the like.

In order to make the technical solution of the present application clearer and easier to understand, the following describes the product quality detection system of the embodiments of the present application in detail with reference to the accompanying drawings.

Referring to the architecture diagram of the product quality detection system shown in fig. 1, a product quality detection system 100 is deployed in a cloud environment 10. Wherein the cloud environment indicates a cluster of central computing devices owned by a cloud service provider for providing computing, storage, communication resources. The product quality detection system 100 may be deployed in particular in one or more computing devices of a central computing device cluster (e.g., a central server). The product quality inspection system 100 is used for quality inspection of a product 200 to be inspected (which may also be simply referred to as a product 200) produced on a production line 20.

The cameras 30 are installed on the production line 20, the cameras 30 are used for shooting the products 200 produced on the production line 20 to obtain images to be detected, the product quality detection system 100 deployed in the cloud environment 10 acquires the images to be detected, and the images to be detected are processed to realize quality detection of the products 200. Further, the product quality detection system 100 may also return quality detection results to the terminal 40 in order to present the quality detection results to a user (e.g., an operator).

In this embodiment, the product quality detection system 100 includes the following functional modules: an interaction module 102, an identification module 104, and a detection module 106. The interaction module 102 is configured to obtain an image to be detected, where the image to be detected is an image obtained by capturing the product 200. The interaction module 102 may provide a data uploading interface, and the camera 30 may report the image to be detected to the product quality detection system 100 through the data uploading interface. The identification module 104 is configured to identify an image to be detected, obtain at least one first detected object and a first reference object in the image to be detected, and then determine a first relative position of the at least one first detected object with respect to the first reference object. The detection module 106 is configured to obtain a second relative position of at least one second detected object in the template image with respect to the second reference object, and perform quality detection on the product 200 according to the first relative position and the second relative position, for example, match the first relative position with the second relative position, so as to obtain a quality detection result.

The template image is an image obtained by shooting a qualified product, and the qualified product is a product meeting the quality requirements specified by related regulations such as legal regulations and industry regulations. In some possible implementations, the second relative position of the at least one second detected object in the template image with respect to the second reference object may be identified and calculated in advance. Specifically, the interaction module 102 may also provide the code of the interaction interface, and the terminal 40 may load the code of the interaction interface to present the interaction interface to the user. The interactive interface may comprise an interface for configuring parameters of the quality detection algorithm, i.e. a parameter configuration interface. Wherein the parameters of the quality detection algorithm may include one or more of a template image, a reference target (e.g., a second reference target in the template image), etc.

Referring to the schematic diagram of the parameter configuration interface shown in fig. 2, the parameter configuration interface 201 carries a product model configuration control 202, a template image configuration control 203 and a reference target configuration control 204, and a user can configure a model of a product to be detected, such as the product 200, through the product model configuration control 202. Wherein the product model configuration control 202 has a drop-down box, and the user can configure the model of the product 200 by means of drop-down selection. In some embodiments, the user may also directly enter the model of the product 200. Next, the user can configure a template image corresponding to the model through the template image configuration control 203. In particular, template image configuration control 203 may enable a user to select one or more images in a file system as template images by browsing the file system. The parameter configuration interface 201 may also present a template image, and accordingly, when the reference target configuration control 204 is triggered, the user may select a target in the template image as a second reference target by clicking. For example, when the product 200 is a mobile phone, the user may select the lower frame of the mobile phone as the second reference target.

The parameter configuration interface 201 also carries a determination control 205 and a cancel control 206. Wherein upon determining that control 205 is triggered, the user-configured parameters described above are submitted to product quality detection system 100. Accordingly, in the offline stage, the recognition module 104 in the product quality detection system 100 may recognize the template image in advance to obtain at least one second detected target and a second reference target in the template image. Wherein the second reference object may be an object selected by the user from a plurality of objects identified by the identification module 104 from the template image. Further, the identification module 204 may determine a second relative position of the at least one second detected object with respect to the second reference object to thereby provide assistance in product quality detection. As shown in fig. 1, the dashed boxes in fig. 1 represent method steps performed in the offline phase, and the implementation boxes represent method steps performed in the online phase. When the cancel control 206 is triggered, then the parameter configuration described above may be canceled.

In some possible implementations, the interactive interface may also include an interface for presenting the quality test results, also referred to as a results presentation interface. Referring to the schematic diagram of the results display interface shown in fig. 3, the results display interface 301 includes quality detection results 302 of the product 200. The quality detection result 302 may be displayed by text and/or images. For example, when the quality detection result 302 is that the product 200 is not acceptable, the result display interface 301 may directly display the text "not acceptable", and display the reason of the failure by the image, for example, that the positional deviation of a certain detected object exceeds a specified value. Further, the result display interface 301 further carries an exception handling control 303, where the exception handling control 303 is configured to handle the unqualified product 200. For example, the exception handling control 303 may be used to indicate that the failed product 200 is to be discarded or reworked.

It should be noted that fig. 1 is only an illustration of the product quality detection system 100 deployed in the cloud environment 10, and in other possible implementations of the embodiments of the present application, the product quality detection system 100 may be deployed in an edge environment or an end-side device.

Wherein the edge environment indicates a cluster of edge computing devices geographically close to the end-side devices for providing computing, storage, communication resources. An edge computing device cluster includes one or more edge computing devices, which may be servers, computing boxes, or the like. The end-side device may be a terminal including, but not limited to, an industrial personal computer, desktop computer, notebook computer, or smart phone.

Fig. 1 illustrates one division of a product quality detection system 100, and in other possible implementations of embodiments of the present application, the product quality detection system 100 may be divided into different modules in other manners. The multiple modules of the product quality detection system 100 may also be distributed deployed in different environments, such as in a cloud environment and an edge environment.

Next, a product quality detection method provided in the embodiment of the present application will be described in detail from the point of view of the product quality detection system 100.

Referring to the flow chart of the product quality inspection method shown in fig. 4, the method can be divided into two stages, off-line and on-line. Wherein, in the off-line stage, the product quality detection system 100 receives the template image configured by the user, identifies the template image, obtains at least one second detected object and a second reference object in the template image, and determines a second relative position of the at least one second detected object with respect to the second reference object. In the online stage, the product quality detection system 100 acquires an image to be detected, identifies the image to be detected through a deep learning algorithm, acquires at least one first detected target and a first reference target in the image to be detected, determines a first relative position of the at least one first detected target relative to the first reference target, and performs target matching according to the first relative position and the second relative position, thereby realizing quality detection of the product. The following describes the steps involved in the method in detail:

s402: the product quality inspection system 100 acquires a template image.

In particular, the product quality detection system 100 may receive a user-configured template image. For example, the product quality inspection system 100 may present a parameter configuration interface to a user through which a user-configured template image is received. The template image may be an image obtained by photographing a composite product. In some embodiments, the user may screen the images of the good product from the history image to specify the template image. In other embodiments, the user may also select a good product, and capture the good product to obtain a template image.

It should be noted that the product quality inspection system 100 may be used to perform quality inspection at different stages of product production, and based on this, the qualified product may be a finished product or a semi-finished product. For example, when the product is a mobile phone, the product quality detection system 100 can perform product quality detection when the display screen of the mobile phone is produced and when the display screen and the circuit board are assembled into the mobile phone. Accordingly, the acceptable product may be an acceptable display screen or an acceptable complete machine.

S404: the product quality detection system 100 determines a second fiducial target in the template image.

The second reference object is a reference object in the template image and can be generally used as a reference object of the second detected object in the template image to determine the relative position of the second detected object. The reference target may be a boundary of the product or a component of the product. Based on this, the second reference target may be a boundary or part of a good product.

Specifically, the product quality detection system 100 may present a parameter configuration interface to the user, which may present a template image in which the user may specify the target, and the product quality detection system 100 determines the user-specified target as the second reference target.

S406: the product quality inspection system 100 identifies a template image to obtain at least one second inspected object.

The product quality detection system 100 may identify the template image by a deep learning algorithm to obtain a plurality of targets. Wherein, the target other than the second reference target is the second detected target. In this manner, the product quality inspection system 100 may obtain at least one second inspected object.

Specifically, the product quality detection system 100 may extract a feature map from a template image through a deep learning algorithm, generate a plurality of candidate frames for each feature point of the feature map, and then classify the candidate frames, for example, whether the candidate frames include a target, and for the classification result, the candidate frames including the target, the product quality detection system 100 regress the same, thereby obtaining the position of the target in the reference coordinate system of the template image.

The position of the object in the reference coordinate system of the template image comprises the position of the second reference object in the reference coordinate system of the template image and the position of the at least one second detected object in the reference coordinate system of the template image. Wherein the position may be characterized by coordinates of the second reference object and the at least one second detected object in a reference coordinate system.

S408: the product quality detection system 100 determines a second relative position of at least one second detected object with respect to a second reference object.

Specifically, the position of the second detected object may be characterized by coordinates of the feature point of the second detected object, and the position of the second reference object may be characterized by coordinates of the feature point of the second reference object. Wherein the feature points may be vertices or center points, etc. For each second detected object, the product quality detection system 100 may calculate a difference between the coordinates of the feature points of the second detected object and the coordinates of the feature points of the second reference object, thereby obtaining a second relative position of the second detected object with respect to the second reference object.

For ease of understanding, the following description is provided in connection with a specific example.

Referring to the schematic diagram of identifying a template image to determine a second relative position shown in fig. 5, the template image is an image of photographing a qualified product, which may be a multifunctional ruler having a hollow groove for drawing a triangle and an ellipse, the user may select an upper boundary of the multifunctional ruler as a second reference target, the product quality detection system 100 identifies the template image through a deep learning algorithm to obtain coordinates of feature points of the second reference target and the second detected target (including the triangle hollow groove and the ellipse hollow groove in the drawing), and then obtains a second relative position of the second detected target with respect to the second reference target according to a difference of the coordinates of the feature points. The feature point of the second reference object may be a left end point of an upper boundary of the product, that is, an upper left corner vertex of the product. The feature points of the two second detected objects may be center points of the two second detected objects. The second relative position of the two second detected objects with respect to the second reference object can be expressed as (x) ₁ ，y ₁ )、(x ₂ ，y ₂ )。

In some possible implementations, the product quality detection system 100 may also directly obtain the relative position of the second detected object with respect to the second reference object through the deep learning algorithm, without first regressing the coordinates of the second detected object and the second reference object in the reference coordinate system, and then calculating the relative position according to the coordinates. For example, the product quality detection system 100 may obtain annotation data annotated with a relative position, and employ a supervised learning training image recognition model by which the template image is recognized, thereby directly outputting the relative position of the second detected object in the template image with respect to the second reference object.

It should be noted that, the product quality detection system 100 may perform the above-mentioned S402 to S408 in advance, store the second relative position obtained in performing the S402 to S408 in the storage device, and when quality detection of the product to be detected is required, obtain the above-mentioned second relative position from the storage device so as to perform quality detection. In other embodiments, the product quality inspection system 100 may also identify the template image in real-time to obtain at least one second inspected object and a second reference object (the object specified by the user among the objects identified by the product quality inspection system 100), and determine a second relative position of the at least one second inspected object with respect to the second reference object. The product quality detection system 100 determines the second relative position in advance, so that the calculation amount of quality detection in the online stage can be reduced, the quality detection efficiency in the online stage can be improved, and the production requirement can be met.

It should be noted that, the product quality detection method according to the embodiment of the present application may not be executed in S402 to S408. For example, when the product quality detection system 100 is preconfigured with a relevant parameter (such as a reference value or a range of values of the relative position of the detected target with respect to the reference target), the product quality detection system 100 may not perform S402 to S408 described above.

S410: the product quality inspection system 100 acquires an image to be inspected.

The product quality detection system 100 is provided with a data uploading interface, and the product quality detection system 100 can receive an image to be detected reported by a camera through the data uploading interface. The image to be detected is an image obtained by shooting the product to be detected. In some embodiments, the product to be detected may run on a production line, and when the product to be detected reaches a shooting position, a camera may be triggered to shoot, and after the camera shoots the product to be detected, the obtained image is uploaded as the image to be detected to the product quality detection system 100.

Since the product quality inspection system 100 may be used for quality inspection at different stages of product production, the product to be inspected may be a finished product or a semi-finished product based thereon. For example, the product quality inspection system 100 may be a semi-finished product when performing quality inspection in an intermediate process of product production. The fork, for example, the product quality inspection system 100 may inspect the quality of the product at the end of its production process.

S412: the product quality inspection system 100 identifies an image to be inspected, and obtains a first reference target and at least one first inspected target.

The first reference target refers to a reference target in the image to be detected, similar to the second reference target and the second detected target, and can be generally used as a reference object of the first detected target in the image to be detected to determine the relative position of the first detected target. Based on this, the first reference object may be a boundary of the product to be detected, for example, an upper boundary, a lower boundary, a left boundary, or a right boundary of the product to be detected. It should be noted that the first reference target is generally consistent with the second reference target. For example, when the second reference object is the upper boundary of the acceptable product, the first reference object is the upper boundary of the product to be inspected. The first detected object may be a component or a boundary of the product to be detected.

In a specific implementation, the product quality detection system 100 may identify an image to be detected through a deep learning algorithm, generate a candidate frame, classify and regress the candidate frame, and obtain a position of at least one first detected target in the image to be detected in a reference coordinate system of the image to be detected and a position of the first reference target in the reference coordinate system of the image to be detected. For example, the product quality detection system 100 may extract features from an image to be detected by a deep learning algorithm to obtain a feature map, then generate a candidate frame based on the feature map, classify and regress the candidate frame by inference, thereby obtaining positions of a first reference target and at least one first detected target in the image to be detected. The positions of the first reference object and the at least one first detected object may in particular be positions of the first reference object and the at least one first detected object in a reference coordinate system of the image to be detected.

S414: the product quality detection system 100 determines a first relative position of at least one first detected object with respect to a first reference object.

Specifically, the position of the first detected object may be characterized by coordinates of the feature point of the first detected object, and the position of the first reference object may be characterized by coordinates of the feature point of the first reference object. Wherein the feature points may be vertices or center points, etc. The types of the characteristic points of the first detected object and the characteristic points of the second detected object are consistent, and the types of the characteristic points of the first reference object and the characteristic points of the second reference object are consistent. As illustrated in fig. 6, since the feature point of the second reference object in fig. 5 is the left end point of the upper boundary of the good product, the feature point of the second detected object is the center point of the detected object in the good product, and thus the feature point of the first reference object may be the left end point of the upper boundary of the product to be detected, and the feature point of the first detected object is the center point of the detected object in the product to be detected.

For each first detected object, the product quality detection system 100 may calculate a difference between the coordinates of the feature points of the first detected object and the coordinates of the feature points of the first reference object, thereby obtaining a first relative position of the first detected object with respect to the first reference object.

Fig. 6 illustrates quality inspection of a product, a camera captures an image of the product to be inspected to obtain an image of the product to be inspected, the product quality inspection system 100 recognizes the image to be inspected to obtain at least one first inspected object and a first reference object, and then determines a first relative position of the at least one first inspected object with respect to the first reference object, specifically a relative position (x 'of a center point of a triangular hollow groove of the product to be inspected with respect to a left end point of an upper boundary of the product to be inspected' ₁ ，y’ ₁ ) A relative position (x 'of the center point of the elliptical hollow groove of the product to be inspected with respect to the left end point of the upper boundary of the product to be inspected' ₂ ，y’ ₂ )。

It should be noted that, the product quality detection system 100 may directly obtain the relative position of the first detected object with respect to the first reference object through the deep learning algorithm, without first regressing the coordinates of the first detected object and the first reference object in the reference coordinate system, and then calculating the relative position according to the coordinates. For example, the product quality detection system 100 may acquire labeling data labeled with a relative position, and train an image recognition model by using supervised learning, and recognize an image to be detected through the image recognition model, so as to directly output a relative position of a first detected target in the image to be detected relative to a first reference target.

S416: the product quality detection system 100 determines whether the number of the first detected targets is consistent with the number of the second detected targets, and if so, S418 is executed; if not, S422 is performed.

Specifically, if the product to be detected is qualified, the number of the first detected targets in the image to be detected is consistent with the number of the second detected targets in the template image, based on which, the product quality detection system 100 may count the first detected targets in the template image first, count the second detected targets in the image to be detected, and determine whether the number of the first detected targets is consistent with the number of the second detected targets. If not, it indicates that the product to be detected is not acceptable, and S422 may be performed. If so, the product quality inspection system 100 may proceed to execute S418 for further inspection.

Note that, S416 may not be executed to execute the product quality detection method according to the embodiment of the present application. For example, the product quality detection system may directly perform S418 quality detection on the product. The product quality detection system 100 may screen out some unqualified products in advance by executing S416, so as to improve quality detection efficiency. Especially, under the condition that the image to be detected comprises a plurality of first detected targets, the product to be detected is initially detected according to the number of the first detected targets and the number of the second detected targets, so that the quality detection efficiency of the product can be effectively improved.

S418: the product quality detection system 100 matches the first relative position with the second relative position, and if so, S420 is executed, and if not, S422 is executed.

Specifically, the product quality detection system 100 may determine a coordinate difference of the first relative position and the second relative position. And when the coordinate difference value is larger than a preset threshold value, determining that the first relative position and the second relative position are not matched. The coordinate difference may include a horizontal coordinate difference value and a vertical coordinate difference value. And when the coordinate difference value is not greater than a preset threshold value, determining that the first relative position is matched with the second relative position.

In this embodiment, the product quality detection system 100 may perform subtraction on the abscissa of the first relative position and the second relative position, and then take the absolute value, thereby obtaining the abscissa difference value, and similarly, the product quality detection system 100 may perform subtraction on the ordinate of the first relative position and the second relative position, and then take the absolute value, thereby obtaining the ordinate difference value. And when the horizontal coordinate difference value is larger than a first preset threshold dx or the vertical coordinate difference value is larger than a second preset threshold dy, determining that the first relative position and the second relative position are not matched. The first preset threshold and the second preset threshold may be user-preconfigured or system preset values.

When the product quality detection system 100 determines that the first relative position of the first detected object with respect to the first reference object and the second relative position of the second detected object with respect to the second reference object do not match, then the product to be detected may be determined to be a defective product. I.e. the quality test result is not acceptable.

When the product quality detection system 100 determines that the first relative position of each first detected object with respect to the first reference object and the second relative position of the second detected object with respect to the second reference object match, then the product to be detected may be determined to be a good product. I.e. the quality detection result is qualified.

Fig. 6 and 7 are respectively exemplary. In the example of fig. 6, the number of first detected objects (referred to as object 1 and object 2) in the image to be detected is identical to the number of second detected objects (both of which are 2) in the template image, and the relative position of the object 1 in the image to be detected with respect to the first reference object is (x' ₁ ，y’ ₁ ) The relative position of the object 2 with respect to the first reference object is (x' ₂ ，y’ ₂ ) Whereas the relative position of the object 1 with respect to the second reference object in the template image is (x 1, y 1), the relative position of the object 2 with respect to the second reference object The position is (x) ₂ ，y ₂ ) The difference in coordinates between the first relative position of the object 1 and said second relative position is smaller than a preset threshold, i.e. |x ₁ -x’ ₁ |＜dx，|y ₁ -y’ ₁ I < dy. Similarly, the difference in coordinates of the first and second relative positions of the object 2 is smaller than a preset threshold, i.e., |x ₂ -x’ ₂ |＜dx，|y ₂ -y’ ₂ I < dy. In this manner, the product quality inspection system 100 may determine that the product to be inspected is a good product.

In the example of fig. 7, the number of first detected objects (referred to as object 1 and object 2) in the image to be detected is identical to the number of second detected objects (both of which are 2) in the template image, and the relative position of the object 1 in the image to be detected with respect to the first reference object is (x' ₁ ，y” ₁ ) The relative position of the object 2 with respect to the first reference object is (x' ₂ ，y” ₂ ) Whereas the relative position of the object 1 in the template image with respect to the second reference object is (x) ₁ ，y ₁ ) The relative position of the object 2 with respect to the second reference object is (x ₂ ，y ₂ ) The difference in coordinates between the first relative position of the object 1 and said second relative position is smaller than a preset threshold, i.e. |x ₁ -x” ₁ |＜dx，|y ₁ -y” ₁ | < dy, however, the difference in coordinates of the first relative position of the object 2 and the second relative position is greater than a preset threshold, i.e., |x ₂ -x” ₂ |＞dx，|y ₂ -y” ₂ | > dy. In this manner, the product quality inspection system 100 may determine that the product to be inspected is a defective product.

It should be noted that, S416 to 418 are only one implementation manner of the product quality detection system 100 according to the first relative position and the second relative position in the embodiment of the present application to perform quality detection on a product to be detected, and in other possible implementation manners of the embodiment of the present application, the product quality detection system 100 may also directly perform target matching according to the first relative position and the second relative position, so as to perform quality detection on the product to be detected.

S420: the product quality inspection system 100 determines that the quality inspection result is acceptable.

S422: the product quality inspection system 100 determines that the quality inspection result is not acceptable.

In some possible implementations, the product quality detection system 100 may also present quality detection results to the user. For example, the product quality detection system 100 may present quality detection results to a user via a results presentation interface. Further, when the quality detection result characterizes that the product to be detected is a defective product, the product quality detection system 100 may further receive a processing instruction of the user on the product to be detected through the result display interface, so as to process the product to be detected according to the processing instruction. Wherein the processing instruction may be an instruction to discard the defective product or to rework the defective product, which is not limited in this embodiment.

In some possible implementations, the product quality detection system 100 may be connected to a control device of the production line, and when the product quality detection system 100 receives a processing instruction of a product to be detected by a user through the result display interface, the processing instruction may be sent to the control device, so that the control device performs corresponding processing on the product to be detected according to the processing instruction.

Based on the above description, the embodiment of the application provides a product quality detection method. According to the method, the reference targets are respectively introduced into the image to be detected and the template image, the quality detection is realized by comparing the relative positions of the detected targets in the image to be detected and the template image relative to the reference targets in the respective images, template registration is not required, affine transformation is not required, and therefore the problem that affine transformation is inaccurate due to the fact that a product is offset or a rotation angle is slightly large in the visual field of a camera is avoided, the false detection rate is reduced, and good robustness is achieved. In addition, affine transformation is not needed, so that the calculated amount is greatly reduced, the detection time is shortened, and the production requirement can be met.

Based on the product quality detection method provided in the embodiments of the present application, the embodiments of the present application further provide a product quality detection system 100 as described above. The product quality inspection system 100 provided in accordance with the embodiments of the present application will be described with reference to the accompanying drawings.

Referring to the schematic structural diagram of the product quality detection system shown in fig. 1, the system 100 includes: an interaction module 102, an identification module 104, and a detection module 106. The interaction module 102 is specifically configured to execute S410 in the foregoing method flow of fig. 4, the identification module 104 is configured to execute S412 and S414 in the foregoing method flow of fig. 4, and the detection module 106 is configured to acquire a second relative position of at least one second detected target in the template image with respect to the second reference target, and then obtain a quality detection result according to the first relative position and the second relative position obtained by executing S414.

In some possible implementations, the image to be detected includes a plurality of first detected objects, and accordingly, the detection module 106 is specifically configured to perform quality detection on the product to be detected according to the number of the first detected objects in the image to be detected, the number of the second detected objects in the template image, the first relative position, and the second relative position, to obtain a quality detection result.

In some possible implementations, the detection module 104 is specifically configured to:

In a specific implementation, the detection module 104 may execute S416 and S418 in the foregoing flow of the method in fig. 4, so as to implement quality detection on the product to be detected. In the method, the product to be detected can be initially screened by executing the step S416, so that partial unqualified products are screened in advance, and the quality detection efficiency is improved.

In some possible implementations, the detection module 106 may first determine a coordinate difference between the first relative position and the second relative position when executing S418, and determine that the first relative position and the second relative position do not match when the coordinate difference is greater than a preset threshold.

In some possible implementations, when executing S412, the identifying module 104 may identify the image to be detected through a deep learning algorithm, generate a candidate frame, classify and regress the candidate frame, and obtain a position of at least one first detected target in the image to be detected in a reference coordinate system of the image to be detected and a position of the first reference target in the image to be detected in the reference coordinate system of the image to be detected;

Accordingly, the identifying module 104 may obtain the first relative position according to the position of the at least one first detected object in the reference coordinate system of the image to be detected and the position of the first reference object in the reference coordinate system of the image to be detected when executing S414.

In some possible implementations, the interaction module 102 is further configured to: and presenting the detection result to a user.

In some possible implementations, the interaction module 102 is further configured to: and when the quality detection result represents that the product to be detected is a disqualified product, receiving a processing instruction of the user on the product to be detected through a result display interface so as to process the product to be detected according to the processing instruction.

The product quality detection system 100 according to the embodiments of the present application may correspond to performing the methods described in the embodiments of the present application, and the above and other operations and/or functions of the respective modules/units of the product quality detection system 100 are respectively for implementing the respective flows of the respective methods in the embodiments shown in fig. 4, and are not repeated herein for brevity.

The embodiment of the application also provides a computing device cluster. The cluster of computing devices includes at least one computing device, any of which may be from a cloud environment or an edge environment, or may be a terminal. The cluster of computing devices is particularly adapted to implement the functionality of the product quality detection system 100 in the embodiment shown in fig. 1.

Fig. 8 provides a schematic diagram of a computing device cluster, as shown in fig. 8, where computing device cluster 80 includes multiple computing devices 800, and computing device 800 includes a bus 801, a processor 802, a communication interface 803, and a memory 804. Communication between the processor 802, the memory 804 and the communication interface 803 is via the bus 801.

Bus 801 may be a peripheral component interconnect standard (peripheral component interconnect, PCI) bus or an extended industry standard architecture (extended industry standard architecture, EISA) bus, among others. The buses may be divided into address buses, data buses, control buses, etc. For ease of illustration, only one thick line is shown in fig. 8, but not only one bus or one type of bus.

The processor 802 may be any one or more of a central processing unit (central processing unit, CPU), a graphics processor (graphics processing unit, GPU), a Microprocessor (MP), or a digital signal processor (digital signal processor, DSP).

The communication interface 803 is used for communication with the outside. For example, the communication interface 803 is configured to obtain an image to be detected, for example, receive a reported image to be detected through a data uploading interface, present a quality detection result to a user, receive a processing instruction of a product to be detected through a result display interface from the user, and so on.

The memory 804 may include volatile memory (RAM), such as random access memory (random access memory). The memory 804 may also include non-volatile memory (ROM), such as read-only memory (ROM), flash memory, hard Disk Drive (HDD), or solid state drive (solid state drive, SSD).

The memory 804 has stored therein computer readable instructions that are executed by the processor 802 to cause the cluster of computing devices 80 to perform the aforementioned product quality detection method (or to implement the functionality of the aforementioned product quality detection system 100).

In particular, in the case of implementing the embodiment of the system shown in fig. 1, and in the case where the functions of the modules of the product quality detection system 100 described in fig. 1, such as the interaction module 102, the identification module 104, and the detection module 106, are implemented by software, software or program code required to perform the functions of the modules in fig. 1 may be stored in at least one memory 804 in the computing device cluster 80. The at least one processor 802 executes program code stored in the memory 804 to cause the cluster of computing devices 80 to perform the aforementioned product quality detection method.

Embodiments of the present application also provide a computer-readable storage medium. The computer readable storage medium may be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk), etc. The computer-readable storage medium includes instructions that instruct a computing device or cluster of computing devices to perform the above-described product quality detection method.

Embodiments of the present application also provide a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, the processes or functions described in accordance with the embodiments of the present application are produced in whole or in part. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website, computing device, or data center to another website, computing device, or data center by a wired (e.g., coaxial cable, fiber optic, digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer program product may be a software installation package that can be downloaded and executed on a computing device or cluster of computing devices in the event that any of the aforementioned methods of product quality detection are desired.

The descriptions of the processes or structures corresponding to the drawings have emphasis, and the descriptions of other processes or structures may be referred to for the parts of a certain process or structure that are not described in detail.

Claims

1. A method for detecting product quality, the method comprising:

acquiring an image to be detected, wherein the image to be detected is an image obtained by shooting a product to be detected;

identifying the image to be detected, and obtaining at least one first detected target and a first reference target in the image to be detected;

determining a first relative position of the at least one first detected object with respect to the first reference object;

acquiring a second relative position of at least one second detected target in a template image relative to a second reference target in the template image, wherein the template image is an image obtained by shooting a qualified product;

and carrying out quality detection on the product to be detected according to the first relative position and the second relative position to obtain a quality detection result.

2. The method according to claim 1, wherein when the image to be detected includes a plurality of first detected objects, the quality detecting the product to be detected according to the first relative position and the second relative position, to obtain a quality detection result, includes:

And carrying out quality detection on the product to be detected according to the number of the first detected targets in the image to be detected, the number of the second detected targets in the template image, the first relative position and the second relative position, and obtaining a quality detection result.

3. The method according to claim 1 or 2, wherein the quality detection of the product to be detected according to the first relative position and the second relative position, to obtain a quality detection result, includes:

4. A method according to claim 3, wherein said matching said first relative position with said second relative position comprises:

5. The method according to any one of claims 1 to 4, wherein said identifying the image to be detected, obtaining at least one first detected object and a first reference object in the image to be detected, comprises:

the determining a first relative position of the at least one first detected object with respect to the first reference object comprises:

6. The method according to any one of claims 1 to 5, further comprising:

and presenting the quality detection result to a user.

7. The method of claim 6, wherein the quality inspection result characterizes the product to be inspected as a reject product, the method further comprising:

and receiving a processing instruction of the user to the product to be detected through a result display interface so as to process the product to be detected according to the processing instruction.

8. The method according to any one of claims 1 to 7, wherein the first reference target is a boundary of the product to be inspected and the second reference target is a boundary of the acceptable product.

9. The method according to any one of claims 1 to 8, wherein the second relative position of at least one second detected object in the template image with respect to a second reference object in the template image is pre-identified and calculated.

10. A product quality inspection system, the system comprising:

11. The system according to claim 10, wherein the detection module is specifically configured to:

12. The system according to claim 10 or 11, wherein the detection module is specifically configured to:

13. The system according to claim 12, wherein the detection module is specifically configured to:

14. The system according to any one of claims 10 to 13, wherein the identification module is specifically configured to:

15. The system of any one of claims 10 to 14, wherein the interaction module is further configured to:

and presenting the quality detection result to a user.

16. The system of claim 15, wherein the interaction module is further configured to:

17. The system of any one of claims 10 to 16, wherein the first reference target is a boundary of the product to be inspected and the second reference target is a boundary of the acceptable product.

18. The system of any one of claims 10 to 17, wherein the second relative position of at least one second detected object in the template image with respect to a second reference object in the template image is pre-identified and calculated.

19. A cluster of computing devices, characterized in that it comprises at least one computing device comprising at least one processor and at least one memory, the at least one memory having stored therein computer-readable instructions that are executed by the at least one processor to cause the cluster of computing devices to perform the method of any of claims 1 to 9.

20. A computer-readable storage medium comprising computer-readable instructions that, when run on a computing device or cluster of computing devices, cause the computing device or cluster of computing devices to perform the method of any of claims 1-9.

21. A computer program product comprising computer readable instructions which, when run on a computing device or cluster of computing devices, cause the computing device or cluster of computing devices to perform the method of any of claims 1 to 9.