WO2024044915A1

WO2024044915A1 - Image comparison method and apparatus for error detection, and computer device

Info

Publication number: WO2024044915A1
Application number: PCT/CN2022/115623
Authority: WO
Inventors: 刘宁; 李岩
Original assignee: 西门子股份公司; 西门子（中国）有限公司
Priority date: 2022-08-29
Filing date: 2022-08-29
Publication date: 2024-03-07

Abstract

Disclosed in the present application are an image comparison method and apparatus for error detection, and a computer device, a storage medium and a computer program product. The image comparison method disclosed in the present application comprises: by using a recognition network based on a variational autoencoder, performing decoupling on a target image and a standard image, so as to obtain illumination-invariant features and illumination features, wherein the target image and the standard image are associated with relative positions of elements to be checked, and the illumination-invariant features are associated with geometric features of said elements; by using a mutual information gap, selecting the illumination-invariant features from the illumination-invariant features and the illumination features; comparing the illumination-invariant features of the target image and the illumination-invariant features of the standard image, so as to determine the similarity therebetween; and determining, according to whether the similarity is less than a threshold value, whether there is an error occurring in said elements, which are associated with the target image.

Description

Method, device, computer equipment for image comparison for error detection

Technical field

The present application relates to the field of image processing technology, and specifically, to a method, device, computer equipment, storage medium and computer program product for image comparison for error detection.

Background technique

Printed circuit board assembly defect detection is a key technology in the field of quality control. Due to the changeable appearance and dense interfaces of components on circuit boards, various assembly defects are prone to occur during manual assembly. Therefore, circuit board assembly defect detection has become particularly important. .

Currently, components assembled on PCB boards need to be inspected. Typically, the detection environment changes with lighting conditions. Existing detection methods cannot cope with this problem well. For example, under certain lighting conditions, components that were installed correctly can be misidentified.

Contents of the invention

This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify any key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

Based on this, this application proposes an image comparison method, including: using a variational autoencoder recognition network to decouple illumination invariant features and illumination features from the target image and the standard image, wherein the target image and The standard image is associated with the relative position between the components to be verified, and the illumination invariant features are associated with the geometric features of the components to be verified; using mutual information intervals, from the illumination invariant features and the illumination Select the illumination-invariant feature among features; compare the similarity between the illumination-invariant feature of the target image and the illumination-invariant feature of the standard image; determine whether the similarity is less than a threshold. Whether the element to be verified associated with the target image has errors.

In this way, images can be accurately compared under different lighting conditions and errors in the images can be identified.

Optionally, in an example of the above aspect, the recognition network is obtained through deep learning and is an approximation of the true posterior distribution. During the deep learning process of the recognition network:

Maximize the probability of producing a true image while keeping the distance between the true posterior distribution and the estimated posterior distribution less than the threshold.

Optionally, in an example of the above aspect, during the deep learning process of the recognition network:

Jointly optimize the loss function using fully differentiable estimators.

Optionally, in an example of the above aspect, using a mutual information interval to select the illumination-invariant feature from the illumination-invariant feature and the illumination-invariant feature includes:

The illumination-invariant features are selected from the illumination-invariant features and the illumination-invariant features using a joint distribution to estimate mutual information intervals between latent variables and gold standard factors.

Optionally, in an example of the above aspect, comparing the similarity between the illumination-invariant features of the target image and the illumination-invariant features of the standard image includes:

The similarity between the illumination-invariant features of the target image and the illumination-invariant features of the standard image is compared by comparing cosine distances between the illumination-invariant features.

This application also proposes an image comparison device, including: a decoupling module for using a variational autoencoder recognition network to decouple illumination invariant features and illumination features from the target image and the standard image, where The target image and the standard image are associated with the relative positions of the components to be verified, and the illumination invariant features are associated with the geometric features of the components to be verified; a selection module is used to use mutual information intervals to select from the Select the illumination invariant feature from the illumination invariant feature and the illumination feature; a comparison module for comparing the illumination invariant feature of the target image with the illumination invariant feature of the standard image a similarity; a determining module configured to determine, in response to the similarity being less than a threshold, that the position of the element to be verified associated with the target image is incorrect.

This application also provides a computer device, including a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the above method is implemented.

This application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented.

This application also provides a computer program product, which includes computer instructions, and the computer instructions instruct a computing device to perform the above method.

Description of drawings

Implementations of the present disclosure are illustrated by way of example and not by way of limitation in the accompanying drawings, in which like reference numbers refer to the same or similar parts.

FIG. 1 is a flow chart of an image comparison method for error detection according to an embodiment of the present application.

FIG. 2 is a schematic diagram of an image comparison device for error detection according to an embodiment of the present application.

3 is a schematic diagram of a computer device for image comparison of error detection according to an embodiment of the present application.

Among them, the reference signs are as follows:

S101-S104: Steps

200: Image comparison device

201: Decoupled module

202: Select module

203: Comparison module

204: Determine module

300: Computer equipment

302: Processor

304: Memory

Detailed ways

In the following specification, numerous specific details are set forth for purposes of explanation. However, it is understood that the invention may be practiced without these specific details. In other instances, well-known circuits, structures, and techniques have not been shown in detail so as not to obscure the understanding of the description.

References throughout this specification to "one implementation," "an implementation," "an example implementation," "some implementations," "various implementations," etc., mean that the described implementations of the invention may include particular features, structures, or Features, however, do not imply that every implementation must contain these specific features, structures, or characteristics. Furthermore, some implementations may have some, all, or none of the features described for other implementations.

In the following description, the terms "coupling" and "connection" and their derivatives may be used. It needs to be understood that these terms are not intended as synonyms for each other. Rather, in a particular implementation, "connected" is used to indicate that two or more components are in direct physical or electrical contact with each other, and "coupled" is used to indicate that two or more components cooperate or interact with each other, however they may, There may also be no direct physical or electrical contact. FIG. 1 is a flowchart of a method of image comparison for error detection according to an embodiment of the present application.

The circuit board assembly process includes the circuit design stage, the circuit board assembly stage, the assembly defect detection stage, and the assembly defect correction stage.

Specifically, during the circuit design stage, software such as Electronic Design Automation (EDA) can be used to perform logical simulation and physical simulation to obtain component layout examples. Then, in the circuit board assembly stage, preprocessing of the printed circuit board and assembly of components on the circuit board are performed based on the component layout instance. In the preprocessing of the printed circuit board, based on the component layout instance, tasks such as drilling are performed. process, etching process, etc. Operations performed during the assembly process include welding, placement of components, etc. After the assembly process, that is, in the assembly defect detection stage, assembly defect detection can be performed on the assembled circuit board, including angular defects, position defects, missing defects, flip defects, etc. of components. Further, in the assembly defect correction stage, the components to be corrected can be marked, and then returned to the circuit board assembly stage for reassembly. It should be understood that the cycle from the circuit board assembly stage to the assembly defect correction stage can be one or more times. When assembly defect detection is performed multiple times, the assembly accuracy of the printed circuit board can be improved. More specifically, during the circuit board assembly stage, component characteristics can be compared between the pre-made circuit board template and the on-site circuit board to obtain defective components.

Among them, the assembly defect detection stage includes the image preprocessing stage and the on-site inspection stage. Part of the data generated in the image preprocessing stage is needed in the on-site detection stage.

In this example, target detection is performed on the circuit board image (an example of the circuit board image to be detected) to obtain the component area. Specifically, the circuit board image can be input into the component detection module and the feature detection module, and the detected component area can be output.

Then, the component area is input into the affine transformation module for affine transformation to obtain the registered component area, that is, the component area after affine transformation. It should be understood that, on the one hand, after target detection, affine transformation is performed to ensure accurate target detection efficiency. In other words, deformation will occur in the component area after affine transformation, and the samples for training the target detection model are usually based on real data, thereby ensuring the accuracy of target detection, that is, ensuring the global detection accuracy of the component area. On the other hand, the affine transformed component area is more conducive to the identification of local defects. That is to say, when performing consistency detection in the consistency detection module, the affine transformed component area is compared with the components of the template image. Compare the areas to determine whether the components corresponding to the affine transformed component areas have assembly defects.

Further, the component area of the template image can be generated from the image preprocessing stage. The component area can be characterized by structural description and image features. The structural description can be obtained by inputting the template image into the component detection module and feature detection module. , further, perform processing based on the structural description to obtain the image features of the component area.

Further, the consistency detection module can perform at least one of angle defects, missing defects, position defects, and flip defects of components.

This application improves the consistency detection method.

FIG. 1 is a flowchart of an image comparison method for error detection according to an embodiment of the present application. Method 100 begins with step 101 . In step 101, use the recognition network of the variational autoencoder to decouple the illumination invariant features and illumination features from the target image and the standard image, where the target image, the standard image and the element to be verified are The relative position of the illumination invariant feature is associated with the geometric feature of the element to be verified.

This application proposes an unsupervised deep learning method that only extracts illumination-invariant features. Here, illumination-invariant features refer to features that have not changed under different lighting conditions, such as geometric features.

This method uses a probabilistic model, where the probabilistic model is used for latent representation learning. The model used in this method includes an encoder and a decoder, where the encoder can be called a recognition network and the decoder can be called a production network. The role of the encoder is to map the image into a low-dimensional latent space.

The recognition network is an approximation of the true posterior distribution p _θ (z|x)

where z is a set of latent variables and x is a set of features of the input image. Here, z is the variable we wish to learn.

Hidden variables are opposite to observed variables and refer to unobservable random variables. Latent variables can be inferred from observed data using mathematical models.

During the training process, we hope to maximize the probability of generating real images while keeping the distance between the real posterior distribution and the estimated posterior distribution less than the threshold ε.

and

In an embodiment, the above formula can be rewritten into the following form. The parameter β is added here.

where z ⁽ⁱ⁾ =m ⁽ⁱ⁾ + (σ ⁽ⁱ⁾ *e ⁽ⁱ⁾ ) and

Formula 3 is the loss function. The loss function has two terms, among which the first term is the KL divergence. To optimize the second term, a fully differentiable estimator is used, with parameters

and θ can be jointly optimized. Among them, the loss function refers to a function that maps an event to a real number of economic costs related to the event.

In one implementation, β is added to the standard variational bounds to simulate the redundancy reduction that allows learning of the decoupled latent space. Because variational autoencoders are able to achieve competitive decoupling, more complex data sets require stronger constraints to achieve interpretable feature separation.

In one implementation, a simple penalty term in the loss function can achieve highly decoupled features. β adjusts the learning constraint imposed on the model, where the constraint imposes a limit on the capacity of the latent information channel. In one implementation, β > 1 and this parameter should be tuned during training. In another implementation, β can be estimated using a decoupling metric.

In an implementation, the architecture of the encoder and decoder can change. For example, a convolutional neural network and a fully connected network can be used as the encoder, and a convolutional neural network can be used as the decoder.

In one implementation, the latent variable z is sampled from a unit Gaussian prior and then fed back through the decoder. Here, the generation network will generate the image.

In one implementation, a representation can be learned in which illumination latent features are sensitive to illumination changes while being insensitive to geometric features. Images can be collected under various lighting conditions. In addition, a synthetic lighting mixer can be used to synthesize real industrial images.

Steps go to 102. In step 102, it is determined which features in z are illumination-invariant features. In one implementation, mutual information intervals can be used to select features. can use

The defined joint distribution estimates the empirical mutual information interval between the latent variable z _j and the gold standard factor v _k .

Higher mutual information means that z _j contains a lot of information about v _k , and when there is a certain irreversible relationship between zj and vj, the mutual information is maximum. A single factor can have high mutual information values for multiple latent variables.

In one implementation, axis alignment can be achieved by measuring the difference between the two latent variables with the highest mutual information.

The full measure of calling mutual information intervals is as follows

in

j ^(k) = argmax _j I _n (z _j ; v _k );

The value range of the mutual information interval is between 0 and 1. In one implementation, MIG is estimated by traversing the data set.

In one implementation, z0, z1, z2 have the highest scores and, therefore, are illumination-invariant features.

Go to step 103. In step 103, the similarity between the illumination-invariant features of the target image and the illumination-invariant features of the standard image is compared. In one implementation, the similarity between the illumination-invariant features of the target image and the illumination-invariant features of the standard image is compared by comparing cosine distances between the illumination-invariant features.

Steps go to 104. In step 104, in response to the similarity being less than a threshold, it is determined whether the element to be verified associated with the target image has an error.

It should be understood that although various steps in the flowchart of FIG. 1 are shown in sequence as indicated by arrows, these steps are not necessarily executed in the order indicated by arrows. Unless otherwise specified in this article, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in Figure 1 may include multiple steps or stages. These steps or stages are not necessarily executed at the same time, but may be executed at different times. The execution order of these steps or stages is also It does not necessarily need to be performed sequentially, but may be performed in turn or alternately with other steps or at least part of steps or stages in other steps.

Figure 2 provides an apparatus 200 for image comparison for error detection. In the device 200, it includes: a decoupling module 201, used to decouple illumination invariant features and illumination features from the target image and the standard image using the recognition network of the variational autoencoder, wherein the target image is associated with the relative position between the standard image and the component to be verified, and the illumination invariant features are associated with the geometric features of the component to be verified; the selection module 202 is configured to use the mutual information interval to select from the illumination Select the illumination invariant feature from the invariant features and the illumination feature; the comparison module 203 is used to compare the illumination invariant feature of the target image with the illumination invariant feature of the standard image. Similarity; determining module 204, configured to determine, in response to the similarity being less than a threshold, that the position of the element to be verified associated with the target image is incorrect.

In one implementation, the recognition network is an approximation of the true posterior distribution, and the decoupling module of the device is further configured to obtain the recognition network in the following manner:

In an implementation, the decoupling module of the device is further configured to obtain the identification network in the following manner:

Jointly optimize the loss function using fully differentiable estimators.

In one implementation, the selection module of the device is further configured to use a joint distribution to estimate the mutual information interval between the latent variable and the gold standard factor, and select the illumination invariant feature and the illumination feature. Illumination invariant features.

In one implementation, the comparison module of the device is further configured to compare the illumination-invariant features of the target image with the illumination of the standard image by comparing cosine distances between the illumination-invariant features. Similarity between invariant features.

It should be noted that the device may contain more or fewer modules to implement the described functionality. For example, at least one module in Figure 2 may be further divided into a plurality of different sub-modules, each sub-module being configured to perform at least a portion of the operations described herein in connection with the corresponding module. In addition, in some examples, the apparatus 200 may also include additional modules for performing other operations that have been described in the specification. Furthermore, those skilled in the art will understand that the exemplary apparatus 200 may be implemented in software, hardware, firmware, or any combination thereof.

Figure 3 provides a computer device for image comparison for error detection. According to one embodiment, computer device 300 may include processor 302 that executes a computer program stored in memory 304. The computer program when executed by a processor implements an image comparison method for error detection.

Those skilled in the art can understand that the structure shown in Figure 3 is only a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer equipment to which the present application is applied. Specific computer equipment may include There may be more or fewer parts than shown, or certain parts may be combined, or may have a different arrangement of parts.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of implementing the above embodiments can be completed by instructing relevant hardware through computer programs. The computer programs can be stored in a non-volatile computer and can be read. In the storage medium, when executed, the computer program may include the processes of the above method embodiments. Any reference to memory, storage, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in many forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM).

The present application also provides a computer-readable storage medium on which a computer program is stored, which is characterized in that the above steps are implemented when the computer program is executed by a processor. Some implementations of the disclosure may include articles of manufacture. The article of manufacture may include storage media for storing logic. Examples of storage media may include one or more types of computer-readable storage media capable of storing electronic data, including volatile or nonvolatile memory, removable or non-removable memory, erasable or non-erasable memory memory, writable or rewritable memory, etc. Examples of logic may include various software units, such as software components, programs, applications, computer programs, applications, system programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions , methods, procedures, software interfaces, application programming interfaces (APIs), sets of instructions, computational code, computer code, code segments, computer code segments, words, values, symbols, or any combination thereof. In some implementations, for example, an article of manufacture may store executable computer program instructions that, when executed by a processor, cause the processor to perform the methods and/or operations described herein. Executable computer program instructions may include any suitable type of code, such as source code, compiled code, interpreted code, executable code, static code, dynamic code, etc. Executable computer program instructions may be implemented in accordance with a predefined computer language, manner, or syntax for commanding a computer to perform specific functions. The instructions may be implemented using any suitable high-level, low-level, object-oriented, visual, compiled and/or interpreted programming language.

Embodiments of the present application also provide a computer program product, including computer instructions, which instruct the computing device to perform any corresponding operation in the multiple method embodiments mentioned above. The above-mentioned methods according to the embodiments of the present application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or by The computer code downloaded by the network is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium, so that the method described here can be stored using a general-purpose computer, a special-purpose processor or a programmable computer. or such software processing on a recording medium of dedicated hardware such as ASIC or FPGA. It will be understood that a computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code when the software or computer code is used by the computer, When accessed and executed by a processor or hardware, the methods described herein are implemented. Furthermore, when a general-purpose computer accesses code for implementing the methods illustrated herein, execution of the code converts the general-purpose computer into a special-purpose computer for performing the methods illustrated herein.

What has been described above includes examples of the disclosed architecture. It is of course not possible to describe every conceivable combination of components and/or methods, but one skilled in the art will appreciate that many other combinations and permutations are possible. Accordingly, the novel architecture is intended to embrace all such alternatives, modifications and variations that fall within the spirit and scope of the appended claims.

Claims

A method for image comparison including:

Using a recognition network of variational autoencoders, illumination-invariant features and illumination features are decoupled from the target image and the standard image, where the target image and the standard image are related to the relative positions of the components to be verified. The illumination invariant feature is associated with the geometric feature of the component to be verified;

Selecting the illumination-invariant feature from the illumination-invariant feature and the illumination-invariant feature using a mutual information interval;

Comparing the similarity between the illumination-invariant features of the target image and the illumination-invariant features of the standard image;

Depending on whether the similarity is less than a threshold, it is determined whether the element to be verified associated with the target image has an error.
The method according to claim 1, wherein the recognition network is obtained through deep learning and is an approximation of the true posterior distribution. During the deep learning process of the recognition network:

Maximize the probability of producing a true image while keeping the distance between the true posterior distribution and the estimated posterior distribution less than the threshold.
The method of claim 2, wherein during the deep learning process of the recognition network:

Jointly optimize the loss function using fully differentiable estimators.
The method according to any one of claims 1 to 3, wherein said using a mutual information interval to select the illumination-invariant feature from the illumination-invariant feature and the illumination-invariant feature includes:

The illumination-invariant features are selected from the illumination-invariant features and the illumination-invariant features using a joint distribution to estimate mutual information intervals between latent variables and gold standard factors.
The method according to any one of claims 1-3, wherein comparing the similarity between the illumination-invariant features of the target image and the illumination-invariant features of the standard image includes:

The similarity between the illumination-invariant features of the target image and the illumination-invariant features of the standard image is compared by comparing cosine distances between the illumination-invariant features.
An apparatus (200) for image comparison, comprising:

The decoupling module (201) is configured to use the recognition network of the variational autoencoder to decouple the illumination invariant features and illumination features from the target image and the standard image, wherein the target image and the standard image are The relative positions between the elements to be verified are associated, and the illumination invariant features are associated with the geometric features of the elements to be verified;

a selection module (202) configured to select the illumination-invariant feature from the illumination-invariant feature and the illumination-invariant feature using a mutual information interval;

A comparison module (203) configured to compare the similarity between the illumination-invariant features of the target image and the illumination-invariant features of the standard image;

The determining module (204) is configured to determine whether the element to be verified associated with the target image has an error according to whether the similarity is less than a threshold.
The device according to claim 6, further comprising a recognition network training module configured to obtain the recognition network through deep learning, wherein in the process of deep learning, the probability of generating a real image is maximized while maintaining the The distance between the true posterior distribution and the estimated posterior distribution is less than the threshold, wherein the recognition network is an approximation of the true posterior distribution.
The device according to claim 7, wherein the recognition network training module is further configured to:

Jointly optimize the loss function using fully differentiable estimators.
The device according to any one of claims 6-8, wherein the selection module is further configured to:

The illumination-invariant features are selected from the illumination-invariant features and the illumination-invariant features using a joint distribution to estimate mutual information intervals between latent variables and gold standard factors.
The device according to any one of claims 6-8, wherein the comparison module is further configured to:

The similarity between the illumination-invariant features of the target image and the illumination-invariant features of the standard image is compared by comparing cosine distances between the illumination-invariant features.
A computer device includes a memory and a processor, the memory stores a computer program, and is characterized in that when the processor executes the computer program, the steps of the method described in any one of claims 1 to 5 are implemented.
A computer-readable storage medium with a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 5 are implemented.
A computer program product comprising computer instructions instructing a computing device to perform the steps of the method according to any one of claims 1 to 5.