CN115615358A - A Color Crosstalk Correction Method for Color Structured Light Based on Unsupervised Deep Learning - Google Patents

Abstract

The invention discloses a color structured light color crosstalk correction method of unsupervised deep learning, which uses a computer to generate a color phase shift fringe image, collects a deformed color fringe image modulated by the height of the measured object, and extracts three gray images from the image. Input the three grayscale images into three deep neural network modules for color crosstalk correction, output three predicted corrected grayscale images, and solve the predicted phase by phase shift method, use this predicted phase for computer Back projection simulation can obtain the deformed color fringe image corresponding to the phase result; calculate the loss value between the actually collected deformed color fringe image and the deformed color fringe image simulated by back projection, and optimize the network parameters through multiple iterations until the loss value is minimized , to obtain the ideal calibration result. The present invention does not need to produce a large amount of training data and corresponding labels, does not depend on a specific data set, and significantly improves the operation efficiency and generalization performance of deep learning.

Description

A Color Crosstalk Correction Method for Color Structured Light Based on Unsupervised Deep Learning

technical field

The invention relates to the technical field of three-dimensional measurement of structured light, in particular to a method for correcting color crosstalk of color structured light with unsupervised deep learning.

Background technique

Structured light three-dimensional measurement technology has the advantages of non-contact, high sensitivity and high precision, and has been widely used in industrial inspection, reverse engineering, intelligent manufacturing and other industries. With the rapid development of computer vision, structured light 3D measurement technology is currently developing towards high-speed measurement and real-time measurement. The traditional phase-shifting method is very limited in high-speed dynamic measurement applications because at least three structured light images are required to recover a measurement result. The color structured light method loads three phase-shifted images into the RGB three channels respectively to form a color structured light image, so that the measurement results can be recovered by only collecting one image, thus meeting the current trend of high-speed dynamic measurement applications.

However, since the wavelength distribution of natural light is continuous, for a specific color of light wavelength, its distribution is a small range of intervals; therefore, there are inevitably overlaps in the responses of projectors or cameras to RGB three colors, As a result, there is a certain degree of error and interference in the separation of RGB channels for color structured light, and this phenomenon is called color crosstalk. Obviously, if the color crosstalk elimination and correction are not performed on the structured light image after channel separation, there must be serious errors in the phase distribution of the measured object obtained from the solution, so that the accurate three-dimensional surface topography cannot be restored. Therefore, how to correct the color crosstalk phenomenon with high quality has become a difficult problem and an important research direction in the color structured light measurement technology.

Contents of the invention

The purpose of the present invention is to provide a color structured light color crosstalk correction method based on unsupervised deep learning, so as to quickly and effectively realize the color structured light color crosstalk correction.

In order to achieve the above tasks, the present invention adopts the following technical solutions:

A color structured light color crosstalk correction method for unsupervised deep learning, including:

Generate a composite color phase shift fringe structured light image I _C by computer and send it to the measurement system;

Use the projection module of the measurement system to project the composite color phase shift fringe structured light image I _C onto the surface of the measured object, and at the same time, the color camera of the measurement system collects a highly modulated deformed color fringe image of the measured object at another angle I _color ;

Separate the RGB three channels of the deformed color fringe image I _color , extract three deformed grayscale fringe structured light images I _R , I _G and I _B respectively, and input the images I _R , I _G and I _B into Color crosstalk correction in three deep neural network modules;

The three deep neural networks output three predicted color crosstalk corrected gray-scale images I' _R , I' _G and I' _B , and use the phase shift method to solve the predicted phase Φ' based on the three gray-scale images;

Utilize the predicted phase Φ' to carry out computer back projection simulation, and obtain the deformed color fringe image I _{rep_color} corresponding to the phase result;

Construct the loss function of the deep neural network by using the collected deformed color stripe image I _color and the deformed color stripe image I _{rep_color} obtained by back projection simulation, and calculate its loss value;

When the loss value calculated by the loss function reaches the minimum value, the final corrected crosstalk corrected ideal fringe image is obtained; the corrected ideal fringe image is combined with the phase shift method to calculate the final ideal phase Φ, and restore the real three-dimensional shape of the measured object.

Further, the computer-generated color structured light image can be expressed as:

Among them, I _C is a composite color phase-shift fringe structured light image, which contains three channel images, which are represented by I ₁ , I ₂ and I ₃ respectively, and each channel is a gray-scale sinusoidal fringe phase-shift image; f represents sinusoidal The frequency of the stripes, x represents the horizontal coordinate index of the image, 2nπ/3 represents its phase shift, and n represents the nth channel.

Further, the measurement system includes a DLP projection module, a color industrial camera and a computer. Among them, the optical axis of the DLP projection module is at an angle of 30 degrees to the measured object to project the structured light image _IC to the surface of the measured object, and the optical axis of the color industrial camera is perpendicular to the measured object to collect images.

Further, the highly modulated deformed color fringe image I _color of the measured object collected by the color camera can be expressed as:

Among them, I _R , I _G , and I _B are RGB three-channel images of I _color , and Φ represents the real phase distribution of the measured object, which is the unknown quantity that needs to be solved in the three-dimensional measurement process.

Furthermore, the three deep neural sub-network modules responsible for processing three deformed gray-scale stripe structured light images I _R , I _G and I _B are all U-shaped networks, and the U-shaped network is composed of an encoder and a decoder; among them, the encoding The filter has 5 layers from top to bottom. Each layer is connected by feature extraction and downsampling to reduce the image size layer by layer. There are 3 convolutional layers arranged in sequence inside each layer, and the convolutional layers are connected by residual blocks. , the output of the last convolutional layer of the previous layer is used as the input of the first convolutional layer of the next layer; the decoder structure is symmetrical to the encoder, with a total of 5 layers, and each layer is restored layer by layer through feature extraction and upsampling connection The size of the original image, and finally get the predicted output; among them, the output of the last convolutional layer of the next layer is used as the input of the first convolutional layer of the previous layer; the bottom layer of the encoder and the bottom layer of the decoder are connected by Attention mechanism module connection.

Further, the weights of the three deep neural sub-networks are not shared.

Further, the phase shift method is used to solve the predicted phase Φ' based on the three grayscale images, including:

Use the grayscale image I' _R , I' _G and I' _B to substitute into the phase shift method formula to solve the predicted phase Φ':

Further, the computer back-projection simulation using the predicted phase Φ' can obtain the deformed color fringe image I _{rep_color} corresponding to the phase result, and the formula is as follows:

Among them, I _{rep_R} , I _{rep_R} , and I _{rep_R} are RGB three-channel images of I _{rep_color} .

Further, the loss function of the deep neural network is expressed as:

Among them, x and y represent the horizontal and vertical coordinate indexes of the image, H and W represent the height and width of the image; λ _R , λ _G and λ _B are the weights of the loss values of the three channels of RGB respectively.

Further, through network training, save the three deep neural networks when the loss value reaches the minimum as the final correction network; input the images I _R , I _G and I _B into the correction network, and use the correction network output corresponding to RGB The three crosstalk corrected ideal fringe images of the three channels are used to solve the phase using the phase shift method, and finally the obtained ideal phase Φ is obtained through nonlinear mapping to obtain the real three-dimensional shape of the measured object.

Further, the nonlinear calibration model adopted by the nonlinear mapping is expressed as:

Among them, h represents the depth information of the three-dimensional topography, and a, b and c are the calibration parameters, which are all determined in the calibration of the measurement system before the measurement.

Compared with the prior art, the present invention has the following technical characteristics:

1. This solution uses an unsupervised deep learning mechanism to correct the distortion and aliasing of the striped structured light image caused by the color crosstalk in the colored structured light, thereby reducing the phase error of the measurement and achieving a higher-precision three-dimensional shape Measurement. Compared with the traditional color crosstalk correction method, the method provided by the present invention does not require complex crosstalk matrix estimation, but uses the powerful nonlinear fitting and prediction capabilities of the deep neural network to perform rapid correction.

2. Compared with the learning mechanism of input-label one-to-one mapping of traditional deep learning, the non-supervised learning mechanism provided by the present invention can completely discard the label data in the data set, greatly reducing the labor of data collection and production in deep learning technology. cost. In addition, the back-projection simulation results used by the method provided by the present invention are simulated strictly according to the physical model of the measurement principle, so compared with the traditional deep learning mechanism, the method of the present invention is more interpretable and can be independent of the training data set It can be applied to any measurement scenario, so it has good generalization performance.

Description of drawings

Fig. 1 is the schematic diagram of the color structured light measuring system used in the present invention

Fig. 2 is the flow chart of unsupervised deep learning method of the present invention

Fig. 3 is the deep neural network structural diagram of the present invention

Description of reference signs: 1-color industrial camera, 2-DLP projection module, 3-computer, 4-R channel coded grayscale fringe image, 5-G channel coded grayscale fringe image, 6-B channel coded grayscale fringe image , 7 - Synthesized color-coded fringe images.

detailed description

The invention provides an unsupervised deep learning color structured light color crosstalk correction method. First, a computer is used to generate a color phase-shifted fringe structured light image, and then a color camera is used to collect a deformed color fringe that is highly modulated by the measured object. image, and separate the R, G, and B channels of the color image to extract three grayscale images respectively; then input the three grayscale images into three deep neural network modules for color crosstalk correction , and then, the deep neural network outputs three predicted corrected grayscale images, based on the three grayscale images, the phase shift method is used to solve the predicted phase; the predicted phase is used for computer back projection simulation, and the deformation corresponding to the phase result can be obtained Color fringe image; finally, calculate the loss value of the collected deformed color fringe image and the deformed color fringe image obtained by back-projection simulation, and optimize the network parameters through multiple iterations until the loss value is minimized, and the ideal correction result is obtained. The unsupervised deep learning mechanism of the present invention does not need to produce a large amount of training data and corresponding labels, does not depend on a specific data set, and significantly improves the operation efficiency and generalization performance of deep learning.

The specific implementation process of the present invention will be described in further detail below in conjunction with the accompanying drawings.

S1, use computer to generate a composite color phase shift fringe structured light image I _C and send it to the measurement system:

The computer-generated color structured light image can be expressed as:

Among them, I _C is a composite color phase-shifted fringe structured light image, which contains three channel images, represented by I ₁ , I ₂ and I ₃ , and each channel is a gray-scale sinusoidal fringe phase-shifted image; f represents sinusoidal The frequency of the stripes, x represents the horizontal coordinate index of the image, 2nπ/3 represents its phase shift, and n represents the nth channel.

S2, using the projection module of the measurement system to project the composite color phase-shifted fringe structured light image I _C onto the surface of the measured object, and at the same time, the color camera of the measurement system collects a highly modulated distorted color image of the measured object at another angle The fringe image I _color ; the other angle refers to other angles different from the projection angle.

Referring to Fig. 1, the measurement system in this embodiment includes a DLP projection module, a color industrial camera and a computer. Among them, the optical axis of the DLP projection module and the measured object form an angle of about 30 degrees to project the structured light image _IC to the surface of the measured object, and the optical axis of the color industrial camera is perpendicular to the measured object to collect images.

The deformed color fringe image I _color that is highly modulated by the measured object collected by the color camera can be expressed as:

Among them, I _R , I _G , I _B (4, 5, 6 in Figure 1) are RGB three-channel images of I _color (7 in Figure 1), and Φ represents the real phase distribution of the measured object, which is a three-dimensional measurement The unknown quantity that needs to be solved in the process; the RGB three channels of the color image I _color are extracted as separate images, and three deformed grayscale striped structured light images I _R , I _G and I _B can be obtained.

S3, separating the RGB three channels of the deformed color fringe image I _color , extracting three deformed grayscale fringe structured light images I _R , I _G and I _B respectively, and converting the images I _R , I _G and I _B respectively Input to three deep neural network modules for color crosstalk correction:

The three deep neural sub-network modules responsible for processing the three deformed gray-scale stripe structured light images I _R , I _G and I _B are all U-shaped networks, and the U-shaped network is composed of an encoder and a decoder; the encoder is from the top There are 5 layers to the bottom. Each layer is connected by feature extraction and downsampling to reduce the image size layer by layer. There are 3 convolutional layers arranged in sequence inside each layer. The convolutional layers are connected by residual blocks. The previous layer The output of the last convolutional layer of the layer is used as the input of the first convolutional layer of the next layer; the decoder structure is symmetrical to the encoder, with a total of 5 layers, and each layer is connected through feature extraction and upsampling to restore the original image size layer by layer , and finally get the predicted output; among them, the output of the last convolutional layer of the next layer is used as the input of the first convolutional layer of the previous layer; the bottom layer of the encoder and the bottom layer of the decoder are connected by the attention mechanism The module connection can improve the network's attention to the stripe edge features; in addition, the corresponding layer of the encoder and the decoder is connected by a skip connection, which is used to directly transmit the feature map of the same size to improve the learning speed.

Since the three deformed gray-scale fringe structured light images I _R , I _G and I _B are all sinusoidal fringe patterns, but the size and characteristics of the crosstalk influence of each image are different, so the weights of the three deep neural sub-networks cannot be shared.

S4, the three deep neural networks output three predicted color crosstalk corrected grayscale images I' _R , I' _G and I' _B , based on the three grayscale images, the phase shift method is used to solve the predicted phase Φ':

Among them, the predicted phase Φ' is obtained by substituting the grayscale images I' _R , I' _G and I' _B into the formula of the phase shift method:

S5, using the predicted phase Φ' to carry out computer back projection simulation, the deformed color fringe image I _{rep_color} corresponding to the phase result can be obtained; this step includes:

The principle of computer back-projection simulation is to regard the predicted phase obtained from the solution as a known quantity, and add it to the generation formula of the colored stripe structured light to obtain a deformed colored stripe image I _{rep_color} modulated by the predicted phase. The generation formula is as follows:

Among them, I _{rep_R} , I _{rep_R} , and I _{rep_R} are RGB three-channel images of I _{rep_color} ; theoretically, if the corrected grayscale image predicted by the network is accurate enough, the deformed color striped structured light image I _{rep_color} obtained by backprojection simulation will be It is almost the same as the actually collected deformed color fringe image I _color .

S6, using the collected deformed color stripe image I _color and the deformed color stripe image I _{rep_color} obtained by back projection simulation to construct a loss function of the deep neural network, and calculate its loss value; this step includes:

Based on the deformed color stripe image I _{rep_color} obtained by the back projection simulation and the collected deformed color stripe image I _color , a constraint relationship is established to optimize and adjust the parameters of the deep neural network and prompt the deep neural network to output higher-quality correction results; the deep neural network The loss function is defined as:

Among them, x and y represent the horizontal and vertical coordinate indexes of the image, and H and W represent the height and width of the image.

In the actual calculation, the RGB three channels of the two color fringe images are used to calculate the loss respectively, and finally add them together. The loss function can be further rewritten as:

Among them, λ _R , λ _G and λ _B are the weights of the loss values of the three channels of RGB respectively, which are set according to the specific performance of the network.

S7, when the loss value calculated by the loss function reaches the minimum value, the final corrected crosstalk corrected ideal fringe image is obtained; the corrected ideal fringe image is combined with the phase shift method to calculate the final ideal phase Φ, and restore the true three-dimensional shape of the measured object ; This step includes:

Through network training, save the three deep neural networks when the loss value reaches the minimum as the final correction network; input the images I _R , I _G and I _B into the correction network, and use the correction network to output the three channels corresponding to RGB The three crosstalk-corrected ideal fringe images of , use the phase shift method in S4 to solve the phase, and finally map the obtained ideal phase Φ through the following nonlinear calibration model to obtain the real three-dimensional shape of the measured object:

Among them, h represents the depth information of the three-dimensional topography, a, b and c are the calibration parameters, which are all determined in the calibration of the measurement system before the measurement.

The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still apply to the foregoing embodiments Modifications to the technical solutions recorded, or equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of each embodiment of the application, and should be included in this application. within the scope of protection.

Claims (10)

1. A color structure light color crosstalk correction method of unsupervised deep learning is characterized by comprising the following steps:

generating a composite color phase shift fringe structure optical image I by computer _C And transmitting to a measuring system;

using a projection module of a measurement system to shift the composite color phase fringe structure light image I _C Projecting the image on the surface of the measured object, and simultaneously collecting a deformed color fringe image I modulated by the height of the measured object at another angle by a color camera of the measuring system _color ；

For the deformed color stripe image I _color The RGB three channels are separated, and three deformed gray scale stripe structure light images I are respectively extracted _R ，I _G And I _B An image I _R ，I _G And I _B Respectively inputting the color crosstalk correction data into three deep neural network modules for color crosstalk correction;

the three deep neural networks output three predicted color crosstalk corrected grayscale images I' _R ，I’ _G And I' _B Solving a predicted phase phi' by using a phase shift method based on the three gray level images;

carrying out computer back projection simulation by utilizing the predicted phase phi' to obtain a deformed color stripe image I corresponding to the phase result _{rep_color} ；

Using acquired deformed colour stripe image I _color Deformed color stripe image I obtained by simulation with inverse projection _{rep_color} Constructing a loss function of the deep neural network, and calculating to obtain a loss value of the loss function;

when the loss value calculated by the loss function reaches the minimum value, obtaining a finally corrected crosstalk correction ideal fringe image; and calculating a final ideal phase phi by using the corrected ideal fringe image and combining a phase shift method, and recovering the real three-dimensional shape of the measured object.

2. The unsupervised deep learning color structured light color crosstalk correction method of claim 1, wherein the computer generated color structured light image is representable as:

in which I _C Is a composite color phase-shift fringe-structured light image comprising three channel images, each with I ₁ ，I ₂ And I ₃ Showing that each channel is a gray scale sine stripe phase shift image; f denotes the frequency of the sine stripe, x denotes the lateral coordinate index of the image, 2n pi/3 denotes its phase shift amount, and n denotes the nth channel.

3. The unsupervised deep learning color structured light color crosstalk correction method of claim 1, wherein the measurement system comprises a DLP projection module, a color industry camera, and a computer. Wherein, DLP projection module optical axis is 30 degrees angles with the measured object with structured light image I _C The image is projected to the surface of the measured object, and the optical axis of the color industrial camera is vertical to the measured object to acquire the image.

4. The method as claimed in claim 1, wherein the deformed color stripe image I acquired by the color camera and modulated by the height of the measured object is used for correcting the color crosstalk of the color structure of the unsupervised deep learning color structure _color Can be expressed as:

wherein,I _R ，I _G ，I _B is I _color The RGB three-channel image phi represents the real phase distribution of the measured object and is an unknown quantity to be solved in the three-dimensional measurement process.

5. The unsupervised deep learning color structure light color crosstalk correction method according to claim 1, characterized in that it is responsible for processing three deformed gray stripe structure light images I _R ，I _G And I _B The three deep neural sub-network modules are all U-shaped networks, and each U-shaped network consists of an encoder and a decoder; the encoder has 5 layers from top to bottom, each layer is connected through feature extraction and downsampling to reduce the image size layer by layer, 3 sequentially arranged convolution layers are arranged in each layer, the convolution layers are connected through a residual block, and the output of the last convolution layer of the previous layer is used as the input of the first convolution layer of the next layer; the decoder structure is symmetrical to the encoder, 5 layers are provided, the original image size is restored layer by layer through feature extraction and upsampling connection between every two layers, and finally a predicted output result is obtained; wherein, the output of the last convolution layer of the next layer is used as the input of the first convolution layer of the previous layer; the lowest layer of the encoder and the lowest layer of the decoder are connected by an attention mechanism module.

6. The unsupervised deep learning color structure light color crosstalk correction method of claim 1, wherein the solving the predicted phase Φ' based on the three gray images by using a phase shift method comprises:

using gray-scale image I' _R ，I’ _G And l' _B Substituting into a phase shift method formula to solve the predicted phase phi':

7. the unsupervised deep learning color structure light color crosstalk correction method of claim 1, wherein the method is characterized in thatThen, the computer inverse projection simulation is carried out by utilizing the predicted phase phi', and the deformed color stripe image I corresponding to the phase result can be obtained _{rep_color} The formula is as follows:

wherein, I _{rep_R} ，I _{rep_R} ,I _{rep_R} Is I _{rep_color} Three channel images of RGB.

8. The unsupervised deep learning color structure light color crosstalk correction method of claim 1, wherein the loss function of the deep neural network is expressed as:

wherein x and y represent the horizontal and vertical coordinate indexes of the image, and H and W represent the height and width of the image; lambda [ alpha ] _R ，λ _G And λ _B The weights of the three channel loss values of RGB are respectively.

9. The method for correcting color crosstalk of an unsupervised deep learning color structure according to claim 1, wherein three deep neural networks with the minimum loss value are saved as a final correction network through network training; image I _R ，I _G And I _B Inputting a correction network, correcting an ideal fringe image by using three crosstalk signals corresponding to three channels of RGB (red, green and blue) output by the correction network, solving a phase by using a phase shift method, and finally obtaining a real three-dimensional shape of a measured object by nonlinear mapping the obtained ideal phase phi.

10. The unsupervised deep learning color structure light color crosstalk correction method according to claim 1, wherein the nonlinear calibration model adopted by the nonlinear mapping is expressed as:

wherein h represents three-dimensional topography depth information, a, b and c are calibration parameters, which are determined in a measurement system calibration link before measurement.

Images