CN110060212A - A kind of multispectral photometric stereo surface normal restoration methods based on deep learning - Google Patents

Abstract

The invention relates to the field of multi-spectral multi-degree stereo surface normal recovery, and particularly discloses a multi-spectral photometric stereo surface normal recovery method based on deep learning. The method uses a multi-spectral photometric stereo system to take a single photo of the object to be restored; separates the RGB channels of the single photo of the object to be restored into three single-channel gray-white images; uses the three-input standard photometric stereo algorithm to input three images A single-channel image is used to obtain the initial surface normal; a single photo of the object to be restored and the initial surface normal are input into the deep network model, and the deep learning algorithm is used to output the accurate surface normal. Compared with the traditional multi-spectral photometric stereo algorithm, the method of the invention does not need to pre-calibrate the object, nor does it need to use other equipment to obtain the initial depth of a part of the position. Instead, it completely uses its own information to obtain the initial surface normal and uses deep learning. The algorithm restores the exact surface normal.

Description

A Deep Learning-Based Multispectral Photometric Stereo Surface Normal Recovery Method

(1) Technical field

The invention relates to the field of multi-spectral multi-degree stereo surface normal recovery, in particular to a multi-spectral photometric stereo surface normal recovery method based on deep learning.

(2) Background technology

Object surface normal recovery is an important part of 3D reconstruction, and it is a research direction with extensive application value in the field of computer vision. The multispectral photometric stereoscopic surface normal recovery method is a method for predicting the surface normal direction from a single image, which can be applied to the surface normal recovery of moving or non-rigid objects. Compared with the two-dimensional image, the surface normal can provide the surface shape data and depth data of the object, and can display the characteristics of the object more comprehensively. Therefore, it has a wide range of applications in many fields such as unmanned driving, geographic measurement, human-computer interaction, and modern medicine.

However, the existing multispectral photometric three-dimensional surface normal recovery methods have great limitations in practical applications: traditional methods require pre-calibration of the object to be measured, or need to know the accurate depth information of the surface part of the object in advance. These conditions are difficult to achieve in practical applications. In addition, since the traditional multispectral photometric stereo methods rely on these pre-information, in practice, the deviation of pre-information often leads to errors in the method's prediction of the surface normal.

(3) Contents of the invention

In order to make up for the deficiencies of the prior art, the present invention provides a deep learning-based multispectral photometric stereoscopic surface normal recovery method with precise steps and high accuracy.

The present invention is achieved through the following technical solutions:

A deep learning-based multispectral photometric stereoscopic surface normal recovery method, comprising the following steps:

(1) Using the multi-spectral photometric stereo system, take a single photo of the object to be restored;

(2) Separate the RGB channel of the single photo of the object to be restored into three single-channel gray-white images;

(3) Use the three-input standard photometric stereo algorithm to input three single-channel images to obtain the initial surface normal;

(4) Input the single photo of the object to be restored and the initial surface normal into the deep network model, and use the deep learning algorithm to output the accurate surface normal.

The present invention uses a three-input standard photometric stereo algorithm to first obtain the initial surface normal direction. Although the initial surface normal direction is inaccurate, the deep learning algorithm is used to fuse with the photographed photos, and an accurate restored surface normal direction can be output. By adding the initial surface normal, the invention improves the deep learning algorithm and improves the accuracy of the multispectral photometric stereoscopic surface normal recovery.

The specific technical scheme of the present invention is:

In step (1), the object to be restored is photographed under the illumination of three lights of red, green and blue, and the object to be restored is taken as the origin of the coordinate axis to establish a Cartesian coordinate system, and the unit vectors of the illumination directions of the three lights of red, green and blue are respectively: , , .

In step (2), the single-channel images separated by RGB channels are stored as , , .

In step (3), the three-input standard photometric stereo algorithm is C = ρ L n; among them, ρ is a fixed scalar that can be obtained, , is the pixel coordinates on the image, , n is the desired The pixel's initial surface normal. In the case of using the three-input standard photometric stereo algorithm, the mixing of red, green and blue lights in different channels and the material information of the object surface are ignored, so the obtained initial surface normal is in error.

In step (4), the deep learning algorithm of the deep network model is to first align the photo of the object to be restored with the initial surface normal, and crop it into a 40*40 pixel block; the cropped photo and the obtained initial surface normal Fusion input 6 layers of convolution layers, the size of the convolution kernel of the first 3 layers of convolution layers is 5*5 pixels, the size of the convolution kernel of the last 3 layers is 3*3 pixels, and all the convolution layers are filled with "SAME" method and the activation function of "Relu", after 6 layers of convolutional layers, the network finally outputs an accurate surface normal corresponding to 40*40 pixels.

The deep learning algorithm uses the mean square error as the loss function, and the specific formula is , where n represents the true normal, Represents the normal predicted by the network; the loss function is minimized using the Adam algorithm with default settings.

The number of feature map channels of the 6-layer convolutional layers are 32, 64, 128, 128, 64, and 32, respectively.

Compared with the traditional multi-spectral photometric stereo algorithm, the method of the invention does not need to pre-calibrate the object, nor does it need to use other equipment to obtain the initial depth of a part of the position. Instead, it completely uses its own information to obtain the initial surface normal and uses deep learning. The algorithm restores accurate surface normals, making multispectral photometric stereo algorithms practical and increasing the accuracy of restoring surface normals.

(4) Description of drawings

The present invention will be further described below in conjunction with the accompanying drawings.

1 is a schematic structural diagram of a deep network model of the present invention;

2 is a single photo of an object to be restored according to an embodiment of the present invention;

Fig. 3 is the initial surface normal direction of the object to be restored according to the embodiment of the present invention;

FIG. 4 is an accurate surface normal of an object to be restored according to an embodiment of the present invention.

(5) Specific implementation methods

The present invention will be further described below in conjunction with the accompanying drawings.

A deep learning-based multispectral photometric stereoscopic surface normal recovery method specifically includes the following steps:

(1) Obtain a single photo of the object to be restored taken by the multispectral photometric stereo system

The object to be restored, namely the experimental object, is installed with a circular bracket, and a camera is placed in the middle of the circular bracket. Three illumination lamps of red, green and blue are evenly distributed on the circular track to provide different illumination directions, and the three illumination lamps have the same The inclination angle is 30 degrees.

Using the multi-spectral photometric stereo system, take a single photo of the object to be restored, as shown in Figure 2: The object to be restored is photographed under the illumination of three lights, red, green and blue. If the object to be restored is taken as the origin of the coordinate axis, a Cartesian image is established. coordinate system, the unit vectors of the red, green and blue light irradiation directions are respectively , , .

(2) Separate the RGB channels of the single photo of the object to be restored into three single-channel images

The photo taken is a color photo with three RGB channels. The RGB channels of the color photo are separated and stored as three single-channel gray and white images. , , .

(3) Use the three-input standard photometric stereo algorithm to input three single-channel images to obtain the initial surface normal (as shown in Figure 3)

The three-input standard photometric stereo algorithm is C =ρ L n; where ρ is a fixed scalar that can be obtained, , is the pixel coordinates on the image, , n to be sought The pixel's initial surface normal. In the case of using the three-input standard photometric stereo algorithm, the mixing of red, green and blue lights in different channels and the material information of the object surface are ignored, so the obtained initial surface normal is in error.

(4) Use the deep learning algorithm to input a single photo of the object to be restored and the initial surface normal, and output the accurate surface normal

Input the initial surface normal and the photo taken in the first step into the deep network model, and use the deep learning algorithm to output the accurate surface normal. The specific structure of the deep network model is shown in Figure 1.

First, align the captured photo with the initial normal and crop it into a 40*40 pixel tile. The cropped photo and the obtained initial normal are fused to input 6 convolutional layers. The convolution kernel size of the first 3 convolutional layers is 5*5 pixels, and the size of the convolutional kernels of the last 3 layers is 3*3 pixels. All The convolutional layer adopts the "SAME" padding method, which means that the size of the convolutional layer remains unchanged at 40*40 pixels. The 6 layers of convolution all use the activation function of "Relu". The number of feature map channels of the 6 layers of convolution layers are 32, 64, 128, 128, 64, and 32 respectively. After the 6 layers of convolution layers, the final output of the network corresponds to 40 *40 pixel exact surface normal, as shown in Figure 4.

The deep learning algorithm uses the mean square error as the loss function, and the specific formula is , where n represents the true normal, Represents the normal predicted by the network; the loss function is minimized using the Adam algorithm with default settings.

Claims (7)

1. A deep learning-based multi-spectral photometric stereoscopic surface normal recovery method, characterized by comprising the following steps: (1) using a multi-spectral photometric stereo system to take a single photo of an object to be restored; (2) photographing The RGB channels of the single photo of the object to be restored are separated into three single-channel gray-white images; (3) use the three-input standard photometric stereo algorithm to input three single-channel images to obtain the initial surface normal; The photo of the restored object and the initial surface normal are input to the deep network model together, and the deep learning algorithm is used to output the accurate surface normal. 2 . The deep learning-based multispectral photometric stereoscopic surface normal restoration method according to claim 1 , wherein in step (1), the object to be restored is photographed under the illumination of three lights, red, green and blue, to be restored. 3 . The object is the origin of the coordinate axis, and a Cartesian coordinate system is established. The unit vectors of the three light irradiation directions of red, green and blue are respectively , , . 3. The deep learning-based multispectral photometric stereoscopic surface normal recovery method according to claim 1, wherein in step (2), the single-channel images separated by RGB channels are stored as , , . 4. The deep learning-based multispectral photometric stereoscopic surface normal recovery method according to claim 1, characterized in that: in step (3), the three-input standard photometric stereo algorithm is C =ρ L n; wherein, ρ is A seekable fixed scalar, , is the pixel coordinates on the image, , n is the desired The pixel's initial surface normal. 5. The deep learning-based multispectral photometric stereoscopic surface normal recovery method according to claim 1, wherein in step (4), the deep learning algorithm of the deep network model is: The initial surface normal is aligned and cropped into a 40*40 pixel tile; the cropped photo and the obtained initial surface normal are fused into 6 convolutional layers, and the convolution kernel size of the first 3 convolutional layers is 5*5 Pixels, the size of the convolution kernel of the last 3 layers is 3*3 pixels, all the convolution layers use the “SAME” padding method and the “Relu” activation function, after 6 layers of convolution layers, the final output of the network corresponds to 40* 40px accurate surface normal. 6. The deep learning-based multispectral photometric stereoscopic surface normal recovery method according to claim 5, wherein the deep learning algorithm adopts mean square error as a loss function, and the specific formula is , where n represents the true normal, Represents the normal predicted by the network. 7. The deep learning-based multispectral photometric stereoscopic surface normal recovery method according to claim 5, wherein the number of feature map channels of the 6-layer convolutional layers are 32, 64, 128, 128, 64 respectively , 32.