CN110060212A - A kind of multispectral photometric stereo surface normal restoration methods based on deep learning - Google Patents
A kind of multispectral photometric stereo surface normal restoration methods based on deep learning Download PDFInfo
- Publication number
- CN110060212A CN110060212A CN201910208408.0A CN201910208408A CN110060212A CN 110060212 A CN110060212 A CN 110060212A CN 201910208408 A CN201910208408 A CN 201910208408A CN 110060212 A CN110060212 A CN 110060212A
- Authority
- CN
- China
- Prior art keywords
- surface normal
- deep learning
- photometric stereo
- restored
- multispectral
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000013135 deep learning Methods 0.000 title claims abstract description 27
- 238000000034 method Methods 0.000 title claims abstract description 23
- 230000006870 function Effects 0.000 claims description 5
- 230000004913 activation Effects 0.000 claims description 3
- 238000013528 artificial neural network Methods 0.000 claims description 3
- 238000003475 lamination Methods 0.000 claims 1
- 239000000463 material Substances 0.000 description 3
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 2
- 238000011084 recovery Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 1
- 239000003814 drug Substances 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
- Image Analysis (AREA)
Abstract
The present invention relates to multispectral more degree three-dimensional surface normal direction to restore field, particularly discloses a kind of multispectral photometric stereo surface normal restoration methods based on deep learning.This method utilizes multispectral photometric stereo system, shoots the photo of individual object to be restored；The RGB channel of individual object picture to be restored of shooting is separated into three single pass grey white images；Three width single channel images are inputted using three input standard photometric stereo algorithms, obtain initial surface normal direction；The photo of individual object to be restored and initial surface normal direction are inputted into depth network model together, using deep learning algorithm, export accurate surface normal.Step of the present invention is simple, it compared to the multispectral photometric stereo algorithm of tradition, does not need to demarcate object in advance, does not need the initial depth for obtaining portion using other equipment yet, but self information is used completely, obtain initial surface normal direction and restores accurate surface normal using deep learning algorithm.
Description
(1) technical field
The present invention relates to multispectral more degree three-dimensional surface normal direction to restore field, in particular to a kind of based on the multispectral of deep learning
Photometric stereo surface normal restoration methods.
(2) background technique
The recovery of body surface normal direction is an important component of three-dimensional reconstruction, is to have to answer extensively in computer vision field
With the research direction of value.Multispectral photometric stereo surface normal restoration methods are a kind of sides of single image prediction surface normal
Method, the surface normal that can act on movement or non-rigid object restore.Surface normal can be provided compared to two-dimensional image
The surface profile data and depth data of object, can more comprehensively show object properties.Therefore in unmanned, geographical survey
The multiple fields such as amount, human-computer interaction, modern medicine, which suffer from, to be widely applied.
But limitation is very big in practical applications: tradition side for existing multispectral photometric stereo surface normal restoration methods
Method needs measuring targets to carry out preparatory material calibration, or needs to be known in advance the accurate depth of body surface portion
Information.These conditions are difficult to realize in practical applications.Further, since traditional multispectral photometric stereo method relies on this
A little preparatory information, in actual operation often since the deviation of preparatory information leads to the error of method prediction surface normal.
(3) summary of the invention
That in order to compensate for the shortcomings of the prior art, the present invention provides a kind of steps is accurate, accuracy is high based on deep learning
Multispectral photometric stereo surface normal restoration methods.
The present invention is achieved through the following technical solutions:
A kind of multispectral photometric stereo surface normal restoration methods based on deep learning, include the following steps:
(1) multispectral photometric stereo system is utilized, the photo of individual object to be restored is shot；
(2) RGB channel of individual object picture to be restored of shooting is separated into three single pass grey white images；
(3) three width single channel images are inputted using three input standard photometric stereo algorithms, obtains initial surface normal direction；
(4) photo of individual object to be restored and initial surface normal direction are inputted into depth network model together, utilize depth
Algorithm is practised, accurate surface normal is exported.
The present invention is using three input standard photometric stereo algorithms, first acquisition initial surface normal direction, although initial surface method
To be it is inaccurate, still, using deep learning algorithm, merged with the photo of shooting, accurate recovery table can be exported
Face normal direction.The present invention improves deep learning algorithm, improves multispectral photometric stereo surface by the way that initial surface normal direction is added
The accuracy that normal direction is restored.
The specific technical proposal of the invention is:
In step (1), object to be restored is shot under three light irradiations of RGB, using object to be restored as coordinate axis origin, is built
Vertical cartesian coordinate system, the unit vector in three light-illuminating directions of RGB are respectively,,。
In step (2), the single channel image separated through RGB channel is saved as respectively,,。
In step (3), three input standard photometric stereo algorithms areC=ρLn；Wherein, ρ is the scalar for the fixation that can be asked,,For pixel coordinate on image,
, n is wait askPixel initial surface normal direction.Using three input standard photometric stereo algorithms, ignore
RGB lamp in different channels mix and the material information of body surface, therefore the initial surface normal direction acquired is that have
Error.
In step (4), the deep learning algorithm of depth network model is, first by the photo and initial table of object to be restored
Face Normal Align is cut to the segment of 40*40 pixel；Photo after cutting merges 6 layers of input with the initial surface normal direction of acquisition
Convolutional layer, the convolution kernel size of preceding 3 layers of convolutional layer are 5*5 pixel, and latter 3 layers of convolution kernel size is 3*3 pixel, all convolutional layers
It is all made of the filling mode of " SAME " and the activation primitive of " Relu ", by 6 layers of convolutional layer, network final output corresponds to 40*
The accurate surface normal of 40 pixels.
Using mean square error as loss function, specific formula is the deep learning algorithm,
In, n represents true normal direction,Represent the normal direction of neural network forecast；Loss function is carried out most using the Adam algorithm of default setting
Smallization.
The characteristic pattern port number of 6 layers of convolutional layer is respectively 32,64,128,128,64,32.
Step of the present invention is simple, compared to the multispectral photometric stereo algorithm of tradition, does not need to demarcate object in advance, also be not required to
It to use other equipment to obtain the initial depth of portion, but use self information completely, obtain initial surface normal direction simultaneously
Restore accurate surface normal using deep learning algorithm, makes multispectral photometric stereo algorithm can be with practical application, and increase
Restore the accuracy of surface normal.
(4) Detailed description of the invention
The present invention will be further described below with reference to the drawings.
Fig. 1 is the structural schematic diagram of depth network model of the present invention；
Fig. 2 is the single photo of object to be restored of the embodiment of the present invention；
Fig. 3 is the initial surface normal direction of object to be restored of the embodiment of the present invention；
Fig. 4 is the exact surface normal direction of object to be restored of the embodiment of the present invention.
(5) specific embodiment
The present invention will be further explained below with reference to the attached drawings.
A kind of multispectral photometric stereo surface normal restoration methods based on deep learning, specifically comprise the following steps:
(1) photo of individual object to be restored of multispectral photometric stereo system photographs is obtained
Circular support is installed in the surface of object, that is, test articles to be restored, and camera, circular rail are placed in circular support middle position
Uniformly distributed three irradiation lights of RGB, provide different illumination directions, and three irradiation lights tilt angle 30 having the same on road
Degree.
Using multispectral photometric stereo system, the photo of individual object to be restored is shot, as shown in Fig. 2: object to be restored
Body is shot under three light irradiations of RGB, if establishing cartesian coordinate system using object to be restored as coordinate axis origin, then red green
The unit vector in blue three light-illuminating directions is respectively,,。
(2) RGB channel of individual object picture to be restored of shooting is separated into three width single channel images
The photo of shooting is the photochrome for having tri- channels RGB, and the RGB channel of photochrome is separated, saves as three respectively
Single pass ash white image,,。
(3) three width single channel images are inputted using three input standard photometric stereo algorithms to obtain initial surface normal direction (such as attached
Shown in Fig. 3)
Three, which input standard photometric stereo algorithms, isC=ρLn；Wherein, ρ is the scalar for the fixation that can be asked,,For pixel coordinate on image,, n to be to be askedPixel initial surface normal direction.Utilizing three input standard photometric stereo algorithms
In the case where, have ignored RGB lamp in different channels mix and the material information of body surface, therefore acquire just
Beginning surface normal has error.
(4) photo and initial surface normal direction of individual object to be restored are inputted using deep learning algorithm, output is accurate
Surface normal
Initial surface normal direction is inputted into depth network model together with the photo shot in the first step, using deep learning algorithm,
Export accurate surface normal.Wherein, depth network model specific structure is as shown in Fig. 1.
First by the photo of shooting and initial Normal Align, it is cut to the segment of 40*40 pixel.Photo after cutting with obtain
The initial normal direction fusion 6 layers of convolutional layer of input obtained, the convolution kernel size of preceding 3 layers of convolutional layer is 5*5 pixel, rear 3 layers of convolution kernel
Size is 3*3 pixel, and all convolutional layers are all made of the filling mode of " SAME ", it means that the size of convolutional layer keeps 40*40
Pixel is constant.6 layers of convolution are all made of the activation primitive of " Relu ", and the characteristic pattern port number of 6 layers of convolutional layer is respectively 32,64,
128,128,64,32, by 6 layers of convolutional layer, network final output corresponds to the accurate surface normal of 40*40 pixel, such as attached
Shown in Fig. 4.
Using mean square error as loss function, specific formula is the deep learning algorithm,
In, n represents true normal direction,Represent the normal direction of neural network forecast；Loss function is carried out most using the Adam algorithm of default setting
Smallization.
Claims (7)
1. a kind of multispectral photometric stereo surface normal restoration methods based on deep learning, it is characterized in that, include the following steps:
(1) multispectral photometric stereo system is utilized, the photo of individual object to be restored is shot；(2) by individual object to be restored of shooting
The RGB channel of photo is separated into three single pass grey white images；(3) three width are inputted using three input standard photometric stereo algorithms
Single channel image obtains initial surface normal direction；(4) photo of individual object to be restored and initial surface normal direction are inputted together
Depth network model exports accurate surface normal using deep learning algorithm.
2. the multispectral photometric stereo surface normal restoration methods according to claim 1 based on deep learning, feature
Be: in step (1), object to be restored is shot under three light irradiations of RGB, using object to be restored as coordinate axis origin, is built
Vertical cartesian coordinate system, the unit vector in three light-illuminating directions of RGB are respectively,,。
3. the multispectral photometric stereo surface normal restoration methods according to claim 1 based on deep learning, feature
It is: in step (2), the single channel image separated through RGB channel is saved as respectively,,。
4. the multispectral photometric stereo surface normal restoration methods according to claim 1 based on deep learning, feature
Be: in step (3), three input standard photometric stereo algorithms areC=ρLn；Wherein, ρ is the scalar for the fixation that can be asked,,For pixel coordinate on image,, n is wait askPixel initial surface normal direction.
5. the multispectral photometric stereo surface normal restoration methods according to claim 1 based on deep learning, feature
Be: in step (4), the deep learning algorithm of depth network model is, first by the photo and initial surface of object to be restored
Normal Align is cut to the segment of 40*40 pixel；Photo after cutting merges 6 layers of volume of input with the initial surface normal direction of acquisition
Lamination, the convolution kernel size of preceding 3 layers of convolutional layer are 5*5 pixel, and latter 3 layers of convolution kernel size is 3*3 pixel, and all convolutional layers are equal
Using the filling mode of " SAME " and the activation primitive of " Relu ", by 6 layers of convolutional layer, network final output corresponds to 40*40
The accurate surface normal of pixel.
6. the multispectral photometric stereo surface normal restoration methods according to claim 5 based on deep learning, feature
Be: using mean square error as loss function, specific formula is the deep learning algorithm, wherein
N represents true normal direction,Represent the normal direction of neural network forecast.
7. the multispectral photometric stereo surface normal restoration methods according to claim 5 based on deep learning, feature
Be: the characteristic pattern port number of 6 layers of convolutional layer is respectively 32,64,128,128,64,32.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910208408.0A CN110060212B (en) | 2019-03-19 | 2019-03-19 | Deep learning-based multispectral luminosity three-dimensional surface normal direction recovery method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910208408.0A CN110060212B (en) | 2019-03-19 | 2019-03-19 | Deep learning-based multispectral luminosity three-dimensional surface normal direction recovery method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110060212A true CN110060212A (en) | 2019-07-26 |
| CN110060212B CN110060212B (en) | 2023-07-14 |
Family
ID=67317214
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910208408.0A Active CN110060212B (en) | 2019-03-19 | 2019-03-19 | Deep learning-based multispectral luminosity three-dimensional surface normal direction recovery method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110060212B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111275680A (en) * | 2020-01-18 | 2020-06-12 | 中国海洋大学 | SAR image change detection method based on Gabor convolution network |
| CN113936117A (en) * | 2021-12-14 | 2022-01-14 | 中国海洋大学 | High-frequency region enhanced luminosity three-dimensional reconstruction method based on deep learning |
| CN118628371A (en) * | 2024-08-12 | 2024-09-10 | 南开大学 | Surface normal restoration method and device based on photometric stereo and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080123937A1 (en) * | 2006-11-28 | 2008-05-29 | Prefixa Vision Systems | Fast Three Dimensional Recovery Method and Apparatus |
| CN107862741A (en) * | 2017-12-10 | 2018-03-30 | 中国海洋大学 | A kind of single-frame images three-dimensional reconstruction apparatus and method based on deep learning |
| CN108510573A (en) * | 2018-04-03 | 2018-09-07 | 南京大学 | A method of the multiple views human face three-dimensional model based on deep learning is rebuild |
| CN108665496A (en) * | 2018-03-21 | 2018-10-16 | 浙江大学 | A kind of semanteme end to end based on deep learning is instant to be positioned and builds drawing method |
-
2019
- 2019-03-19 CN CN201910208408.0A patent/CN110060212B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080123937A1 (en) * | 2006-11-28 | 2008-05-29 | Prefixa Vision Systems | Fast Three Dimensional Recovery Method and Apparatus |
| CN107862741A (en) * | 2017-12-10 | 2018-03-30 | 中国海洋大学 | A kind of single-frame images three-dimensional reconstruction apparatus and method based on deep learning |
| CN108665496A (en) * | 2018-03-21 | 2018-10-16 | 浙江大学 | A kind of semanteme end to end based on deep learning is instant to be positioned and builds drawing method |
| CN108510573A (en) * | 2018-04-03 | 2018-09-07 | 南京大学 | A method of the multiple views human face three-dimensional model based on deep learning is rebuild |
Non-Patent Citations (4)
| Title |
|---|
| HIROAKI SANTO: "Deep Photometric Stereo Network" * |
| LIANG LU: "Three-Dimensional Reconstruction from Single Image Base on Combination of CNN and Multi-Spectral Photometric Stereo" * |
| XIAOJUN QI: "GeoNet: Geometric Neural Network for Joint Depth and Surface Normal Estimation" * |
| 张军等: "一种基于多面体模型的从明暗恢复物体形状方法", 《计算机工程与应用》 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111275680A (en) * | 2020-01-18 | 2020-06-12 | 中国海洋大学 | SAR image change detection method based on Gabor convolution network |
| CN111275680B (en) * | 2020-01-18 | 2023-05-26 | 中国海洋大学 | SAR image change detection method based on Gabor convolution network |
| CN113936117A (en) * | 2021-12-14 | 2022-01-14 | 中国海洋大学 | High-frequency region enhanced luminosity three-dimensional reconstruction method based on deep learning |
| CN113936117B (en) * | 2021-12-14 | 2022-03-08 | 中国海洋大学 | High-frequency region enhanced luminosity three-dimensional reconstruction method based on deep learning |
| CN118628371A (en) * | 2024-08-12 | 2024-09-10 | 南开大学 | Surface normal restoration method and device based on photometric stereo and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110060212B (en) | 2023-07-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11115633B2 (en) | Method and system for projector calibration | |
| KR100966592B1 (en) | Method for calibrating a camera with homography of imaged parallelogram | |
| CN109919911B (en) | Mobile three-dimensional reconstruction method based on multi-view photometric stereo | |
| CN109920007B (en) | Three-dimensional imaging device and method based on multispectral photometric stereo and laser scanning | |
| US6671399B1 (en) | Fast epipolar line adjustment of stereo pairs | |
| CN111028155B (en) | Parallax image splicing method based on multiple pairs of binocular cameras | |
| CN112884682B (en) | Stereo image color correction method and system based on matching and fusion | |
| US20170032565A1 (en) | Three-dimensional facial reconstruction method and system | |
| CN110060212A (en) | A kind of multispectral photometric stereo surface normal restoration methods based on deep learning | |
| US9883167B2 (en) | Photometric three-dimensional facial capture and relighting | |
| Ackermann et al. | Geometric Point Light Source Calibration. | |
| KR101983586B1 (en) | Method of stitching depth maps for stereo images | |
| CN111027415B (en) | Vehicle detection method based on polarization image | |
| CN109242898B (en) | Three-dimensional modeling method and system based on image sequence | |
| CN104539928A (en) | Three-dimensional printing image synthesizing method for optical grating | |
| CN108592823A (en) | A kind of coding/decoding method based on binocular vision color fringe coding | |
| CN112907573B (en) | Depth completion method based on 3D convolution | |
| CN103632334A (en) | Infinite image alignment method based on parallel optical axis structure cameras | |
| Chen et al. | 3d face reconstruction using color photometric stereo with uncalibrated near point lights | |
| CN110021067B (en) | Method for constructing three-dimensional face normal based on specular reflection gradient polarized light | |
| CN114241059B (en) | Synchronous calibration method for camera and light source in photometric stereo vision system | |
| CN113034590B (en) | AUV dynamic docking positioning method based on visual fusion | |
| CN110874863A (en) | Three-dimensional reconstruction method and system for three-dimensional reconstruction | |
| CN104463964A (en) | Method and equipment for acquiring three-dimensional model of object | |
| CN110874862A (en) | System and method for three-dimensional reconstruction |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |