CN110570503B - Method for acquiring normal vector, geometry and material of three-dimensional object based on neural network - Google Patents

Method for acquiring normal vector, geometry and material of three-dimensional object based on neural network Download PDF

Info

Publication number
CN110570503B
CN110570503B CN201910829629.XA CN201910829629A CN110570503B CN 110570503 B CN110570503 B CN 110570503B CN 201910829629 A CN201910829629 A CN 201910829629A CN 110570503 B CN110570503 B CN 110570503B
Authority
CN
China
Prior art keywords
vector
neural network
dimensional object
normal vector
normal
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201910829629.XA
Other languages
Chinese (zh)
Other versions
CN110570503A (en
Inventor
吴鸿智
周昆
康凯彰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hangzhou Faceunity Technology Co ltd
Zhejiang University ZJU
Original Assignee
Hangzhou Faceunity Technology Co ltd
Zhejiang University ZJU
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hangzhou Faceunity Technology Co ltd, Zhejiang University ZJU filed Critical Hangzhou Faceunity Technology Co ltd
Priority to CN201910829629.XA priority Critical patent/CN110570503B/en
Publication of CN110570503A publication Critical patent/CN110570503A/en
Application granted granted Critical
Publication of CN110570503B publication Critical patent/CN110570503B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/005—General purpose rendering architectures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/04—Texture mapping
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/50—Lighting effects

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Graphics (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Image Analysis (AREA)
  • Image Generation (AREA)

Abstract

The invention discloses a neural network-based method for acquiring normal vectors, geometry and materials of a three-dimensional object, and provides an acquisition method combined with a neural network on the idea of actively irradiating a plurality of specific patterns on the object, simultaneously acquiring photos and calculating the obtained photos to acquire the normal vectors of the object. Further, the method uses the obtained normal vectors for optimizing the model of the object. The method can also acquire the material characteristic information while acquiring the normal vector. And finally, obtaining a high-quality geometric and material acquisition result in a combined manner. The number of the illumination patterns obtained by the method is small, the accuracy of the obtained normal vector is high, and the method is not limited to a certain specific acquisition device.

Description

Method for acquiring normal vector, geometry and material of three-dimensional object based on neural network
Technical Field
The invention relates to a method for acquiring a normal vector of a three-dimensional object based on a neural network, a geometric optimization method based on the method and a material acquisition method, and belongs to the field of computer graphics and computer vision.
Background
Digitizing real objects has long been a problem in the field of computer graphics/vision. Currently, digitized real objects can be represented by three-dimensional grids and six-dimensional Spatially Varying Bidirectional Reflectance Distribution Functions (SVBRDFs). Based on the expression, the appearance of the object under any observation angle and illumination condition can be well rendered.
However, how to efficiently acquire reflection information and geometric information of an object simultaneously with high quality still remains a great challenge. On the one hand, high quality requires that as many observations as possible be obtained at the time of measurement. On the other hand, in the actual acquisition process, it is desirable to shorten the measurement time as much as possible. In addition, the high-quality normal vector can optimize a coarse three-dimensional grid so as to obtain a fine geometric model, but no method for efficiently acquiring the normal vector information with high quality exists at present, and the problem is still a great challenge
The existing method/system can collect another piece of information under the condition of known geometry or material. For example, scenes with less complex reflection properties can be acquired using structured light (Daniel Scharstein and Richard Szeliski.2003. high-acquisition stereo depth maps using structured light. in CVPR.) or SFM (structure-from-Motion) (Johannes Lutz Sch6nberger and Jan-Michael Frahm.2016.structure-from-Motion reconstructed. in CVPR.) techniques. For another example, in the illumination stage used for acquiring the material property, it can be assumed that the object is in a high beam (Andrew Gardner, Chris Tchou, Tim Hawkins, and Paul debevec.2003.linear light source reflectance. acm trans. graph.22, 3(2003), 749 + 758.), or the sampling difficulty can be simplified by using the prior knowledge that the acquisition object is a specific geometric object. Although there are some predecessors who have collected geometric and material information of an object at the same time, these methods have strong simplifying assumptions. For example, assume that the object is in the far light (Borom Tunwattanap, Graham Fyffe, Paul Graham, Jay Busch, xueaming Yu, Abhijeet Ghosh, and Paul Debevec.2013.acquiring reflecting and Shape from content polymeric biological reflecting. ACM transmitting. Graph.32, 4, aromatic 109 (annular 2013), 12 pages.), or the material information is isotropic (Rui Xia, Yue Dong, Pieter Peer, Xin Tong. recording and recording sharing. recording sharing and distributing reflecting Surface reflecting adapting Upper reflecting and reflecting Surface reflecting adapting illuminating. ACM transmitting. 35, 6, approximate participant (approximate participant) (the angle of gravity of 12. and drawing), and the material information is also limited by the simplified angle of the question of the company, 9, 12. and 12. by the question of the company, the question of the company, the name of the company, the question of the name of the object of the company, the question of the company, the question of.
Deep learning is a machine learning method that has gained wide application and great success in recent years in the fields of computer vision and computer graphics. The method is characterized in that a network is fitted with an objective function by specifying a loss function and utilizing a gradient descent method.
Disclosure of Invention
The invention aims to provide a method for acquiring normal vectors, geometry and materials of a three-dimensional object based on a neural network, aiming at the defects of the prior art.
The purpose of the invention is realized by the following technical scheme: a three-dimensional object normal vector obtaining method based on a neural network comprises the following steps:
(1) generating training data: acquiring parameters of acquisition equipment, including the distance and angle from a light source to an acquisition space origin, a characteristic curve of the light source, the distance and angle from a camera to the acquisition space origin, and internal and external parameters of the camera; these parameters are used to generate acquisition results that simulate actual cameras as training data.
(2) Training a neural network according to the training data obtained in the step (1), wherein the neural network is characterized as follows:
a. the input of the neural network is Lumitexel, which describes the reflection intensity of the sampling point to the incident light from each light source along a certain observation direction;
b. the output and regression target of the neural network is a normal vector corresponding to the input Lumitexel;
c. the first layer of the neural network comprises a linear fully-connected layer, and a parameter matrix of the linear fully-connected layer is obtained by training the following formula:
Wl＝fw(Wraw)
wherein WrawIs a parameter to be trained; wlThe illumination matrix is n multiplied by m, n is the sampling precision of Lumitexel, and m is the number of illumination patterns; f. ofwIs a mapping for WrawTransforming to enable the generated illumination matrix to correspond to the possible luminous intensity of the acquisition equipment;
d. the second layer and the later layers are nonlinear mapping networks, wherein the length of the output vector of the last layer is 3;
and after the training is finished, taking out the illumination matrix of the first layer of linear full-connection layer.
(3) Make miningThe collecting equipment generates illumination patterns according to the illumination matrix taken out in the step (2), sequentially illuminates the target three-dimensional object and obtains a group of photos r1，r2...，rm(ii) a Sequentially traversing the picture's pixels and channels, from r each time1，r2...，rmThe pixel values of the ith row, the jth column and the kth channel are taken out to form a vector a ═ a1，a2，…，amTaking a as an output vector of a first layer of linear full-connection layer of the neural network, and calculating to obtain an output vector of a last layer, wherein the obtained output vector is a normal vector of the surface of the three-dimensional object corresponding to the pixel; and traversing all the pixels to obtain a normal vector characteristic diagram of the surface of the three-dimensional object.
Further, in the step (1), a specific method for generating an acquisition result simulating an actual camera is as follows: randomly selecting a sampling point in a collection space where a target three-dimensional object may appear, randomly sampling a material parameter of the point, and generating Lumitexel by using a rendering model.
Further, the rendering model adopts a GGX model, and the generation formula is as follows:
Figure BDA0002190237810000031
wherein f isr(ωi；ωoN, t, p) is the bidirectional reflectance distribution function, ωoIs the outgoing direction, omegaiIs the incident direction, n represents the normal vector under the world coordinate system, t represents the x-axis direction of the local coordinate system of the sampling point under the world coordinate system, p is the material parameter vector including alphax，αy，ρd，ρsIn which α isxAnd alphayDenotes the roughness coefficient, ρdRepresenting the diffuse reflectance, ρsRepresents the specular reflectance; omegahIs a half way vector, DGGXIs a differential surface distribution term, F is a Fresnel term, GGGXRepresenting a shading coefficient function.
Further, in the step (2), f is mappedwSelecting a combination of normalization function and inversion function, requiring mIs an even number, WrawIs of a size of
Figure BDA0002190237810000033
Figure BDA0002190237810000034
Can combine WrawViewed as a
Figure BDA0002190237810000035
The combination of individual column vectors, expressed as follows:
Figure BDA0002190237810000036
fwto WrawIs normalized to be a unit vector, and then the negative number of each column is turned over to form a new vector, and the formula is expressed as follows:
fw(Wraw)＝fflip(fnormalize(Wraw))
Figure BDA0002190237810000032
Wnormalizedcan be regarded as
Figure BDA0002190237810000041
A column vector, represented as follows:
Figure BDA0002190237810000042
Figure BDA0002190237810000043
further, in the step (2), a normalization layer may be further connected after the last layer of the neural network, and is used to normalize the output vector to be a unit vector.
The prediction of material characteristic information (such as diffuse reflection characteristic vector and specular reflection characteristic vector) can be added during the neural network training. The specific method comprises the following steps:
(1) for a sampling point, a cube with the side length d and the center at the origin of the acquisition space is adopted to surround the sampling point, and a plurality of points are uniformly sampled on each surface to serve as virtual light sources.
(2) Generating a material characteristic vector, wherein the formula is as follows:
V(I，P)＝Fnear(xl，xp，ωi，np，nl)I(l)fr(ωi；ωo，np，t，p)
wherein I represents lighting information of each virtual light source l, including: spatial position x of virtual light source llNormal vector n of virtual light source lIAnd the luminous intensity I (l) of the virtual light source l, wherein P comprises parameter information of the sampling point, and the method comprises the following steps: spatial position x of the sampling pointpNormal vector n in world coordinate systempAnd the material parameter p of the sampling point of the x-axis direction t of the local coordinate system under the world coordinate system comprises alphax，αy，ρd，ρsIn which α isx，αyDenotes the roughness coefficient, ρdRepresenting the diffuse reflectance, ρsRepresents the specular reflectance; omegaiRepresenting the incident vector, ω, in the world coordinate systemoRepresenting an emergent vector under a world coordinate system; f. ofr(ωi；ωo，npT, p) is the bidirectional reflectance distribution function, Fnear(xl，xp，ωi，np，nl) For the low beam factor, the formula is as follows:
Figure BDA0002190237810000044
(3) setting the specular reflectivity of the sampling point to 0 to generate a diffuse reflection characteristic vector vd。
(4) Setting the diffuse reflectance to 0, generating a specular reflection feature vector vs。
(5) Spirit of the inventionOutput over network increment vector v'd，v′sP ', where v'dAnd vdSame length, v'sAnd vsThe lengths are the same, p 'is 3 and vector v'd,v′sP' are respectively the diffuse reflection eigenvectors v after eliminating the low beam factordSpecular reflection vector v after eliminating passing light factorsAnd spatial position xpAnd (4) predicting.
(6) The loss function of the material characteristic part is expressed as follows:
Lossdiffusc＝||v′dFnear(xl，p′，ω′i，n′，nl)-vd||
Lossspecular＝||v′sFnear(xl，p′，ω′i，n′，nl)-vs||
when training the neural network, adding the diffuse reflection Loss function LossdiffuseAnd Loss of specular reflection function Lossspecular。
In actual use, the material characteristic information can be obtained while the normal vector is obtained.
(7) And acquiring a three-dimensional object material map according to the material characteristic information.
Further, in the step (7), the process of obtaining the material map of the three-dimensional object according to the material characteristic information is as follows:
(7.1) sampling m angles of the target three-dimensional object by using a camera, and then taking a picture obtained by sampling as input to obtain a three-dimensional grid; for each vertex, choose to have ni·ωoMinimum and visible sampling angle ω, niAs a normal vector, omega, of the vertices of a three-dimensional meshoThe emission vector in the world coordinate system is shown.
And (7.2) taking the three-dimensional grid of the three-dimensional object as input to obtain a mapping relation diagram from the material mapping to the grid.
(7.3) mapping each effective pixel on the mapping relation map to a sampling angle omega; the material characteristic v of the point obtained by combining the neural networkiFitting materialA quality parameter; and traversing the effective pixels on all the mapping relational graphs to obtain the final material mapping graph.
Further, in the step (7.3), the L-BFGS-B method is used for fitting the material parameters, and the expression is as follows:
minimize(||vi-NN[fr(ωi；ωo，n′，t，p′)]||)
wherein p ' and t are fitting quantities, p ' is a material parameter vector, t represents the x-axis direction of a local coordinate system of a sampling point of a world coordinate system, n ' is a normal vector of neural network prediction, and NN represents the mapping of a neural network.
A method for optimizing a three-dimensional mesh using normal vectors, the method comprising the steps of:
(1) sampling m angles of a target three-dimensional object by a camera, irradiating illumination patterns obtained by training by using the method at each angle, and taking a picture obtained by sampling as input to obtain an initial three-dimensional point cloud and a three-dimensional grid; with an initial normal vector n at each vertex of the three-dimensional meshi。
(2) And (4) carrying out re-meshing (remesh) on the three-dimensional grid obtained in the step (1).
(3) For each vertex, choose to have ni·ωoA minimum sampling angle omega visible to the point, under which the normal vector predicted value n 'is obtained by the normal vector obtaining method'iAs the normal vector n of the pointi。
(4) Every vertex position P in the current meshiChange to New position P'iTo obtain a new mesh, the process includes a loss function L on the normal vectornormalAnd a vertex position loss function LpositionOptimizing; binding of LnormalAnd LpositionObtaining a joint loss function LoptAnd optimizing L using least squaresopt。
(5) Returning to the step (3) until LoptAnd converging to complete the optimization of the three-dimensional grid, and taking the three-dimensional grid as a final geometric model.
Further, in the step (4),
normal vector loss function Lnormal: polygon { e }0，e1，…，ekAs an approximation of the tangent plane, k is greater than 2, where ejIs the j-th side of a polygon formed by PiA central and adjacent vertex, a loss function LnormalIs expressed as follows:
Figure BDA0002190237810000061
n is the number of three-dimensional mesh vertexes;
vertex position loss function Lposition：
Figure BDA0002190237810000062
Wherein
Figure BDA0002190237810000063
I is an identity matrix, alpha and beta are preset parameters alpha, beta belongs to [0, + ∞ ], and the new position P is realized by adjusting alpha and betaiCorresponding joint loss function LoptThe optimization is achieved;
joint loss function Lopt：
Lopt＝λLposition+(1-λ)Lnormal
Where λ ∈ [0, 1] is used to control the weight of both loss functions.
The invention has the beneficial effects that: the invention provides an acquisition method combined with a neural network on the idea of actively irradiating a plurality of specific patterns on an object, simultaneously acquiring photos and calculating the obtained photos to obtain the normal vector of the object. The average error of the normal vector obtained by the method is 3 degrees, which is much higher than the precision of the prior art. Further, the method uses the obtained normal vectors for optimizing the model of the object. The method can also acquire the material characteristic information while acquiring the normal vector, and finally combine to obtain a high-quality geometric and material acquisition result. The acquisition of material characteristic information does not require making assumptions that the spatial location is known. The three-dimensional object model and the material obtained by combination have extremely strong reality sense, which is higher than the prior level. The number of the illumination patterns obtained by the method is small, the accuracy of the obtained normal vector is high, and the method is not limited to a certain specific acquisition device. In addition, the mode of obtaining the illumination matrix by the method can directly ensure that the generated matrix is effective.
Drawings
FIG. 1 is a schematic perspective view of an acquisition apparatus in an embodiment;
fig. 2 (a) is a plan development view of the acquisition apparatus in the embodiment, 64 × 64 is the number of light sources per plane, and (b) is a side view;
FIG. 3 is a schematic flow chart of an embodiment;
FIG. 4 is a schematic diagram of a neural network used in an embodiment;
FIG. 5 is one of the resulting illumination patterns of the training in the example;
FIG. 6 is a typical Lumitexel used for training in the examples.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be further described in detail with reference to the accompanying drawings and specific embodiments.
The invention provides an acquisition method combined with a neural network on the idea of actively irradiating a plurality of specific patterns on an object, simultaneously acquiring photos and calculating the obtained photos to obtain the normal vector of the object. Further, the method uses the obtained normal vectors for optimizing the model of the object. The method can also acquire the material characteristic information while acquiring the normal vector. And finally, obtaining a high-quality geometric and material acquisition result in a combined manner. The following describes the specific implementation of the three parts in detail:
a three-dimensional object normal vector obtaining method based on a neural network comprises the following steps:
(1) generating training data: acquiring parameters of acquisition equipment, including the distance and angle from a light source to the origin of an acquisition space (a space for placing an acquisition object in the acquisition equipment), a characteristic curve of the light source, the distance and angle from a camera to the origin of the acquisition space, and internal and external parameters of the camera; generating an acquisition result simulating an actual camera by using the parameters as training data; the specific method for generating the acquisition result of the simulated actual camera is as follows:
randomly selecting a sampling point in a collection space where a target three-dimensional object may appear, randomly sampling a material parameter of the point, and generating Lumitexel by using a rendering model, wherein the Lumitexel describes the reflection light intensity of the sampling point to incident light from each light source along a certain observation direction; the rendering model may adopt a GGX model, but is not limited thereto, and the generation formula using the GGX model is as follows:
Figure BDA0002190237810000081
wherein f isr(ωi；ωoN, t, p) is the bidirectional reflectance distribution function, ωoIs the outgoing direction, omegaiIs the incident direction, n represents the normal vector under the world coordinate system, t represents the x-axis direction of the local coordinate system of the sampling point under the world coordinate system, p is the material parameter vector including alphax，αy，ρd，ρsIn which α isxAnd alphayDenotes the roughness coefficient, ρdRepresenting the diffuse reflectance, ρsRepresents the specular reflectance; omegahIs a half way vector, DGGXIs a differential surface distribution term, F is a Fresnel term, GGGXRepresenting a shading coefficient function.
(2) Training a neural network according to the training data obtained in the step (1), wherein the neural network is characterized as follows:
a. the input of the neural network is Lumitexel;
b. the output and regression target of the neural network is a normal vector corresponding to the input Lumitexel;
c. the first layer of the neural network comprises a linear fully-connected layer, and a parameter matrix of the linear fully-connected layer is obtained by training the following formula:
Wl＝fw(Wraw)
wherein WrawIs a parameter to be trained; wlThe illumination matrix is n multiplied by m, n is the sampling precision of Lumitexel (namely the number of light sources of the acquisition equipment), and m is the number of illumination patterns; f. ofwIs a mapping for WrawThe transformation is performed such that the generated illumination matrix can correspond to the possible luminous intensities of the acquisition device (i.e. has a physical meaning). WrawThe size of the matrix and the value range of each element depend on the actual use of fwAnd then. f. ofwA combination of normalization and inversion functions may be selected, where m is an even number, WrawIs of a size of
Figure BDA0002190237810000082
Figure BDA0002190237810000083
Can combine WrawViewed as a
Figure BDA0002190237810000084
The combination of individual column vectors, expressed as follows:
Figure BDA0002190237810000085
fwto WrawIs normalized to be a unit vector, and then the negative number of each column is turned over to form a new vector, and the formula is expressed as follows:
fw(Wraw)＝fflip(fnormalize(Wraw))
Figure BDA0002190237810000091
Wnormalizedcan be regarded as
Figure BDA0002190237810000092
A column vector, represented as follows:
Figure BDA0002190237810000093
Figure BDA0002190237810000094
d. the second and subsequent layers are nonlinear mapping networks, wherein the output vector length of the last layer is 3, and preferably, the last layer can be followed by a normalization layer for normalizing the output vector to be a unit vector.
And after the training is finished, taking out the parameter matrix (namely the illumination matrix) of the first layer of linear full-connected layer.
(3) Enabling the acquisition equipment to generate illumination patterns according to the illumination matrix extracted in the step (2), and sequentially illuminating the target three-dimensional object to obtain a group of photos r1，r2...，rm(ii) a Sequentially traversing the picture's pixels and channels, from r each time1，r2...，rmThe pixel values of the ith row, the jth column and the kth channel are taken out to form a vector a ═ a1，a2，…，amAnd a is used as an output vector of the first layer of linear full-connection layer of the neural network, and the output vector of the last layer is obtained through calculation, namely the normal vector of the surface of the three-dimensional object corresponding to the pixel is obtained. And traversing all the pixels to obtain a normal vector characteristic diagram of the surface of the three-dimensional object.
Secondly, a method for obtaining the material of the three-dimensional object based on the neural network, which comprises the following steps:
the prediction of material characteristic information (such as diffuse reflection characteristic vector and specular reflection characteristic vector) can be added during the neural network training. The specific method comprises the following steps:
(1) for a sampling point, a cube with the side length of d and the center at the origin of the acquisition space is adopted to surround the sampling point, a plurality of points are uniformly sampled on each surface to serve as virtual light sources, the virtual light sources are point light sources, and the number of the virtual light sources on the cube is called the sampling precision of the characteristic vector. The use of virtual light sources eliminates the undesirable effects of gaps that may exist between the light sources of the acquisition device.
(2) The formula for generating the material feature vector is as follows:
V(I，P)＝Fnear(xl，xp，ωi，np，nl)I(l)fr(ωi；ωo，np，t，p)
wherein I represents lighting information of each virtual light source l, including: spatial position x of virtual light source llNormal vector n of virtual light source llAnd the luminous intensity I (l) of the virtual light source l, wherein P comprises parameter information of the sampling point, and the method comprises the following steps: spatial position x of the sampling pointpNormal vector n in world coordinate systempAnd the material parameter p of the sampling point of the x-axis direction t of the local coordinate system under the world coordinate system comprises alphax，αy，ρd，ρsIn which α isx，αyDenotes the roughness coefficient, ρdRepresenting the diffuse reflectance, ρsRepresents the specular reflectance; omegaiRepresenting the incident vector, ω, in the world coordinate systemoThe emission vector in the world coordinate system is shown. f. ofr(ωi；ωo，npT, p) is the bidirectional reflectance distribution function, Fnear(xl，xp，ωi，np，nl) For the low beam factor, the formula is as follows:
Figure BDA0002190237810000101
(3) using the above formula, a diffuse reflection feature vector v is generated with the specular reflectance of the sample point set to 0d。
(4) Using the above formula, a specular reflection feature vector v is generated with the diffuse reflectance set to 0s。
(5) Output of neural network is incremented by vector v'd，v′sP ', where v'dAnd vdSame length, v'sAnd vsThe lengths are the same, p 'is 3 and vector v'd，v′sP' are respectively the diffuse reflection eigenvectors v after eliminating the low beam factordSpecular reflection vector v after eliminating passing light factorsAnd spatial position xpAnd (4) predicting.
(6) The loss function of the material characteristic part is expressed as follows:
Lossdiffusc＝||v′dFnear(xl，p′，ω′i，n′，nl)-vd|||
Lossspecular＝||v′sFnear(xl，p′，ω′i，n′，nl)-vs||
when training the neural network, adding the diffuse reflection Loss function LossdiffuseAnd Loss of specular reflection function Lossspecular。
In actual use, the material characteristic information can be obtained while the normal vector is obtained.
(7) Obtaining a three-dimensional object material chartlet according to the material characteristic information, wherein the process is as follows:
(7.1) sampling m angles of the target three-dimensional object by using a camera, taking a picture obtained by sampling as input, and obtaining the three-dimensional grid by using a tool COLMAP disclosed in the industry. For each vertex, choose to have ni·ωoThe smallest and visible sampling angle ω for that point. n isiAs a normal vector, omega, of the vertices of a three-dimensional meshoThe emission vector in the world coordinate system is shown.
And (7.2) using an Iso-characters in the field, taking the three-dimensional grid of the three-dimensional object as input, and obtaining a mapping relation diagram (hereinafter referred to as a mapping relation diagram) from the material mapping to the grid.
(7.3) mapping each effective pixel on the mapping relation map to a sampling angle omega; the material characteristic v of the point obtained by combining the neural networki(including diffuse and specular reflectance characteristics), the material parameters were fitted using the L-BFGS-B method. The expression is as follows:
minimize(||vi-NN[fr(ωi；ωo，n′，t，p′)]||)
wherein p ' and t are fitting quantities, p ' is a material parameter vector, t represents the x-axis direction of a local coordinate system of a sampling point of a world coordinate system, n ' is a normal vector of neural network prediction, and NN represents the mapping of a neural network. And traversing the effective pixels on all the mapping relational graphs to obtain the final material mapping graph. The fitting method at this stage is not limited to the L-BFGS-B method.
Thirdly, a method for optimizing a three-dimensional grid by using a normal vector, which comprises the following steps:
(1) the target three-dimensional object is sampled by a camera at m angles, the illumination pattern obtained by the training of the first part is used for illumination at each angle, the picture obtained by sampling is used as input, and an initial three-dimensional point cloud and a three-dimensional grid can be obtained by using a tool COLMAP disclosed in the industry. With an initial normal vector n at each vertex of the three-dimensional meshi。
(2) And (2) performing re-meshing (remesh) on the three-dimensional grid obtained in the step (1), for example, Delaunay triangulation can be adopted.
(3) For each vertex, choose to have ni·ωoA sampling angle omega at which the point is visible is obtained, and a normal vector predicted value n 'is obtained through the neural network-based three-dimensional object normal vector obtaining method under the sampling angle'iAs the normal vector n of the pointi。
(4) Every vertex position P in the current meshiChange to New position P'iTo obtain a new mesh, the process includes a loss function L on the normal vectornormalAnd a vertex position loss function LpositionAnd (4) optimizing.
Normal vector loss function LnormalIn that the property that the normal vector is perpendicular to the tangent plane is utilized to limit P 'in the new grid'iThe tangent plane and the corresponding normal vector niPerpendicular to each other, a polygon { e'0，e′1，…，e′kAs an approximation of the tangent plane, k is greater than 2, where ejIs the j-th side of a polygon composed of P'iA central and adjacent vertex, a loss functionNumber LnormalIs expressed as follows:
Figure BDA0002190237810000121
n is the number of three-dimensional mesh vertexes;
at the same time, the new position P 'is required'iLimiting to prevent singular points, thus requiring a large degree of freedom in the movement of the new position in the normal vector direction, and a relatively small degree of freedom in the tangential direction, thus having a loss function Lposition：
Figure BDA0002190237810000122
Wherein
Figure BDA0002190237810000123
I is an identity matrix, alpha and beta are preset parameters alpha, beta belongs to [0, + ∞ ], and the new position P 'is realized by adjusting alpha and beta'iCorresponding joint loss function LoptAnd the optimization is achieved. (preferred values are relatively small α and larger β). Binding of LnormalAnd LpositionObtaining a joint loss function LoptAnd optimizing L using least squaresopt；
Lopt＝λLposition+(1-λ)Lnormal
Where λ ∈ [0, 1] is used to control the weight of both loss functions (preferred value is [0.1, 0.3 ]).
(5) Returning to the step (3) until LoptAnd (4) converging, finishing the optimization of the three-dimensional grid, taking the three-dimensional grid as a final geometric model, and applying the optimized three-dimensional grid to the step (7) of the second part to obtain a high-quality material result.
A specific example of the collecting device is given below, and as shown in fig. 1 and 2, the collecting device is composed of 48 lamp panels, and a camera is fixed on the upper portion of the collecting device and used for collecting images. The center of the device is provided with a rotating platform driven by a stepping motor and used for placing a sampling object. 20480 LED lamp beads are densely arranged on each lamp panel. The lamp beads are controlled by the FPGA, and the brightness and the lighting time can be adjusted.
An example of an acquisition system applying the method of the present invention is given below, and the system is generally divided into the following modules: a preparation module: the method comprises two parts, a data set is provided for network training, and a group of BRDF parameters, a spatial position where a point is located and a camera position are input by using a GGX BRDF model, so that a Lumitexel can be obtained. The network training portion uses a Tensorflow open source framework, depicts the network as a graph, and trains using an Adam optimizer. The network structure is shown in fig. 4, each rectangle represents a layer of neurons, and the numbers in the rectangles represent the number of neurons in the layer. The leftmost layer is the input layer and the rightmost layer is the output layer. Full connection is used between layers. An acquisition module: the device is shown in fig. 1 and 2, and the specific structure is described above. A recovery module: and loading the trained neural network, and firstly calculating to obtain normal vector information and material characteristic vectors. The coarse three-dimensional mesh obtained using the COLMAP software was then optimized using the method described above. And calculating to obtain a mapping relation graph from the optimized geometric model. And performing material parameter fitting on each effective pixel on the mapping relation diagram.
Fig. 3 is a flowchart of the present embodiment. Firstly, generating training data, randomly sampling to obtain 2 hundred million Lumitexels, taking 80% as a training set, and taking the rest as a verification set. And initializing parameters by using an Xavier method during network training, wherein the learning rate is 1 e-4. The number of target shots is 32, so the size of the illumination matrix is: (20480,32). After the training is finished, the illumination matrix is taken out as an illumination pattern, the parameters of each column specify the light intensity of the lamp at the position, and fig. 5 shows the illumination pattern obtained by the training. The object is sampled at 24 angles, and the object is shot for 32 times at each angle according to the luminous intensity of the illumination pattern to obtain an acquisition result. And combining the pixel values of the 32 acquired pictures into a vector for each point on the acquired pictures. The following processing method comprises the following steps: 1. and loading parameters after the second layer of the network, and inputting the parameters into the acquired vectors. The method is used for recovering and obtaining the spatial position, the normal vector information and the diffuse reflection information at each position. 2. And performing geometric recovery by using COLMAP open source software to obtain point cloud and three-dimensional grid. 3. And using the normal vector acquisition method to obtain the geometry and the predicted normal vector as input, and iteratively optimizing the geometry. 4. And obtaining a mapping relation graph by using Isochart. 5. And finding corresponding real shooting data for each effective pixel on the mapping relation graph, and recovering the diffuse reflection characteristic vector and the specular reflection characteristic vector. A fitting module: and fitting the BRDF parameters to each collected point pixel by using an L-BFGS-B method.
The above description is only a preferred embodiment of the present invention, and the present invention is not limited to the above embodiment, and the present invention shall fall within the protection scope of the present invention as long as the technical effects of the present invention are achieved by the same means. Within the scope of protection of the present invention, various modifications and variations of the technical solution and/or embodiments thereof are possible.

Claims (10)

1. A three-dimensional object normal vector acquisition method based on a neural network is characterized by comprising the following steps:
(1) generating training data: acquiring parameters of acquisition equipment, including the distance and angle from a light source to an acquisition space origin, a characteristic curve of the light source, the distance and angle from a camera to the acquisition space origin, and internal and external parameters of the camera; generating an acquisition result simulating an actual camera by using the parameters as training data;
(2) training a neural network according to the training data obtained in the step (1), wherein the neural network is characterized as follows:
a. the input of the neural network is Lumitexel, which describes the reflection intensity of the sampling point to the incident light from each light source along a certain observation direction;
b. the output and regression target of the neural network is a normal vector corresponding to the input Lumitexel;
c. the first layer of the neural network comprises a linear fully-connected layer, and a parameter matrix of the linear fully-connected layer is obtained by training the following formula:
Wl＝fw(Wraw)
wherein WrawIs a parameter to be trained; wlThe illumination matrix is n multiplied by m, n is the sampling precision of Lumitexel, and m is the number of illumination patterns; f. ofwIs a mapping for WrawTransforming to enable the generated illumination matrix to correspond to the possible luminous intensity of the acquisition equipment;
d. the second layer and the later layers are nonlinear mapping networks, wherein the length of the output vector of the last layer is 3;
after training is finished, taking out the illumination matrix of the first layer of linear full-connection layer;
(3) enabling the acquisition equipment to generate illumination patterns according to the illumination matrix extracted in the step (2), and sequentially illuminating the target three-dimensional object to obtain a group of photos r1，r2...，rm(ii) a Sequentially traversing the picture's pixels and channels, from r each time1，r2...，rmThe pixel values of the ith row, the jth column and the kth channel are taken out to form a vector a ═ a1，a2，…，amTaking a as an output vector of a first layer of linear full-connection layer of the neural network, and calculating to obtain an output vector of a last layer, wherein the obtained output vector is a normal vector of the surface of the three-dimensional object corresponding to the pixel; and traversing all the pixels to obtain a normal vector characteristic diagram of the surface of the three-dimensional object.
2. The method for obtaining the normal vector of the three-dimensional object based on the neural network as claimed in claim 1, wherein in the step (1), the specific method for generating the acquisition result simulating the actual camera is as follows: randomly selecting a sampling point in a collection space where a target three-dimensional object may appear, randomly sampling a material parameter of the point, and generating Lumitexel by using a rendering model.
3. The method for obtaining the normal vector of the three-dimensional object based on the neural network as claimed in claim 2, wherein the rendering model adopts a GGX model, and the generation formula is as follows:
Figure FDA0002946205640000021
wherein f isr(ωi；ωoN, t, p) is the bidirectional reflectance distribution function, ωoIs the outgoing direction, omegaiIs the incident direction, n represents the normal vector under the world coordinate system, t represents the x-axis direction of the local coordinate system of the sampling point under the world coordinate system, p is the material parameter vector including alphax，αy，ρd，ρsIn which α isxAnd alphayDenotes the roughness coefficient, ρdRepresenting the diffuse reflectance, ρsRepresents the specular reflectance; omegahIs a half way vector, DGGXIs a differential surface distribution term, F is a Fresnel term, GGGXRepresenting a shading coefficient function.
4. The method for obtaining the normal vector of the three-dimensional object based on the neural network as claimed in claim 1, wherein in the step (2), f is mappedwSelecting a combination of a normalization function and a turnover function, wherein m is an even number, and W is requiredrawIs of a size of
Figure FDA0002946205640000022
Can combine WrawViewed as a
Figure FDA0002946205640000023
The combination of individual column vectors, expressed as follows:
Figure FDA0002946205640000024
fwto WrawIs normalized to be a unit vector, and then the negative number of each column is turned over to form a new vector, and the formula is expressed as follows:
fw(Wraw)＝fflip(fnormalize(Wraw))
Figure FDA0002946205640000025
Wnormalizedcan be regarded as
Figure FDA0002946205640000026
A column vector, represented as follows:
Figure FDA0002946205640000027
Figure FDA0002946205640000028
5. the method according to claim 1, wherein in the step (2), a normalization layer is further connected after the last layer of the neural network, and is used for normalizing the output vector to be a unit vector.
6. A method for obtaining the material quality of a three-dimensional object based on a neural network, which is characterized in that the method adds the prediction of the material characteristic information when the neural network is trained according to claim 1, and comprises the following steps:
(1) for a sampling point, a cube with the side length d and the center at the origin of an acquisition space is adopted to surround the sampling point, and a plurality of points are uniformly sampled on each surface to serve as virtual light sources;
(2) generating a material characteristic vector, wherein the formula is as follows:
V(I，P)＝Fnear(xl，xp，ωi，np，nl)I(l)fr(ωi；ωo，np，t，p)
wherein I represents lighting information of each virtual light source l, including: spatial position x of virtual light source llNormal vector n of virtual light source llVirtual, virtualThe luminous intensity i (l) of the light source l, P contains parameter information of the sampling points, and includes: spatial position x of the sampling pointpNormal vector n in world coordinate systempAnd the material parameter p of the sampling point of the x-axis direction t of the local coordinate system under the world coordinate system comprises alphax，αy，ρd，ρsIn which α isx，αyDenotes the roughness coefficient, ρdRepresenting the diffuse reflectance, ρsRepresents the specular reflectance; omegaiRepresenting the incident vector, ω, in the world coordinate systemoRepresenting an emergent vector under a world coordinate system; f. ofr(ωi；ωo，npT, p) is the bidirectional reflectance distribution function, Fnear(xl，xp，ωi，np，nl) For the low beam factor, the formula is as follows:
Figure FDA0002946205640000031
(3) setting the specular reflectivity of the sampling point to 0 to generate a diffuse reflection characteristic vector vd；
(4) Setting the diffuse reflectance to 0, generating a specular reflection feature vector vs；
(5) Output of neural network is incremented by vector v'd，v′sP ', where v'dAnd vdSame length, v'sAnd vsThe lengths are the same, p 'is 3 and vector v'd，v′sP' are respectively the diffuse reflection eigenvectors v after eliminating the low beam factordSpecular reflection vector v after eliminating passing light factorsAnd spatial position xpPredicting;
(6) the loss function of the material characteristic part is expressed as follows:
Lossdiffuse＝||v′dFnear(xl，p′，ω′i，n′，nl)-vd||
Lossspecular＝||v′sFnear(xl，p′，ω′i，n′，nl)-vs||
when training the neural network, adding the diffuse reflection Loss function LossdiffuseAnd Loss of specular reflection function Lossspecular；
In actual use, the material characteristic information can be obtained while the normal vector is obtained;
(7) and acquiring a three-dimensional object material map according to the material characteristic information.
7. The method for obtaining the material quality of the three-dimensional object based on the neural network as claimed in claim 6, wherein in the step (7), the process of obtaining the material quality map of the three-dimensional object according to the material characteristic information is as follows:
(7.1) sampling m angles of the target three-dimensional object by using a camera, and then taking a picture obtained by sampling as input to obtain a three-dimensional grid; for each vertex, choose to have ni·ωoMinimum and visible sampling angle ω, niAs a normal vector, omega, of the vertices of a three-dimensional meshoRepresenting an emergent vector under a world coordinate system;
(7.2) taking the three-dimensional grid of the three-dimensional object as input to obtain a mapping relation graph from the material mapping to the grid;
(7.3) mapping each effective pixel on the mapping relation map to a sampling angle omega; the material characteristic v of the point obtained by combining the neural networkiFitting material parameters; and traversing the effective pixels on all the mapping relational graphs to obtain the final material mapping graph.
8. The method for obtaining the material of the three-dimensional object based on the neural network as claimed in claim 7, wherein in the step (7.3), the L-BFGS-B method is used to fit the material parameters, and the expression is as follows:
minimize(||vi-NN[fr(ωi；ωo，n′，t，p′)]||)
wherein p ' and t are fitting quantities, p ' is a material parameter vector, t represents the x-axis direction of a local coordinate system of a sampling point of a world coordinate system, n ' is a normal vector of neural network prediction, and NN represents the mapping of a neural network.
9. A method for optimizing a three-dimensional mesh using normal vectors, comprising the steps of:
(1) sampling m angles of a target three-dimensional object by a camera, irradiating the target three-dimensional object by using an illumination pattern obtained by training according to the method of claim 1 at each angle, and taking a picture obtained by sampling as input to obtain an initial three-dimensional point cloud and a three-dimensional grid; with an initial normal vector n at each vertex of the three-dimensional meshi；
(2) Carrying out re-meshing (remesh) on the three-dimensional grid obtained in the step (1);
(3) for each vertex, choose to have ni·ωoA minimum and this point visible sampling angle ω at which normal vector predictor n 'is derived by the method of claim 1'iAs the normal vector n of the pointi；
(4) Every vertex position P in the current meshiChange to New position P'iTo obtain a new mesh, the process includes a loss function L on the normal vectornormalAnd a vertex position loss function LpositionOptimizing; binding of LnormalAnd LpositionObtaining a joint loss function LoptAnd optimizing L using least squaresopt；
(5) Returning to the step (3) until LoptAnd converging to complete the optimization of the three-dimensional grid, and taking the three-dimensional grid as a final geometric model.
10. The method of optimizing a three-dimensional mesh using normal vectors according to claim 9, wherein in the step (4),
normal vector loss function Lnormal: polygon { e'0，e′1，…，e′kAs an approximation of the tangent plane, k is greater than 2, where e'jIs the j-th side of a polygon composed of P'iA central and adjacent vertex, a loss function LnormalIs expressed as follows:
Figure FDA0002946205640000051
n is the number of three-dimensional mesh vertexes;
vertex position loss function Lposition：
Figure FDA0002946205640000052
Wherein
Figure FDA0002946205640000053
I is an identity matrix, alpha and beta are preset parameters alpha, beta belongs to [0, + ∞ ], and the new position P 'is realized by adjusting alpha and beta'iCorresponding joint loss function LoptThe optimization is achieved;
joint loss function Lopt：
Lopt＝λLposition+(1-λ)Lnormal
Where λ ∈ [0, 1] is used to control the weight of both loss functions.
CN201910829629.XA 2019-09-03 2019-09-03 Method for acquiring normal vector, geometry and material of three-dimensional object based on neural network Active CN110570503B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910829629.XA CN110570503B (en) 2019-09-03 2019-09-03 Method for acquiring normal vector, geometry and material of three-dimensional object based on neural network

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910829629.XA CN110570503B (en) 2019-09-03 2019-09-03 Method for acquiring normal vector, geometry and material of three-dimensional object based on neural network

Publications (2)

Publication Number Publication Date
CN110570503A CN110570503A (en) 2019-12-13
CN110570503B true CN110570503B (en) 2021-04-16

Family

ID=68777640

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910829629.XA Active CN110570503B (en) 2019-09-03 2019-09-03 Method for acquiring normal vector, geometry and material of three-dimensional object based on neural network

Country Status (1)

Country Link
CN (1) CN110570503B (en)

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112419334A (en) * 2020-11-18 2021-02-26 山东大学 Micro surface material reconstruction method and system based on deep learning
CN112488176B (en) * 2020-11-26 2023-11-21 江苏科技大学 Processing characteristic recognition method based on triangular mesh and neural network
CN112734832A (en) * 2021-01-22 2021-04-30 逆可网络科技有限公司 Method for measuring real size of on-line object in real time
CN115249215A (en) * 2021-04-28 2022-10-28 北京字跳网络技术有限公司 Image processing method, image processing device, electronic equipment and readable storage medium
WO2022256962A1 (en) * 2021-06-07 2022-12-15 浙江大学 Freestyle acquisition method for high-dimensional material
CN113362466B (en) * 2021-06-07 2022-06-21 浙江大学 Free type collection method for high-dimensional material
CN115375847B (en) * 2022-08-25 2023-08-29 北京百度网讯科技有限公司 Material recovery method, three-dimensional model generation method and model training method

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7319467B2 (en) * 2005-03-29 2008-01-15 Mitsubishi Electric Research Laboratories, Inc. Skin reflectance model for representing and rendering faces
CN105389843A (en) * 2015-12-09 2016-03-09 河海大学 Global illumination real-time rendering method based on radial basis function neural network fitting
CN107679477A (en) * 2017-09-27 2018-02-09 深圳市未来媒体技术研究院 Face depth and surface normal Forecasting Methodology based on empty convolutional neural networks
CN108985333A (en) * 2018-06-15 2018-12-11 浙江大学 A kind of material acquisition methods neural network based and system
CN109658489A (en) * 2018-12-17 2019-04-19 清华大学 A kind of stereoscopic grid data processing method neural network based and system
CN109993692A (en) * 2019-04-09 2019-07-09 大连海事大学 A kind of data fusion method of electronic chart and radar image based on deep learning

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102096941B (en) * 2011-01-30 2013-03-20 北京航空航天大学 Consistent lighting method under falsehood-reality fused environment
CN109784288B (en) * 2019-01-22 2023-01-06 天津师范大学 Pedestrian re-identification method based on discrimination perception fusion

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7319467B2 (en) * 2005-03-29 2008-01-15 Mitsubishi Electric Research Laboratories, Inc. Skin reflectance model for representing and rendering faces
CN105389843A (en) * 2015-12-09 2016-03-09 河海大学 Global illumination real-time rendering method based on radial basis function neural network fitting
CN107679477A (en) * 2017-09-27 2018-02-09 深圳市未来媒体技术研究院 Face depth and surface normal Forecasting Methodology based on empty convolutional neural networks
CN108985333A (en) * 2018-06-15 2018-12-11 浙江大学 A kind of material acquisition methods neural network based and system
CN109658489A (en) * 2018-12-17 2019-04-19 清华大学 A kind of stereoscopic grid data processing method neural network based and system
CN109993692A (en) * 2019-04-09 2019-07-09 大连海事大学 A kind of data fusion method of electronic chart and radar image based on deep learning

Also Published As

Publication number Publication date
CN110570503A (en) 2019-12-13

Similar Documents

Publication Publication Date Title
CN110570503B (en) Method for acquiring normal vector, geometry and material of three-dimensional object based on neural network
US20210209340A1 (en) Methods for obtaining normal vector, geometry and material of three-dimensional objects based on neural network
CN109410256B (en) Automatic high-precision point cloud and image registration method based on mutual information
Huang et al. Point cloud labeling using 3d convolutional neural network
CN107945267A (en) A kind of method and apparatus for human face three-dimensional model grain table
CN109712172A (en) A kind of pose measuring method of initial pose measurement combining target tracking
CN104115074B (en) hologram processing method and system
CN111679291B (en) Inspection robot target positioning configuration method based on three-dimensional laser radar
CN109410321A (en) Three-dimensional rebuilding method based on convolutional neural networks
CN114972617B (en) Scene illumination and reflection modeling method based on conductive rendering
CN106846392A (en) The method and apparatus of three-dimensional modeling
CN110349247A (en) A kind of indoor scene CAD 3D method for reconstructing based on semantic understanding
CN108154104A (en) A kind of estimation method of human posture based on depth image super-pixel union feature
CN108090896B (en) Wood board flatness detection and machine learning method and device and electronic equipment
CN107657656A (en) Homotopy mapping and three-dimensional rebuilding method, system and photometric stereo camera shooting terminal
WO2021219835A1 (en) Pose estimation method and apparatus
CN108985333B (en) Material obtaining method and system based on neural network
CN109523619A (en) A method of 3D texturing is generated by the picture of multi-angle polishing
CN116958420A (en) High-precision modeling method for three-dimensional face of digital human teacher
Xie Three-dimensional reconstruction based on multi-view photometric stereo fusion technology in movies special-effect
Sorokin et al. Restoration of lighting parameters in mixed reality systems using convolutional neural network technology based on RGBD images
CN111915735A (en) Depth optimization method for three-dimensional structure contour in video
CN115661251A (en) Imaging simulation-based space target identification sample generation system and method
CN115953447A (en) Point cloud consistency constraint monocular depth estimation method for 3D target detection
CN114241059B (en) Synchronous calibration method for camera and light source in photometric stereo vision system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant