CN113362466B - Free type collection method for high-dimensional material - Google Patents

Abstract

The invention discloses a free-style acquisition method for high-dimensional materials, belonging to the fields of computer graphics and computer vision. The method proposes to transform the learning of material information into a geometric learning problem on an unstructured point cloud. The acquisition results under the illumination and observation direction form a high-dimensional point cloud, and each point in the point cloud is a vector composed of the image measurement value and the pose information of the object when the image was captured. Efficiently aggregate information from unstructured views in homogeneous and precision-limited high-dimensional point clouds to recover high-quality material properties.

Description

A free-style acquisition method for high-dimensional materials

technical field

The invention relates to a free-style acquisition method for high-dimensional materials, belonging to the fields of computer graphics and computer vision.

Background technique

Digitization of real-world objects is one of the central problems in computer graphics and vision. At present, digitized real objects can be represented by three-dimensional mesh models and six-dimensional space-varying bidirectional reflectance distribution functions (SVBRDF). There are important applications in areas such as heritage, e-commerce, computer games and film production.

While high-precision geometric models can be readily obtained with commercial mobile 3D scanners, it is also desirable to develop a lightweight device for free-looking scanning for the following reasons. First, it can scan objects of different sizes as long as the pose of the camera can be estimated reliably. Second, the mobility of the device makes it possible to perform on-site scans of objects that are not allowed to be transported, such as precious cultural relics. Furthermore, the short time and low cost required to manufacture lightweight devices make them accessible to a wider audience. It also provides a user-friendly experience similar to geometry scanning.

Despite the surge in demand, effective non-planar appearance scanning remains an open problem. On the one hand, most of the existing mobile appearance scanning work is shot in the case of single point/parallel light, which leads to the low sampling efficiency of four-dimensional observation and illumination direction, and the exchange of spatial resolution for angular accuracy requires first. empirical knowledge (Giljoo Nam, Joo Ho Lee, Diego Gutierrez, and Min H Kim. 2018. Practical SVBRDF acquisition of 3D objects with unstructured flash photography. In SIGGRAPH Asia Technical Papers. 267.). On the other hand, fixed acquisition systems rely on fixed view conditions when lighting changes, and it is unclear how to extend this to mobile devices, which have unstructured, constantly changing views, and due to their small form factor size without fully covering the lighting field.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a free-form acquisition method for high-dimensional materials in view of the deficiencies of the prior art. The quality of the object material property.

The invention discloses a free-form acquisition method for high-dimensional materials. The main idea of the method is that the free-form appearance scan can be converted into a geometric learning problem on an unstructured point cloud, and each point in the point cloud can represent An image measurement and the pose information of the object when the image was taken. Based on this idea, the present invention designs a neural network that can effectively aggregate information in different unstructured views, reconstruct spatially independent reflection properties, and can optimize the lighting patterns used in the acquisition stage, and finally obtain high-quality materials Collect results. The present invention does not depend on a specific collection device, and a fixed object can be used for collection by a person holding the device, or the object is placed on a turntable for rotation and collected by a fixed device, but is not limited to these two methods.

This method proposes to transform the learning of material information into a geometric learning problem on an unstructured point cloud. Multiple sampling results under different illumination and viewing directions are formed into a high-dimensional point cloud, and each point in the point cloud is an image measurement. This method can effectively aggregate the information of unstructured views from high-dimensional point clouds that are disordered, irregular, unevenly distributed, and limited in accuracy, and recover high-resolution images. Quality material properties. The formal representation is as follows:

F(G(high dimensional point cloud))=m

Among them, the point cloud data feature extraction method G is not limited to a specific network structure, and other methods that can extract features from point clouds are also applicable; the nonlinear mapping network F is not limited to a fully connected network; the expression of object material properties is not limited to Lumitexel vector m.

The method consists of two phases: training phase and acquisition phase.

The training phase includes the following steps:

(1) Obtain the parameters of the acquisition device, and generate the acquisition results that simulate the actual camera as training data;

(2) Use the generated training data to train the neural network. The characteristics of the neural network are as follows:

(2.1) The input of the neural network is the Lumitexel vector under k unstructured samples, where k is the number of samples, and each value of the Lumitexel describes the reflection of the incident light from each light source along a certain viewing direction by the sampling point Light intensity, Lumitexel has a linear relationship with the luminous intensity of the light source, which is simulated by a linear fully connected layer;

(2.2) The first layer of the neural network includes a linear fully connected layer, which is used to simulate the illumination pattern used in actual acquisition, and transform the k Lumitexels into camera acquisition results, which are respectively related to the corresponding sampling points. The pose information is combined to form a high-dimensional point cloud;

(2.3) Starting from the second layer as a feature extraction network, independently perform feature extraction from each point in the high-dimensional point cloud to obtain a feature vector;

(2.4) The feature extraction network is followed by a maximum pooling layer, which is used to aggregate the feature vectors extracted from k unstructured views to obtain a global feature vector;

(2.5) After the maximum pooling layer, a nonlinear mapping network is used to restore high-dimensional material information according to the global feature vector;

The collection stage includes the following steps:

(1) Material collection: The collection device sequentially illuminates the target 3D object according to the illumination pattern, the camera obtains a set of photos under unstructured views, and uses the photos as input to obtain the geometric model of the sampled object with texture coordinates and shoot The pose of the camera when the photo is taken;

(2) Material recovery: According to the pose of the camera when the photo was taken in the material collection stage, the pose of the vertex corresponding to each valid texture coordinate on the sampled object when each photo was taken is obtained; according to the collected photos and pose information, The high-dimensional point cloud is formed as the input of the second layer feature extraction network of the neural network, and the high-dimensional material information is obtained by calculation.

Further, the unstructured sampling is a free random sampling with a non-fixed viewing angle, and the sampling data is disordered, irregular, and unevenly distributed, and a fixed object can be used, which can be collected by a hand-held collection device, or the object can be placed on a turntable. Rotating, stationary equipment acquisition.

Further, in the training data generation process, when the light source is colored, it is necessary to correct the spectral response relationship between the light source, the sampled object and the camera. The correction method is as follows:

λ表示波长，c₁表示RGB三通道中的一个，光强为{I_R，I_G，I_B}的光源的光谱分布曲线L(λ)可以表示为：The spectral distribution curve of the unknown color light source L is defined as

λ represents the wavelength, c ₁ represents one of the three RGB channels, and the spectral distribution curve _L (λ) of a light source whose light intensity is {IR , _IG , _IB } can be expressed as:

的线性组合，c₂表示RGB三通道中的一个：The reflection spectral distribution curve p(λ) of any sampling point p is expressed as three unknown bases with coefficients p _R , p _G , and p _B respectively

A linear combination of , c ₂ represents one of the three RGB channels:

的线性组合；对于在光强为{I_R，I_G，I_B}的光源照射下，摄像机对于反射系数为{p_R，p_G，p_B}的采样点在某一特定通道c₃的测量值如下式：The spectral distribution curve of camera C is expressed as

_{linear combination of _ _ _ _} The measured value is as follows:

Under the illumination condition {IR, _IG , _IB }={1, 0, 0} _{/{0, 1, 0}/{0, 0, 1}, for the known reflection coefficient {p R ,} P _G , p _B } color test card for shooting, establish a linear equation system according to the measurement values collected by the camera, and solve the color correction matrix δ(c ₁ , c ₂ , c ₃ ) with a size of 3×3×3, Represents the spectral response relationship between the light source, the sampled object, and the camera.

Further, in step (2.1) of the training phase, the relationship between the observation value B on the photo of a sampling point p on the surface of the object, the reflection function _fr and the light intensity of each light source can be described as:

Among them, I represents the luminous information of each light source 1, including: the spatial position x _l of the light source 1, the normal vector n _l of the light source 1, the luminous intensity I(l) of the light source 1, and P contains the parameter information of the sampling point p, including : the spatial position x _p of the sampling point, the material parameters n, t, α _x , α _y , ρ _d , ρ _s . Ψ(x _l , ·) describes the light intensity distribution of light source l under different incident directions, V represents the binary function of the visibility of x _l to x _p , and (·) ⁺ is the dot product operation of two vectors. fr (ω′ _i ; ω′ _o , P _{) is the two-dimensional reflection function about ω′ i when} ω′ _o is fixed.

The input of the neural network is the Lumitexel vector under k unstructured sampling, which is denoted as m(l;P);

In the above formula, B is the representation in a single channel. When the light source is a color light source, B is expanded to the following form:

(ω′ _i ·n _p ) ⁺ (-ω′ _i ·n _l ) ⁺ δ(c ₁ , c ₂ , c ₃ )dx _l

的结果。Among them, fr (ω′ _i ; ω′ _o , P, c ₂ ) is in _{fr (ω′ i ;} ω′ _o , P)

the result of.

Further, in step (2.3) of the training phase, the formula of the feature extraction network is as follows:

分别为第j次采样时采样点的空间位置，采样点几何法向量和几何切向量，

由几何模型获得，

为与

正交的任意单位向量，通过第j次采样摄像机的位姿可以将

转换得到

V_feature(j)为网络输出的第j次采样的特征向量。Among them, f is a one-dimensional convolution function, the size of the convolution kernel is 1×1, B(I, P ^j ) represents the result of the first layer network output or the collected measurement value,

are the spatial position of the sampling point in the jth sampling, the geometric normal vector and geometric tangent vector of the sampling point,

obtained from the geometric model,

for and

Orthogonal arbitrary unit vector, by sampling the pose of the camera at the jth time, the

convert to get

V _feature (j) is the feature vector of the jth sampling output by the network.

Further, in step (2.5) of the training phase, the nonlinear mapping network is formally expressed as follows:

和

为最大池化层输出的全局特征向量。Among them, f _i+1 is the mapping function of the i+1 layer network, W _i+1 is the parameter matrix of the i+1 layer network, b _i+1 is the offset vector of the i+1 layer network, y _{i +1} is the output of the i+1th layer network, d and s represent the two branches of diffuse reflection and specular reflection, respectively.

and

The global feature vector output for the max pooling layer.

Further, the loss function of the neural network is designed as follows:

z轴方向为

为几何法向量，

为与

正交的任意单位向量；(1) A virtual Lumitextel space is a cube whose center is at the spatial position x _p of the sampling point, and the direction of the x-axis of the cube center coordinate system is

The z-axis direction is

is the geometric normal vector,

for and

Orthogonal arbitrary unit vectors;

(2) A virtual camera, the observation direction is the positive direction of the z-axis of the cube;

(3) For the diffuse reflection Lumitexel, the cube resolution is 6×N _d ² , for the specular reflection Lumitexel, the cube resolution is 6×N _s ² , that is, N _d ² and N _s ² points are uniformly sampled on each surface as A virtual point light source whose light intensity is unit light intensity;

a. Set the specular reflectivity ρ _s of the sampling point to 0, and generate the diffuse reflection feature vector in this Lumitexel space

b. Set the diffuse reflectance ρ _d to 0, and generate the specular reflection feature vector in this Lumitexel space

长度相同，m_s与

长度相同，向量m_d，m_s分别为漫反射特征向量

镜面反射向量

的预测；c. The output of the neural network is a vector m _d , m _s , where m _d is the same as

the same length, m _s is the same as

The lengths are the same, and the vectors m _d and m _s are the diffuse reflection feature vectors respectively.

Specular Vector

Prediction;

(4) The loss function of the material feature part is expressed as follows:

Among them, λ _d and λ _s represent the loss weights of m _d and m _s , respectively, the confidence β is used to measure the loss of the specular Lumitexel, and the log acts on each dimension of the vector; the confidence β is determined as follows:

项表示第j次采样所有单光源渲染值的最大值的对数，

项表示第j次采样理论上可获得的单光源渲染值的最大值的对数，∈为比值调整因子。in,

The term represents the logarithm of the maximum value of all single-light rendering values for the jth sample,

The term represents the logarithm of the maximum value of the theoretically obtainable single light source rendering value for the jth sampling, and ∈ is the ratio adjustment factor.

Further, in the acquisition stage, geometric alignment is performed after the material acquisition is completed, and then material recovery is performed. The geometric alignment is specifically: using a scanner to scan the object to obtain a geometric model, aligning it with the three-dimensional reconstructed geometric model, and replacing the three-dimensional model. Reconstructed geometric model.

Further, for the effective texture coordinates, according to the collected photos and the pose information of the sampling points, the pixels in the photos are taken out in turn, the validity of the pixels is judged, and the corresponding vertex poses are combined to form a high-dimensional point cloud; A point p on the surface of the sampled object determined by valid texture coordinates, and the judgment criterion that the jth sampling is valid for vertex p is expressed as follows:

在该采样下对于摄像机是可见的，且

位于训练网络时定义的采样空间内；(1) Position of vertex p

is visible to the camera at this sample, and

is located in the sampling space defined when training the network;

(·)为点积操作，θ为有效采样角度的下界，ω′_o表示世界坐标系下出射光方向，

表示第j次顶点p的法向量；(2)

( ) is the dot product operation, θ is the lower bound of the effective sampling angle, ω′ _o represents the outgoing light direction in the world coordinate system,

Represents the normal vector of the jth vertex p;

(3) The value of each channel of the pixel on the photo is in the interval [a, b], a, b are the lower and upper bounds of the effective sampling brightness;

When all three conditions are satisfied, it is considered that the jth sampling is valid for vertex p, and the result of the jth sampling is added to the high-dimensional point cloud.

Further, after restoring the material information, the material parameters can be fitted, which is divided into two steps:

(1) Fitting the local coordinate system and roughness: For a point p on the surface of the sampled object determined by a valid texture coordinate, use the L-BFGS-B method to fit the material parameters according to the single-channel specular reflection vector output by the network. The local coordinate system and roughness in ;

(2) Fitting reflectance: use the trust region algorithm to solve the specular reflectance and diffuse reflectance, fix the local coordinate system and roughness obtained in the previous process, and synthesize the observation value under the viewing angle used for the acquisition, so that it is consistent with the acquisition. The resulting observations are as close as possible.

The beneficial effects of the present invention are as follows: the method of the present invention proposes to transform the learning of material information into a geometric learning problem on an unstructured point cloud, and a plurality of sampling results under different illumination and observation directions are formed into a high-dimensional point cloud. Each point in is a vector composed of the image measurement value and the pose information of the object when the image was taken. This method can effectively aggregate unstructured from high-dimensional point clouds that are disordered, irregular, unevenly distributed, and limited in accuracy. The information of the visualization view is restored to restore high-quality material properties.

Description of drawings

1 is a three-dimensional schematic diagram of a collection device in an embodiment of the present invention;

2 is a front view of a collection device in an embodiment of the present invention;

3 is a side view of a collection device in an embodiment of the present invention;

4 is a schematic diagram of the relationship between a collection device and a sampling space in an embodiment of the present invention;

5 is a flowchart of a collection method according to an embodiment of the present invention;

6 is a schematic diagram of a neural network structure according to an embodiment of the present invention;

FIG. 7 is a single-channel display of a lighting pattern obtained by an embodiment of the present invention, using grayscale values to represent luminous intensity;

FIG. 8 is a Lumitexel vector result recovered using the system of the embodiment of the present invention;

FIG. 9 shows the result of the material properties of the sampled object recovered by using the system of the embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be described in detail below with reference to the accompanying drawings.

The method for free collection of high-dimensional materials proposed by the present invention can be specifically implemented into the following steps:

First, the training stage:

1. Generate training data and obtain the parameters of the acquisition device, including the distance and angle from the light source to the origin of the sampling space, the characteristic curve of the light source, the distance and angle from the camera to the origin of the sampling space, and the internal and external parameters of the camera. Use these parameters to generate acquisition results that simulate actual cameras as training data. The rendering model used when generating training data is the GGX model, and the generation formula is as follows:

和

D_GGX为微表面分布项，F为菲涅尔项，G_GGX表示阴影系数函数。Among them, f _r (ω′ _i , ω′ _o ; P) is the four-dimensional reflection function about ω′ _i , ω′ _o , ω′ _i represents the incident light direction in the world coordinate system, ω′ _o represents the outgoing light in the world coordinate system The direction of the incident light, ω _i is the incident direction in the local coordinate system, ω _o is the outgoing direction in the local coordinate system, and ω _h is the halfway vector in the local coordinate system. P contains the parameter information of the sampling point, including the material parameters of the sampling point n, t, α _x , α _y , ρ _d , ρ _s , where n represents the normal vector in the world coordinate system, and t represents the local sample point in the world coordinate system The x-axis direction of the coordinate system, n and t are used to transform the incident and outgoing directions from the world coordinate system to the local coordinate system. α _x , α _y represent the roughness coefficient, ρ _d represents the diffuse reflectance, ρ _s represents the specular reflectivity, ρ _d and ρ _s are both a scalar in a single channel, and three scalars respectively in the case of color

and

D _GGX is the microsurface distribution term, F is the Fresnel term, and G _GGX is the shadow coefficient function.

λ表示波长，c₁表示RGB三通道中的一个，光强为{I_R，I_G，I_B}的光源的光谱分布曲线L(λ)可以表示为：When the light source used for acquisition is color, the spectral response relationship between the light source, the sampled object and the camera should be corrected first. The correction method is as follows: define the spectral distribution curve of the unknown color light source L as

λ represents the wavelength, c ₁ represents one of the three RGB channels, and the spectral distribution curve _L (λ) of a light source whose light intensity is {IR , _IG , _IB } can be expressed as:

的线性组合，c₂表示RGB三通道中的一个：Similarly, the reflection spectral distribution curve p(λ) of any sampling point p can be expressed as three unknown bases with coefficients p _R , p _G , and p _B respectively.

A linear combination of , c ₂ represents one of the three RGB channels:

的线性组合。所以对于在光强为{I_R，I_G，I_B}的光源照射下，摄像机对于反射系数为{p_R，p_G，p_B}的采样点在某一特定通道c₃的测量值如下式：Similarly, the spectral distribution curve of camera C can be expressed as

linear combination of . Therefore, under the illumination of a light source with a light intensity of { _IR , _{IG, IB}, the measurement value of the camera for a sampling point with reflection coefficients {p R , p G , p B } in a} specific channel c ₃ is as follows Mode:

Under the illumination condition {IR, _IG , _IB }={1, 0, 0} _{/{0, 1, 0}/{0, 0, 1}, for the known reflection coefficient {p R ,} p _G , p _B } color test card for shooting, establish a linear equation system according to the measurement values collected by the camera, and solve the color correction matrix δ(c ₁ , c ₂ , c ₃ ) with a size of 3×3×3 , which represents the spectral response relationship between the light source, the sampled object, and the camera.

2. Using the generated training data, train the neural network shown in Figure 4. The characteristics of the neural network are as follows:

(1) The observation value B on the photo of a sampling point p on the surface of the object, the relationship between the reflection function _fr and the light intensity of each light source can be described as:

Among them, I represents the luminous information of each light source 1, including: the spatial position x _l of the light source 1, the normal vector n _l of the light source 1, the luminous intensity I(l) of the light source 1, and P contains the parameter information of the sampling point p, including : the spatial position x _p of the sampling point, the material parameters n, t, α _x , α _y , ρ _d , ρ _s . Ψ(x _l , ) describes the light intensity distribution of light source l under different incident directions, V represents the binary function of the visibility of x _l to x _p , ( ) ⁺ is the dot product operation of two vectors, negative values will be truncated to 0. fr (ω′ _i ; ω′ _o , P _{) is the two-dimensional reflection function about ω′ i when} ω′ _o is fixed.

The input of the neural network is the Lumitexel under k disordered and irregular sampling, k is the number of samples, and the Lumitexel is a vector, denoted as m(l; P), where each value describes the sampling point pair from each The reflected light intensity of the incident light of the light source along a certain viewing direction;

In the above formula, B is the representation in a single channel. When the light source is a color light source, B is expanded to the following form:

的结果。B与光源发光强度成线性关系，可以用线性全连接层模拟。Among them, f _r (ω′ _i ; ω′ _o , P, c ₂ ) is _fr (ω′ _i ; ω′ _o , P) in the formula

the result of. B has a linear relationship with the luminous intensity of the light source, which can be simulated with a linear fully connected layer.

(2) The first layer of the neural network includes a linear fully connected layer, and the parameter matrix of the linear fully connected layer is trained by the following formula:

W _l = f _W (W _raw )

Among them, W _raw is the parameter to be trained; W _l is the illumination matrix, for a single-channel light source, the size is 1 × N, for a color light source, the size is 3 × N, N is the vector length of Lumitexel; f _W is a mapping, with In order to transform W _raw so that the generated illumination matrix can correspond to the possible luminous intensity of the light source, the Sigmoid function is selected for the mapping f _W in this example, and the initialization value of the illumination matrix W _l of the first layer network is limited to (0, 1 ), but f _W is not limited to the Sigmoid function.

Taking W _l as the luminous intensity of the light source, according to the relationship described in the above (1), k sampling observation values B(I, P ¹ ), B(I, P ² )...B(I, P ^k ) are obtained by calculation.

(3) Starting from the second layer as a feature extraction network, k samples independently perform feature extraction to obtain feature vectors. The formula is as follows:

分别为第j次采样时采样点的空间位置，采样点几何法向量和几何切向量，

可通过三维重建或扫描仪扫描后得到的几何模型获得，

为与

正交的任意单位向量，通过第j次采样摄像机的位姿可以将

转换得到

V_feature(j)为网络输出的第j次采样的特征向量。Among them, f is a one-dimensional convolution function, the size of the convolution kernel is 1 × 1,

are the spatial position of the sampling point in the jth sampling, the geometric normal vector and geometric tangent vector of the sampling point,

It can be obtained by 3D reconstruction or the geometric model obtained after scanning with a scanner,

for and

Orthogonal arbitrary unit vector, by sampling the pose of the camera at the jth time, the

convert to get

V _feature (j) is the feature vector of the jth sampling output by the network.

(4) The feature extraction network is followed by a maximum pooling layer. The maximum pooling operation formula is as follows:

V _feature = max(V _feature (1), V _feature (2), ..., V _feature (k))

Among them, the max pooling operation is performed on each dimension of V _feature (1), V _feature (2), ..., V _feature (k).

(5) After the maximum pooling layer is a nonlinear mapping network;

和

为V_feature。Among them, f _i+1 is the mapping function of the i+1 layer network, W _i+1 is the parameter matrix of the i+1 layer network, b _i+1 is the offset vector of the i+1 layer network, y _{i +1} is the output of the i+1th layer network, d and s represent the two branches of diffuse reflection and specular reflection, respectively.

and

is V _feature .

(6) The loss function of the neural network is as follows:

z轴方向为

(6.1) A virtual Lumitextel space is a cube whose center is at the spatial position x _p of the sampling point, and the x-axis direction of the cube center coordinate system is

The z-axis direction is

(6.2) A virtual camera, the observation direction is the positive direction of the z-axis of the cube;

(6.3) For the diffuse reflection Lumitexel, the cube resolution is 6×N _d ² , for the specular reflection Lumitexel, the cube resolution is 6×N _s ² , that is, N _d ² and N _s ² points are uniformly sampled on each surface as A virtual point light source whose light intensity is unit light intensity, in this embodiment, N _d =8, N _s =32;

a. Set the specular reflectivity ρ _s of the sampling point to 0, and generate the diffuse reflection feature vector in this Lumitexel space

b. Set the diffuse reflectance ρ _d to 0, and generate the specular reflection feature vector in this Lumitexel space

长度相同，m_s与

长度相同，向量m_d，m_s分别为漫反射特征向量

、镜面反射向量

的预测；c. The output of the neural network is a vector m _d , m _s , where m _d is the same as

the same length, m _s is the same as

The lengths are the same, and the vectors m _d and m _s are the diffuse reflection feature vectors respectively.

, the specular reflection vector

Prediction;

(6.4) The loss function of the material feature part is expressed as follows:

Among them, λ _d and λ _s represent the loss weight of m _d and m _s , respectively, the confidence β is used to measure the loss of the specular Lumitexel, and the log acts on each dimension of the vector;

The confidence level β is determined as follows:

项表示第j次采样所有单光源渲染值的最大值的对数，

项表示第j次采样理论上可获得的单光源渲染值的最大值的对数，∈为比值调整因子，本实施例中取∈＝50％。in,

The term represents the logarithm of the maximum value of all single-light rendering values for the jth sample,

The term represents the logarithm of the maximum value of the theoretically obtainable single light source rendering value of the jth sampling, and ∈ is a ratio adjustment factor, which is taken as ∈=50% in this embodiment.

3. After the training, the parameter W _raw of the linear fully connected layer of the network is taken out and transformed by the formula W _l =f _W (W _raw ) as a lighting pattern.

2. Collection stage: The collection stage can be further subdivided into a material collection stage, a geometric alignment stage (optional) and a material recovery stage.

1. Material collection stage

The acquisition device sequentially illuminates the target 3D object according to the illumination pattern, the camera obtains a set of photos in unstructured views, and the photos are used as input. Using the 3D reconstruction tools publicly available in the industry, the geometric model of the sampled object and the camera when the photo was taken can be obtained. 's pose.

2. Geometric alignment stage (optional)

(1) Use a high-precision scanner to scan the object to obtain a geometric model;

(2) Align the geometric model scanned by the scanner with the three-dimensional reconstructed geometric model and replace the three-dimensional reconstructed geometric model. The alignment method can use the public method CPD in the field (A.Myronenko and X.Song.2010. PointSet Registration: Coherent Point Drift. IEEE PAMI 32, 12 (2010), 2262-2275. https://doi.org/10.1109/TPAMI.2010.46).

3. Material recovery stage:

(1) Obtain the pose of each vertex on the sampled object when the jth photo is taken according to the pose of the camera when the photo is taken according to the material collection step

(2) For the geometric model of the sampled object obtained by the three-dimensional reconstruction or the geometric model of the sampled object obtained by scanning the aligned scanner, use the public tool Iso-charts in the field to obtain a geometric model with texture coordinates;

组成高维点云，作为神经网络的第二层特征提取网络的输入向量，计算得到最后一层的输出向量m_d和m_s。(3) For the effective texture coordinates, according to the above-mentioned set of collected photos r ₁ , r ₂ , ..., r _π and the pose information of the sampling points, take out the pixels in the photos in turn, judge the validity of the pixels, and combine the corresponding vertex pose

A high-dimensional point cloud is formed, which is used as the input vector of the feature extraction network of the second layer of the neural network, and the output vectors m _d and m _s of the last layer are calculated.

For a point p on the surface of the sampled object determined by a valid texture coordinate, the judgment criterion that the jth sampling is valid for the point p is expressed as follows:

在该采样下对于摄像机是可见的，且

位于训练网络时定义的采样空间内；1)

is visible to the camera at this sample, and

is located in the sampling space defined when training the network;

(·)为点积操作，θ为有效采样角度的下界，本实施例中取θ＝0.3；2)

( ) is the dot product operation, θ is the lower bound of the effective sampling angle, and θ=0.3 in this embodiment;

3) The value of each channel of the pixel on the photo is in the interval [a, b], a, b are the lower and upper bounds of the effective sampling brightness, and in this embodiment, a=32, b=224;

When all three conditions are met, the jth sampling is considered valid for point p, and the result of the jth sampling is added to the high-dimensional point cloud;

(4) Fitting material parameters is divided into two steps:

1) Fitting the local coordinate system and roughness

For a point p on the surface of the sampled object determined by a valid texture coordinate, according to the single-channel specular reflection vector output by the network, the L-BFGS-B method is used to fit the local coordinate system and roughness in the material parameters. The optimization goal is :

where i is the serial number of the virtual light source in (6.3), m _s (l) represents the value of the l-th dimension of the specular reflection eigenvector predicted by the network, and ω′ _i represents the incident angle formed by the virtual light source with serial number l and the sampling point , ω′ _o represents the outgoing angle from the sampling point to the virtual camera in (6.2), P contains the normal vector n′, tangent vector t′ and other material parameters p′ of the sampling point for rendering, because the selected model Different and different. As the GGX model used in this project, p' includes anisotropic roughness, specular reflectance, and diffuse reflectance. In the above optimization objectives, n', t', and p' are parameters that can be optimized.

2) Fitting reflectivity

The process uses the trust region algorithm to solve the specular reflectance and diffuse reflectance, and the goal of fitting is:

表示该像素在第j个用于拟合的视角下，被上述优化得到的光照图案照射后，摄像机的观测值。B_j表示该像素在第j个用于拟合的视角下，使用上一过程得到的n′和t′及粗糙度合成出的观测值。对于彩色光源则合成参数还包括校正得到的颜色校正矩阵δ(c₁，c₂，c₃)。B_j计算过程如下：in,

It represents the observed value of the camera after the pixel is illuminated by the illumination pattern obtained by the above optimization at the jth viewing angle used for fitting. B _j represents the observation value synthesized by using the n' and t' and roughness obtained in the previous process under the jth viewing angle for fitting. For the color light source, the synthesis parameters also include the color correction matrix δ(c ₁ , c ₂ , c ₃ ) obtained by correction. The calculation process of _Bj is as follows:

再将漫反射率设为0，镜面反射率设为1，使用上一步得到的用于渲染的坐标系和粗糙度，渲染出第j个视角下的镜面反射Lumitexel，

将两个Lumitexel连接起来，形成矩阵

大小为N×2，其中N为采样设备的光源数。将采样时所用大小为3×N的光照图案矩阵W_l与M_j相乘，得到W_lM_j，大小为3×2。再与大小为2×3的可优化变量ρ_d，s相乘，得到First, set the diffuse reflectance to 1 and the specular reflectance to 0, and use the coordinate system and roughness obtained in the previous step for rendering to render the diffuse reflection Lumitexel at the jth viewing angle.

Then set the diffuse reflectance to 0 and the specular reflectance to 1, and use the coordinate system and roughness obtained in the previous step for rendering to render the specular Lumitexel at the jth viewing angle,

Connect two Lumitexels to form a matrix

The size is N×2, where N is the number of light sources of the sampling device. Multiply the light pattern matrix W _l and M _j with a size of 3×N used for sampling to obtain W _l M _j with a size of 3×2. Multiply by the optimizable variable ρ _{d, s} of size 2 × 3 to get

T=W _l M _j ρ _{d, s}

在后两维上求和，最终得到三维向量B_j。对于彩色光源，

要与校准设备所得的颜色校正矩阵δ(c₁，c₂，c₃)逐元素相乘，再在后两维上求和。The size of T is 3 × 3, and its copies are concatenated to form a tensor

Summing over the last two dimensions results in a three-dimensional vector B _j . For colored light sources,

It is multiplied element-wise by the color correction matrix δ(c ₁ , c ₂ , c ₃ ) obtained by the calibration device, and then summed in the last two dimensions.

A specific example of the acquisition device system is given below. Figure 1 is a three-dimensional display of the system example, Figure 2 is a front view, and Figure 3 is a side view. Acquire images. There are 512 LED lamp beads densely arranged on the light board. The lamp beads are controlled by FPGA, which can adjust the light-emitting brightness and light-emitting time.

An example of an acquisition system applying the method of the present invention is given below, and the system is generally divided into the following modules:

Preparation module: Provide a data set for network training. This part uses the GGX model to input a set of material parameters, the pose information of k sampling points, and the camera position, and a high-dimensional point cloud composed of k reflections can be obtained. The network training part uses the Pytorch open source framework and uses the Adam optimizer for training. The network structure is shown in Figure 6. Each rectangle represents a layer of neurons, and the number in the rectangle represents the number of neurons in the layer. The leftmost layer is the input layer, and the rightmost layer is the output layer. The solid arrows between layers represent full connections, and the dashed arrows represent convolutions.

Acquisition module: The equipment is shown in Figures 1, 2, and 3. The specific structure has been described above. The size of the sampling space defined by this system and the spatial relationship between the acquisition equipment and the sampling space are shown in Figure 4.

Recovery module: Use the geometric model of the sampled object obtained by 3D reconstruction or the geometric model of the sampled object scanned by the aligned scanner to obtain the geometric model with texture coordinates, load the trained neural network, and analyze the texture coordinates. For each vertex on the geometry model, predict the material feature vector, fit the coordinate system and material parameters used for rendering.

FIG. 5 is a workflow of this embodiment. First, generate training data, randomly sample 200 million Lumitexels, take 80% as the training set, and the rest as the validation set. The Xavier method is used to initialize the parameters when training the network, and the learning rate is 1e-4. The illumination pattern is color, the size of the illumination matrix is (3,512), and the three rows of the matrix represent the illumination patterns of the red, green and blue channels. After the training, the illumination matrix is taken out and transformed into an illumination pattern. The parameters of each column specify the luminous intensity of the light source at that position. Figure 7 shows the red, green, and blue three-channel illumination pattern obtained by a network training. The next process is: 1. Handheld device, the light board emits light according to the light pattern, and the camera shoots the object at the same time to obtain a set of sampling results. 2. For the geometric model of the sampled object obtained by 3D reconstruction or the geometric model of the sampled object obtained by scanning with the aligned scanner, use Isochart to obtain the geometric model with texture coordinates. 3. For each vertex on the geometric model with texture coordinates, find the corresponding valid real shot data according to the pose during sampling and the pixel value of the sampled photo, form a high-dimensional point cloud input network, and restore the diffuse reflection feature vector and specular eigenvectors. 4. According to the diffuse and specular eigenvectors output by the network, use the LBFGS-B method for each vertex to fit the coordinate system and roughness used for rendering, and use the trust region algorithm to solve the specular reflectance and diffuse reflectance.

右侧一列为对应的m_s。Figure 8 shows two Lumitexel vectors in the validation set recovered using the above system, with the column on the left

The column on the right is the corresponding m _s .

三个分量，第二行分别表示采样物体

三个分量，第三行分别表示采样物体粗糙度系数α_x，α_y，灰度值代表数值大小。Figure 9 shows the results of the material properties recovered from the material appearance scanning of the sampled objects using the above system. The first row represents the sampled objects respectively.

Three components, the second line represents the sampled object respectively

Three components, the third row represents the roughness coefficients α _x and α _y of the sampled object respectively, and the gray value represents the numerical value.

The above descriptions are only examples of preferred embodiments, and the present invention is not limited to the above-mentioned embodiments. As long as the technical effect of the present invention is achieved by the same means, it should belong to the protection scope of the present invention. Within the protection scope of the present invention, various modifications and changes may be made to its technical solutions and/or implementations.

Claims (10)

1. a free-style acquisition method of high-dimensional material, is characterized in that, this method comprises training phase and acquisition phase; The training phase includes the following steps: (1) Obtain the parameters of the acquisition device, and generate the acquisition results that simulate the actual camera as training data; (2) Use the generated training data to train the neural network. The characteristics of the neural network are as follows: (2.1) The input of the neural network is the Lumitexel vector under k unstructured samples, where k is the number of samples, and each value of the Lumitexel describes the reflection of the incident light from each light source along a certain viewing direction by the sampling point Light intensity, Lumitexel has a linear relationship with the luminous intensity of the light source, which is simulated by a linear fully connected layer; (2.2) The first layer of the neural network includes a linear fully connected layer, which is used to simulate the illumination pattern used in actual acquisition, and transform the k Lumitexels into camera acquisition results, which are respectively related to the corresponding sampling points. The pose information is combined to form a high-dimensional point cloud; (2.3) Starting from the second layer as a feature extraction network, independently perform feature extraction from each point in the high-dimensional point cloud to obtain a feature vector; (2.4) The feature extraction network is followed by a maximum pooling layer, which is used to aggregate the feature vectors extracted from k unstructured views to obtain a global feature vector; (2.5) After the maximum pooling layer, a nonlinear mapping network is used to restore high-dimensional material information according to the global feature vector; The collection stage includes the following steps: (1) Material collection: The collection device sequentially illuminates the target 3D object according to the illumination pattern, the camera obtains a set of photos under unstructured views, and uses the photos as input to obtain the geometric model of the sampled object with texture coordinates and shoot The pose of the camera when the photo is taken; (2) Material recovery: According to the pose of the camera when the photo was taken in the material collection stage, the pose of the vertex corresponding to each valid texture coordinate on the sampled object when each photo was taken is obtained; according to the collected photos and pose information, The high-dimensional point cloud is formed as the input of the second layer feature extraction network of the neural network, and the high-dimensional material information is obtained by calculation. 2 . The free-style acquisition method for high-dimensional materials according to claim 1 , wherein the unstructured sampling is a free random sampling with a non-fixed viewing angle, and the sampling data is disordered, irregular, and unevenly distributed. 3 . , a fixed object can be used, which can be collected by a person holding a collection device, or the object can be rotated on a turntable and collected by a fixed device. 3. The free-form acquisition method of a high-dimensional material according to claim 1, wherein in the training data generation process, when the light source is colored, it is necessary to respond to the spectral response between the light source, the sampling object and the camera The relationship is corrected, and the correction method is as follows: The spectral distribution curve of the unknown color light source L is defined as

λ represents the wavelength, c ₁ represents one of the three RGB channels, and the spectral distribution curve _L (λ) of a light source whose light intensity is {IR , _IG , _IB } can be expressed as:

The reflection spectral distribution curve p(λ) of any sampling point p is expressed as three unknown bases with coefficients p _R , p _G , and p _B respectively

A linear combination of , c ₂ represents one of the three RGB channels:

The spectral distribution curve of camera C is expressed as

_{linear combination of _ _ _ _} The measured value is as follows:

Under the illumination condition {IR, _IG , _IB }={1, 0, 0} _{/{0, 1, 0}/{0, 0, 1}, for the known reflection coefficient {p R ,} P _G , p _B } color test card for shooting, establish a linear equation system according to the measurement values collected by the camera, and solve the color correction matrix δ(c ₁ , c ₂ , c ₃ ) with a size of 3×3×3, Represents the spectral response relationship between the light source, the sampled object, and the camera. 4. The free-form acquisition method of a high-dimensional material according to claim 3, wherein in the training stage step (2.1), the observation value B of a sampling point p on the photo of the object surface, the reflection function _fr The relationship with the light intensity of each light source can be described as:

Among them, I represents the luminous information of each light source 1, including: the spatial position x _l of the light source 1, the normal vector n _l of the light source 1, the luminous intensity I(l) of the light source 1, and P contains the parameter information of the sampling point p, including : the spatial position x _p of the sampling point, the material parameters n, t, α _x , α _y , ρ _d , ρ _s ; Ψ(x _l , ·) describes the light intensity distribution of the light source l in different incident directions, V represents The binary function of x _l for the visibility of x _p , ( ) ⁺ is the dot product operation of two vectors; ω′ _i represents the incident light direction in the world coordinate system, ω′ _o represents the outgoing light direction in the world coordinate system, f _r (ω′ _i ; ω′ _o , P) is the two-dimensional reflection function about ω′ _i when ω′ _o is fixed; n represents the normal vector in the world coordinate system, and t represents the local coordinate system of the sampling point in the world coordinate system. In the x-axis direction, n and t are used to convert the incident and outgoing directions from the world coordinate system to the local coordinate system; α _x , α _y represent the roughness coefficient, ρ _d represents the diffuse reflectance, and ρ _s represents the specular reflectance; The input of the neural network is the Lumitexel vector under k unstructured sampling, which is denoted as m(l;P);

In the above formula, B is the representation in a single channel. When the light source is a color light source, B is expanded to the following form:

(ω′ _i · n _p ) ⁺ (-ω _i ′ · n _l ) ⁺ δ(c ₁ , c ₂ , c ₃ )dx _l Among them, fr (ω′ _i ; ω′ _o , P, c ₂ ) is in _{fr (ω i ′} ; ω′ _o , P)

the result of. 5. The free-form collection method of a high-dimensional material according to claim 1, wherein, in the training stage step (2.3), the formula of the feature extraction network is as follows:

Among them, f is a one-dimensional convolution function, the size of the convolution kernel is 1×1, B(I, P ^j ) represents the result of the first layer network output or the collected measurement value,

are the spatial position of the sampling point in the jth sampling, the geometric normal vector and geometric tangent vector of the sampling point,

obtained from the geometric model,

for and

Orthogonal arbitrary unit vector, by sampling the pose of the camera at the jth time, the

convert to get

V _feature (j) is the feature vector of the jth sampling output by the network. 6. The free-form acquisition method of a high-dimensional material according to claim 1, characterized in that, in the training phase step (2.5), the nonlinear mapping network is formally expressed as follows:

Among them, f _i+1 is the mapping function of the i+1 layer network, W _i+1 is the parameter matrix of the i+1 layer network, b _i+1 is the offset vector of the i+1 layer network, y _{i +1} is the output of the i+1th layer network, d and s represent the two branches of diffuse reflection and specular reflection, respectively.

and

The global feature vector output for the max pooling layer. 7. The free-form acquisition method of a high-dimensional material according to claim 1, wherein the loss function of the neural network is designed as follows: (1) A virtual Lumitextel space is a cube whose center is at the spatial position x _p of the sampling point, and the direction of the x-axis of the cube center coordinate system is

The z-axis direction is

is the geometric normal vector,

for and

Orthogonal arbitrary unit vectors; (2) A virtual camera, the observation direction is the positive direction of the z-axis of the cube; (3) For the diffuse reflection Lumitexel, the cube resolution is 6×N _d ² , for the specular reflection Lumitexel, the cube resolution is 6×N _s ² , that is, N _d ² and N _s ² points are uniformly sampled on each surface as A virtual point light source whose light intensity is unit light intensity; a. Set the specular reflectivity ρ _s of the sampling point to 0, and generate the diffuse reflection feature vector in this Lumitexel space

b. Set the diffuse reflectance ρ _d to 0, and generate the specular reflection feature vector in this Lumitexel space

c. The output of the neural network is a vector m _d , m _s , where m _d is the same as

the same length, m _s is the same as

The lengths are the same, and the vectors m _d and m _s are the diffuse reflection feature vectors respectively.

Specular Vector

Prediction; (4) The loss function of the material feature part is expressed as follows:

Among them, λ _d and λ _s represent the loss weight of m _d and m _s , respectively, the confidence β is used to measure the loss of the specular Lumitexel, and the log acts on each dimension of the vector; The confidence level β is determined as follows:

in,

The term represents the logarithm of the maximum value of all single-light rendering values for the jth sample,

The term represents the logarithm of the maximum value of the theoretically obtainable single light source rendering value for the jth sampling, and ∈ is the ratio adjustment factor. 8. The free-style collection method of high-dimensional materials according to claim 1, wherein in the collection stage, geometric alignment is performed after material collection is completed, and then material recovery is performed, and the geometric alignment is specifically: Use a scanner to scan an object to obtain a geometric model, align it with the three-dimensional reconstructed geometric model, and replace the three-dimensional reconstructed geometric model. 9 . The free-form acquisition method of high-dimensional materials according to claim 1 , wherein, for effective texture coordinates, pixels in the photo are sequentially taken out according to the collected photo and the pose information of the sampling point. 10 . , judge the validity of the pixel, and combine the corresponding vertex pose to form a high-dimensional point cloud; for a point p on the surface of the sampled object determined by a valid texture coordinate, the jth sampling is valid for the vertex p. The judgment criteria are expressed as follows: (1) Position of vertex p

is visible to the camera at this sample, and

is located in the sampling space defined when training the network; (2)

( ) is the dot product operation, θ is the lower bound of the effective sampling angle, ω′ _o represents the outgoing light direction in the world coordinate system,

Represents the normal vector of the jth vertex p; (3) The value of each channel of the pixel on the photo is in the interval [a, b], a, b are the lower and upper bounds of the effective sampling brightness; When all three conditions are satisfied, it is considered that the jth sampling is valid for vertex p, and the result of the jth sampling is added to the high-dimensional point cloud. 10. The free-form collection method of high-dimensional materials according to claim 1, wherein after recovering the material information, the material parameters can be fitted, which is divided into two steps: (1) Fitting the local coordinate system and roughness: For a point p on the surface of the sampled object determined by a valid texture coordinate, use the L-BFGS-B method to fit the material parameters according to the single-channel specular reflection vector output by the network. The local coordinate system and roughness in ; (2) Fitting reflectance: use the trust region algorithm to solve the specular reflectance and diffuse reflectance, fix the local coordinate system and roughness obtained in the previous process, and synthesize the observation value under the viewing angle used for the acquisition, so that it is consistent with the acquisition. The resulting observations are as close as possible.

Images