WO2022256962A1 - 一种高维材质的自由式采集方法 - Google Patents

一种高维材质的自由式采集方法 Download PDF

Info

Publication number
WO2022256962A1
WO2022256962A1 PCT/CN2021/098576 CN2021098576W WO2022256962A1 WO 2022256962 A1 WO2022256962 A1 WO 2022256962A1 CN 2021098576 W CN2021098576 W CN 2021098576W WO 2022256962 A1 WO2022256962 A1 WO 2022256962A1
Authority
WO
WIPO (PCT)
Prior art keywords
sampling
light source
dimensional
vector
collection
Prior art date
Application number
PCT/CN2021/098576
Other languages
English (en)
French (fr)
Inventor
吴鸿智
周昆
马晓鹤
Original Assignee
浙江大学
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 浙江大学 filed Critical 浙江大学
Priority to JP2023558118A priority Critical patent/JP2024511083A/ja
Priority to PCT/CN2021/098576 priority patent/WO2022256962A1/zh
Publication of WO2022256962A1 publication Critical patent/WO2022256962A1/zh
Priority to US18/493,831 priority patent/US20240062460A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/04—Texture mapping
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/50—Lighting effects
    • G06T15/506—Illumination models
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G06T7/75—Determining position or orientation of objects or cameras using feature-based methods involving models
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/90—Determination of colour characteristics
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30244—Camera pose

Definitions

  • the invention relates to a free-form acquisition method of high-dimensional materials, belonging to the fields of computer graphics and computer vision.
  • Digitization of real-world objects is one of the central problems in computer graphics and vision.
  • digital real objects can be represented by a three-dimensional grid model and a six-dimensional space-varying bidirectional reflectance distribution function (SVBRDF).
  • SVBRDF space-varying bidirectional reflectance distribution function
  • the purpose of the present invention is to address the deficiencies in the prior art and provide a method for free-style collection of high-dimensional materials. This method can effectively utilize the collection condition information of each view, and restore high-level data from disordered and unevenly distributed collection results.
  • the object material property for mass is to address the deficiencies in the prior art and provide a method for free-style collection of high-dimensional materials. This method can effectively utilize the collection condition information of each view, and restore high-level data from disordered and unevenly distributed collection results.
  • the object material property for mass is to address the deficiencies in the prior art and provide a method for free-style collection of high-dimensional materials. This method can effectively utilize the collection condition information of each view, and restore high-level data from disordered and unevenly distributed collection results.
  • the object material property for mass is to address the deficiencies in the prior art and provide a method for free-style collection of high-dimensional materials.
  • the invention discloses a free-style collection method of high-dimensional materials.
  • the main idea of the method is that free-style appearance scanning can be converted into a geometric learning problem on an unstructured point cloud, and each point in the point cloud can represent An image measurement and the pose information of the object at the time the image was taken.
  • the present invention designs a neural network that can efficiently aggregate information in different unstructured views, reconstruct spatially independent reflection properties, and optimize the lighting patterns used in the acquisition stage, resulting in high-quality materials Collect results.
  • the present invention does not depend on a specific collection device, and can use a fixed object to be collected by a hand-held device, or place the object on a turntable to rotate and collect it with a fixed device, and is not limited to these two methods.
  • This method proposes to transform the learning of material information into a geometric learning problem on an unstructured point cloud, and to form a high-dimensional point cloud from multiple sampling results under different illumination and observation directions.
  • Each point in the point cloud is an image measurement value and the pose information of the object when the image is taken, this method can effectively aggregate the information of the unstructured view from the disordered, irregular, unevenly distributed and precision-limited high-dimensional point cloud, and restore the high-level The material property for mass.
  • the formal representation is as follows:
  • the point cloud data feature extraction method G is not limited to a specific network structure, and other methods that can extract features from point clouds are also applicable;
  • the nonlinear mapping network F is not limited to the fully connected network;
  • the expression of object material properties is not limited to Lumitexel vector m.
  • the method consists of two phases: a training phase and an acquisition phase.
  • the training phase includes the following steps:
  • the input of the neural network is the Lumitexel vector under k unstructured samples, k is the number of samples, and each value of Lumitexel describes the reflection of the sampling point on the incident light from each light source along a certain viewing direction Light intensity, Lumitexel has a linear relationship with the luminous intensity of the light source, which is simulated by a linear fully connected layer;
  • the first layer of the neural network includes a linear fully connected layer, which is used to simulate the illumination pattern used in the actual collection, and transforms the k Lumitexels into camera collection results, and these k collection results are respectively compared with the corresponding sampling points
  • the pose information is combined to form a high-dimensional point cloud
  • each point in the high-dimensional point cloud is independently extracted to obtain a feature vector
  • the feature extraction network is the maximum pooling layer, which is used to aggregate the feature vectors extracted from k unstructured views to obtain the global feature vector;
  • the maximum pooling layer is a nonlinear mapping network, which is used to restore high-dimensional material information according to the global feature vector;
  • the acquisition phase includes the following steps:
  • Material acquisition the acquisition device sequentially irradiates the target three-dimensional object according to the illumination pattern, and the camera obtains a set of photos in an unstructured view, and takes the photos as input to obtain the geometric model of the sampled object with texture coordinates and shoot The pose of the camera at the time of the photo;
  • the unstructured sampling is free random sampling with a non-fixed viewing angle, and the sampling data is disordered, irregular, and unevenly distributed.
  • a fixed object can be used for collection by a hand-held collection device, or the object can be placed on a turntable Rotating, stationary equipment collection.
  • ⁇ represents the wavelength
  • c 1 represents one of the three channels of RGB
  • L( ⁇ ) of the light source with light intensity ⁇ I R , I G , I B ⁇ can be expressed as:
  • the spectral distribution curve of camera C is expressed as A linear combination of ; under the illumination of a light source with light intensity ⁇ I R , I G , I B ⁇ , the camera’s reflection coefficient is ⁇ p R , p G , p B ⁇ at a specific channel c 3
  • the measured value is as follows:
  • step (2.1) of the training phase the observed value B of a sampling point p on the object surface on the photo, the relationship between the reflection function f r and the light intensity of each light source can be described as:
  • I represents the luminous information of each light source l, including: the spatial position x l of light source l, the normal vector n l of light source l, and the luminous intensity I(l) of light source l
  • P contains the parameter information of sampling point p, including : Spatial position x p of sampling point, material parameters n, t, ⁇ x , ⁇ y , ⁇ d , ⁇ s .
  • ⁇ (x l , ⁇ ) describes the light intensity distribution of the light source l under different incident directions
  • V represents the binary function of the visibility of x l to x p
  • ( ⁇ ) + is the dot product operation of two vectors.
  • f r ( ⁇ ′ i ; ⁇ ′ o ,P) is the two-dimensional reflection function about ⁇ ′ i when ⁇ ′ o is fixed.
  • the input of the neural network is the Lumitexel vector under k unstructured samples, denoted as m(l; P);
  • B is the representation under a single channel.
  • f r ( ⁇ ′ i ; ⁇ ′ o ,P,c 2 ) is f r ( ⁇ ′ i ; ⁇ ′ o ,P) the result of.
  • step (2.3) of the training phase the formula of the feature extraction network is as follows:
  • f is a one-dimensional convolution function
  • the convolution kernel size is 1 ⁇ 1
  • B(I,P j ) represents the output result of the first layer network or the measured value collected
  • the spatial position of the sampling point, the geometric normal vector and the geometric tangent vector of the sampling point at the jth sampling time, obtained from the geometric model, for with Orthogonal arbitrary unit vector the pose of the jth sampling camera can be converted to V feature (j) is the feature vector of the jth sample output by the network.
  • step (2.5) of the training phase the nonlinear mapping network is formalized as follows:
  • f i+1 is the mapping function of the i+1 layer network
  • W i+1 is the parameter matrix of the i+1 layer network
  • b i+1 is the offset vector of the i+1 layer network
  • y i +1 is the output of the i+1 layer network
  • d and s respectively represent the two branches of diffuse reflection and specular reflection, input and The global feature vector output by the max pooling layer.
  • the loss function of the neural network is designed as follows:
  • a virtual Lumitextel space is a cube whose center is at the spatial position x p of the sampling point, and the x-axis direction of the cube center coordinate system is The z-axis direction is is the geometric normal vector, for with Orthogonal arbitrary unit vectors;
  • a camera is virtualized, and the viewing direction is the positive direction of the z-axis of the cube;
  • the cube resolution is 6 ⁇ N d 2
  • the cube resolution is 6 ⁇ N s 2 , that is, N d 2 and N s 2 points are evenly sampled on each surface as
  • the light intensity is a virtual point light source with unit light intensity
  • the output of the neural network is a vector m d , m s , where m d and same length, m s and The lengths are the same, the vectors m d , m s are the diffuse reflection feature vectors Specular Vector Prediction;
  • ⁇ d and ⁇ s represent the loss weights of m d and m s respectively
  • the confidence degree ⁇ is used to measure the loss of specular reflection Lumitexel, and log acts on each dimension of the vector; the confidence degree ⁇ is determined as follows:
  • the item represents the logarithm of the maximum value of all single light source rendering values in the jth sample
  • the term represents the logarithm of the maximum value of the single light source rendering value theoretically obtainable at the jth sampling
  • ⁇ is the ratio adjustment factor
  • geometric alignment is performed after material acquisition, and then material recovery is performed.
  • the geometric alignment specifically includes: using a scanner to scan an object to obtain a geometric model, aligning it with the 3D reconstructed geometric model, and then replacing the 3D Reconstructed geometry model.
  • the pixels in the photos are sequentially taken out, the validity of the pixels is judged, and the corresponding vertex poses are combined to form a high-dimensional point cloud;
  • a point p on the surface of the sampled object determined by valid texture coordinates, the criteria for judging that the jth sampling is valid for the vertex p are expressed as follows:
  • Vertex p position is visible to the camera at this sample, and Located in the sampling space defined when training the network;
  • the j-th sampling is valid for the vertex p, and the result of the j-th sampling is added to the high-dimensional point cloud.
  • the material parameters can be fitted, which is divided into two steps:
  • the method of the present invention proposes to transform the learning of material information into a geometric learning problem on an unstructured point cloud, and to form a high-dimensional point cloud from a plurality of sampling results under different illumination and observation directions.
  • Each point in is a vector composed of the image measurement value and the pose information of the object when the image was taken.
  • This method can effectively aggregate unstructured point clouds from disordered, irregular, unevenly distributed and limited precision high-dimensional point clouds.
  • the information of the simplified view can be restored to high-quality material properties.
  • Fig. 1 is a three-dimensional schematic diagram of a collection device in an embodiment of the present invention
  • Fig. 2 is a front view of a collection device in an embodiment of the present invention.
  • Fig. 3 is a side view of a collection device in an embodiment of the present invention.
  • Fig. 4 is a schematic diagram of the relationship between a collection device and sampling space in an embodiment of the present invention.
  • Fig. 5 is the flow chart of the collection method of the embodiment of the present invention.
  • FIG. 6 is a schematic diagram of a neural network structure according to an embodiment of the present invention.
  • Fig. 7 is a single-channel display of the illumination pattern obtained in the embodiment of the present invention, using the gray value to represent the luminous intensity;
  • Fig. 8 is the Lumitexel vector result restored by the system using the embodiment of the present invention.
  • FIG. 9 shows the results of the material properties of the sampled objects recovered by using the system according to the embodiment of the present invention.
  • f r ( ⁇ ′ i , ⁇ ′ o ; P) is the four-dimensional reflection function about ⁇ ′ i , ⁇ ′ o , ⁇ ′ i represents the direction of incident light in the world coordinate system, ⁇ ′ o represents the outgoing light in the world coordinate system ⁇ i is the incident direction in the local coordinate system, ⁇ o is the outgoing direction in the local coordinate system, and ⁇ h is the halfway vector in the local coordinate system.
  • P contains the parameter information of the sampling point, including the material parameters n, t, ⁇ x , ⁇ y , ⁇ d , ⁇ s of the sampling point, where n represents the normal vector in the world coordinate system, and t represents the local sampling point in the world coordinate system
  • n represents the normal vector in the world coordinate system
  • t represents the local sampling point in the world coordinate system
  • the x-axis direction of the coordinate system, n and t are used to convert the incident direction and outgoing direction from the world coordinate system to the local coordinate system.
  • ⁇ x , ⁇ y represent the roughness coefficient
  • ⁇ d represents the diffuse reflectance
  • ⁇ s represents the specular reflectance
  • ⁇ d and ⁇ s are both a scalar in the single channel
  • three scalar in the case of color and D GGX is the microsurface distribution item
  • F is the Fresnel item
  • G GGX is the shadow coefficient function.
  • the spectral response relationship between the light source, sampling object and camera must be corrected first.
  • the correction method is as follows: define the spectral distribution curve of the unknown color light source L as ⁇ represents the wavelength, c 1 represents one of the three channels of RGB, and the spectral distribution curve L( ⁇ ) of the light source with light intensity ⁇ I R , I G , I B ⁇ can be expressed as:
  • the reflection spectral distribution curve p( ⁇ ) of any sampling point p can be expressed as three unknown bases with coefficients p R , p G , p B
  • a linear combination of , c 2 represents one of the three RGB channels:
  • the spectral distribution curve of camera C can be expressed as linear combination of . Therefore, under the illumination of a light source with light intensity ⁇ I R , I G , I B ⁇ , the measured value of the camera for a sampling point with reflection coefficient ⁇ p R , p G , p B ⁇ in a specific channel c 3 is as follows Mode:
  • I represents the luminous information of each light source l, including: the spatial position x l of light source l, the normal vector n l of light source l, and the luminous intensity I(l) of light source l
  • P contains the parameter information of sampling point p, including : Spatial position x p of sampling point, material parameters n, t, ⁇ x , ⁇ y , ⁇ d , ⁇ s .
  • ⁇ (x l , ⁇ ) describes the light intensity distribution of the light source l under different incident directions
  • V represents the binary function of the visibility of x l to x p
  • ( ⁇ ) + is the dot product operation of two vectors, and the negative value will be truncated to 0.
  • f r ( ⁇ ′ i ; ⁇ ′ o ,P) is the two-dimensional reflection function about ⁇ ′ i when ⁇ ′ o is fixed.
  • the input of the neural network is k random and irregularly sampled Lumitexels, k is the number of samples, and Lumitexel is a vector, denoted as m(l; P), where each value describes the pair of sampling points from each The reflected light intensity of the incident light of the light source along a certain viewing direction;
  • B is the representation under a single channel.
  • f r ( ⁇ ′ i ; ⁇ ′ o ,P,c 2 ) is f r ( ⁇ ′ i ; ⁇ ′ o ,P) in the formula the result of.
  • B has a linear relationship with the luminous intensity of the light source, which can be simulated by a linear fully connected layer.
  • the first layer of the neural network includes a linear fully connected layer, and the parameter matrix of the linear fully connected layer is trained by the following formula:
  • W raw is the parameter to be trained
  • W l is the illumination matrix, for a single-channel light source, the size is 1 ⁇ N, for a color light source, the size is 3 ⁇ N, and N is the vector length of Lumitexel
  • f W is a mapping, using In order to transform W raw so that the generated illumination matrix can correspond to the possible luminous intensity of the light source, in this example, the mapping f W uses the Sigmoid function, and the initial value of the illumination matrix W l of the first-layer network is limited to (0, 1 ), but f W is not limited to the Sigmoid function.
  • f is a one-dimensional convolution function
  • the convolution kernel size is 1 ⁇ 1
  • the spatial position of the sampling point, the geometric normal vector and the geometric tangent vector of the sampling point at the jth sampling time It can be obtained through 3D reconstruction or the geometric model obtained after scanning with a scanner, for with Orthogonal arbitrary unit vector, the pose of the jth sampling camera can be converted to V feature (j) is the feature vector of the jth sample output by the network.
  • the maximum pooling operation formula is as follows:
  • V feature max(V feature (1),V feature (2),...,V feature (k))
  • V feature (1) the maximum pooling operation is performed on each dimension of V feature (1), V feature (2), ..., V feature (k).
  • f i+1 is the mapping function of the i+1 layer network
  • W i+1 is the parameter matrix of the i+1 layer network
  • b i+1 is the offset vector of the i+1 layer network
  • y i +1 is the output of the i+1 layer network
  • d and s respectively represent the two branches of diffuse reflection and specular reflection, input and is V feature .
  • a virtual Lumitextel space is a cube whose center is at the spatial position x p of the sampling point, and the x-axis direction of the cube center coordinate system is The z-axis direction is
  • the cube resolution is 6 ⁇ N d 2
  • the cube resolution is 6 ⁇ N s 2 , that is, N d 2 and N s 2 points are evenly sampled on each surface as
  • the light intensity is a virtual point light source with unit light intensity.
  • the output of the neural network is a vector m d , m s , where m d and same length, m s and The lengths are the same, the vectors m d , m s are the diffuse reflection feature vectors Specular Vector Prediction;
  • ⁇ d and ⁇ s represent the loss weights of m d and m s respectively
  • the confidence degree ⁇ is used to measure the loss of specular reflection Lumitexel, and log acts on each dimension of the vector;
  • the confidence degree ⁇ is determined as follows:
  • the item represents the logarithm of the maximum value of all single light source rendering values in the jth sample
  • the term represents the logarithm of the maximum value of the theoretically obtainable single light source rendering value of the jth sampling
  • ⁇ is the ratio adjustment factor
  • Collection phase The collection phase can be subdivided into material collection phase, geometric alignment phase (optional) and material restoration phase.
  • the acquisition device sequentially irradiates the target three-dimensional object according to the illumination pattern, and the camera obtains a set of photos in an unstructured view.
  • the photos are used as input, and the geometric model of the sampled object and the camera when taking photos can be obtained by using the 3D reconstruction tool released by the industry. pose.
  • the alignment method can use the public method CPD in the field (A.Myronenko and X.Song.2010.Point Set Registration: Coherent Point Drift. IEEE PAMI 32, 12(2010), 2262–2275.
  • n', t', p' are the parameters that can be optimized.
  • This process uses the trust region algorithm to solve the specular reflectance and diffuse reflectance, and the fitting target is:
  • B j represents the observation value synthesized by using the n' and t' obtained in the previous process and the roughness of the pixel at the jth viewing angle used for fitting.
  • the synthesis parameters also include the corrected color correction matrix ⁇ (c 1 , c 2 , c 3 ). The calculation process of B j is as follows:
  • T 3 ⁇ 3
  • B j 3 ⁇ 3
  • ⁇ (c 1 ,c 2 ,c 3 ) the color correction matrix
  • Figure 1 is a three-dimensional display of the system example
  • Figure 2 is a front view
  • Figure 3 is a side view. Capture images.
  • the lamp beads are controlled by FPGA, which can adjust the luminous brightness and luminous time.
  • Preparation module Provide data sets for network training.
  • This part uses the GGX model to input a set of material parameters, pose information of k sampling points, camera position, and obtain a high-dimensional point cloud composed of k reflections.
  • the network training part uses the Pytorch open source framework and uses the Adam optimizer for training.
  • the network structure is shown in Figure 6. Each rectangle represents a layer of neurons, and the number in the rectangle represents the number of neurons in this layer. The leftmost layer is the input layer, and the rightmost layer is the output layer.
  • the solid arrows between layers represent full connections, and the dotted arrows represent convolutions.
  • Restoration module use the geometric model of the sampled object obtained by 3D reconstruction or the geometric model of the sampled object scanned by the aligned scanner to calculate the geometric model with texture coordinates, load the trained neural network, and calculate the geometric model with texture coordinates For each vertex on the geometric model, predict the material feature vector, fit the coordinate system and material parameters for rendering.
  • Fig. 5 is the workflow of this embodiment.
  • First generate training data randomly sample 200 million Lumitexels, take 80% as the training set, and the rest as the verification set.
  • the Xavier method is used to initialize the parameters, and the learning rate is 1e-4.
  • the illumination pattern is colored, the size of the illumination matrix is (3,512), and the three rows of the matrix represent the illumination patterns of the red, green, and blue channels.
  • After the training take out the illumination matrix and transform it into an illumination pattern.
  • the parameters of each column specify the luminous intensity of the light source at that position.
  • Figure 7 shows a red, green, and blue three-channel illumination pattern obtained by network training.
  • the next process is: 1.
  • the handheld device the light panel emits light according to the light pattern, and the camera shoots the object at the same time to obtain a set of sampling results.
  • the geometric model of the sampling object obtained by 3D reconstruction or the geometric model of the sampling object scanned by the aligned scanner, use Isochart to obtain the geometric model with texture coordinates.
  • For each vertex on the geometric model with texture coordinates find the corresponding effective real-time shooting data according to the pose at the time of sampling and the pixel value of the sampled photo, form a high-dimensional point cloud input network, and restore the diffuse reflection feature vector and specular feature vector.
  • the LBFGS-B method to fit the coordinate system and roughness used for rendering for each vertex, and use the trust region algorithm to solve the specular reflectance and diffuse reflectance.
  • Figure 8 shows two Lumitexel vectors recovered from the verification set using the above system, and the left column is The right column is the corresponding m s .
  • Figure 9 shows the results of the material properties recovered by scanning the material appearance of the sampled objects using the above system.
  • the first row represents the sampled objects Three components
  • the second line represents the sampling object
  • the third line respectively represents the roughness coefficient ⁇ x and ⁇ y of the sampled object
  • the gray value represents the numerical value.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Software Systems (AREA)
  • Computer Graphics (AREA)
  • General Health & Medical Sciences (AREA)
  • Biophysics (AREA)
  • Evolutionary Computation (AREA)
  • Computational Linguistics (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Mathematical Physics (AREA)
  • Biomedical Technology (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Geometry (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

一种高维材质的自由式采集方法，属于计算机图形学和计算机视觉领域。将材质信息的学习转化为非结构化点云上的几何学习问题，将多个处于不同光照及观察方向下的采集结果组成高维点云，点云中的每个点为图像测量值和图像拍摄时物体的位姿信息组成的向量。本方法可以从无序、非规则、分布不均匀且精度受限的高维点云中有效地聚合非结构化视图的信息，恢复出高质量的材质属性。

Description

一种高维材质的自由式采集方法 技术领域
本发明涉及一种高维材质的自由式采集方法，属于计算机图形学和计算机视觉领域。
背景技术
真实世界物体的数字化是计算机图形学和视觉的核心问题之一。目前，数字化的真实物体可以用三维网格模型和六维的随空间变化的双向反射分布函数(SVBRDF)来表达，数字化的真实物体可以在任何视角和光照条件下逼真地再现其原貌，在文化遗产、电子商务、计算机游戏和电影制作等领域有着重要的应用。
虽然高精度的几何模型可以方便地用商业移动3D扫描仪获得，但是出于以下原因，也希望开发一种轻量型设备来进行自由的外观扫描。首先，只要能够可靠地估计出摄像机的姿态，它就可以对不同大小的物体进行扫描。第二，设备的可移动性使得可以对不允许运输的对象，例如珍贵的文物执行现场扫描。此外，制造轻量级设备所需的时间短、成本低，使其能够为更广泛的受众所接受。它还提供了类似于几何扫描的用户友好的体验。
尽管需求激增，但有效的非平面外观扫描仍然是一个有待解决的问题。一方面，现有的大多数移动外观扫描工作都是在单点/平行光的情况下进行拍摄，这导致了四维的观察和光照方向的采样效率较低，用空间分辨率换取角度精度需要先验知识(Giljoo Nam,Joo Ho Lee,Diego Gutierrez,and Min H Kim.2018.Practical SVBRDF acquisition of 3D objects with unstructured flash photography.In SIGGRAPH Asia Technical Papers.267.)。另一方面，固定采集系统在光照变化时依靠于固定的视图条件，目前尚不清楚如何将其扩展到移动设备，因为移动设备具有非结构化、不断变化的视图，并且由于其较小的外形尺寸而不能完全覆盖照明领域。
发明内容
本发明的目的在于针对现有技术的不足，提供一种高维材质的自由式采集方法，本方法可以有效利用各个视图的采集条件信息，从无序、分布不均匀的采集结果中恢复出高质量的物体材质属性。
本发明公开了一种高维材质的自由式采集方法，本方法的主要思想是，自由式的外观扫描可以转换为非结构化点云上的几何学习问题，点云中的每个点可以表示一个图像测量值和图像拍摄时物体的位姿信息。基于这一思想，本发明设计了一个神经网络，可在不同的非结构化视图中有效地聚合信息，重建空间独立的反射属性，并可以优化采集阶段使用的光照图 案，最终得到高质量的材质采集结果。本发明不依赖于某一特定的采集设备，可采用固定物体，由人手持设备进行采集，或将物体放在转盘上旋转，固定设备采集，且不限于此两种方式。
本方法提出将材质信息的学习转化为非结构化点云上的几何学习问题，将多个处于不同光照及观察方向下的采样结果组成高维点云，点云中的每个点为图像测量值和图像拍摄时物体的位姿信息组成的向量，本方法可以从无序、非规则、分布不均匀且精度受限的高维点云中有效地聚合非结构化视图的信息，恢复出高质量的材质属性。形式化的表示如下：
F(G(high dimensional point cloud))＝m
其中，点云数据特征提取方法G不限于某一特定的网络结构，其他可以从点云中提取特征的方法也适用；非线性映射网络F不限于全连接网络；物体材质属性的表达不限于Lumitexel向量m。
本方法包含两个阶段：训练阶段和采集阶段。
所述训练阶段包括以下步骤：
(1)获取采集设备的参数，生成模拟实际摄像机的采集结果，作为训练数据；
(2)使用生成的训练数据，对神经网络进行训练，神经网络的特征如下：
(2.1)神经网络的输入为k个非结构化采样下的Lumitexel向量，k为采样个数，Lumitexel的每个值描述了采样点对来自每个光源的入射光沿着某个观察方向的反射光强，Lumitexel与光源发光强度成线性关系，用线性全连接层模拟；
(2.2)神经网络的第一层包括线性全连接层，用于模拟实际采集时所用的光照图案，将所述k个Lumitexel变换为相机采集结果，这k个采集结果分别与对应的采样点的位姿信息结合组成高维点云；
(2.3)从第二层开始为特征提取网络，从所述高维点云中每个点独立地进行特征提取得到特征向量；
(2.4)特征提取网络后为最大池化层，用于聚合从k个非结构化视图中提取到的特征向量，得到全局特征向量；
(2.5)最大池化层后为非线性映射网络，用于根据所述全局特征向量恢复出高维材质信息；
所述采集阶段包括以下步骤：
(1)材质采集：采集设备按照所述光照图案依次对目标三维物体进行照射，摄像机获得一组非结构化视图下的照片，将照片作为输入，获得采样物体带有纹理坐标的几何模型和拍摄照片时摄像机的位姿；
(2)材质恢复：根据材质采集阶段拍摄照片时摄像机的位姿，得到拍摄每张照片时采样物 体上每个有效纹理坐标对应顶点的位姿；根据所述采集到的照片及位姿信息，组成所述高维点云，作为神经网络的第二层特征提取网络的输入，计算得到高维材质信息。
进一步地，所述非结构化采样为非固定视角的自由随机采样，采样数据无序、非规则、分布不均，可采用固定物体，由人手持采集设备进行采集，或将物体放在转盘上旋转，固定设备采集。
进一步地，在训练数据生成过程中，当光源为彩色时，需要对光源、采样物体和摄像机之间的光谱响应关系进行校正，校正方法如下：
定义未知的彩色光源L的光谱分布曲线为
Figure PCTCN2021098576-appb-000001
λ表示波长，c 1表示RGB三通道中的一个，光强为{I R,I G,I B}的光源的光谱分布曲线L(λ)可以表示为：
Figure PCTCN2021098576-appb-000002
将任何采样点p的反射光谱分布曲线p(λ)表示为系数分别为p R，p G，p B的三个未知的基
Figure PCTCN2021098576-appb-000003
的线性组合，c 2表示RGB三通道中的一个：
Figure PCTCN2021098576-appb-000004
摄像机C的光谱分布曲线表示为
Figure PCTCN2021098576-appb-000005
的线性组合；对于在光强为{I R,I G,I B}的光源照射下，摄像机对于反射系数为{p R,p G,p B}的采样点在某一特定通道c 3的测量值如下式：
Figure PCTCN2021098576-appb-000006
Figure PCTCN2021098576-appb-000007
在光照条件为{I R,I G,I B}＝{1,0,0}/{0,1,0}/{0,0,1}下，对已知反射系数为{p R,p G,p B}的色彩测试卡进行拍摄，根据摄像机采集到的测量值建立线性方程组，求解出大小为3×3×3的颜色校正矩阵δ(c 1,c 2,c 3)，表示光源、采样物体和摄像机之间的光谱响应关系。
进一步地，训练阶段步骤(2.1)中，物体表面一采样点p在照片上的观测值B，反射函数f r和每个光源的光强的关系可以描述为：
Figure PCTCN2021098576-appb-000008
其中，I表示每个光源l的发光信息，包括：光源l的空间位置x l、光源l的法向量n l、光源l的发光强度I(l)，P包含采样点p的参数信息，包括：采样点的空间位置x p、材质参数n，t，α x，α y，ρ d，ρ s。Ψ(x l,·)描述了光源l在不同入射方向下的光强分布，V表示x l对于x p可见性的二值函数，(·) +为两个向量的点积操作。f r(ω′ i；ω′ o,P)为ω′ o固定时关于ω′ i的二维反射函数。
神经网络的输入为k个非结构化采样下的Lumitexel向量记为m(l；P)；
Figure PCTCN2021098576-appb-000009
上式中B为在单通道下的表示，当光源为彩色光源时，将B拓展为如下形式：
Figure PCTCN2021098576-appb-000010
(ω′ i·n p) +(-ω′ i·n l) +δ(c 1,c 2,c 3)dx l
其中，f r(ω′ i；ω′ o,P,c 2)为f r(ω′ i；ω′ o,P)中
Figure PCTCN2021098576-appb-000011
的结果。
进一步地，训练阶段步骤(2.3)中，特征提取网络的公式如下：
Figure PCTCN2021098576-appb-000012
其中，f为一维卷积函数，卷积核大小为1×1，B(I,P j)表示第一层网络输出的结果或采集得到的测量值，
Figure PCTCN2021098576-appb-000013
分别为第j次采样时采样点的空间位置，采样点几何法向量和几何切向量，
Figure PCTCN2021098576-appb-000014
由几何模型获得，
Figure PCTCN2021098576-appb-000015
为与
Figure PCTCN2021098576-appb-000016
正交的任意单位向量，通过第j次采样摄像机的位姿可以将
Figure PCTCN2021098576-appb-000017
转换得到
Figure PCTCN2021098576-appb-000018
V feature(j)为网络输出的第j次采样的特征向量。
进一步地，训练阶段步骤(2.5)中，非线性映射网络形式化表达如下：
Figure PCTCN2021098576-appb-000019
Figure PCTCN2021098576-appb-000020
其中，f i+1为第i+1层网络的映射函数，W i+1为第i+1层网络的参数矩阵，b i+1为第i+1层网络的偏移向量，y i+1为第i+1层网络的输出，d与s分别表示漫反射和镜面反射两个分支，输入
Figure PCTCN2021098576-appb-000021
和
Figure PCTCN2021098576-appb-000022
为最大池化层输出的全局特征向量。
进一步地，所述神经网络的损失函数设计如下：
(1)虚拟一个Lumitextel空间，为一个中心在采样点空间位置x p的立方体，立方体中心坐标系x轴方向为
Figure PCTCN2021098576-appb-000023
z轴方向为
Figure PCTCN2021098576-appb-000024
为几何法向量，
Figure PCTCN2021098576-appb-000025
为与
Figure PCTCN2021098576-appb-000026
正交的任意单位向量；
(2)虚拟一个摄像机，观察方向为立方体z轴的正方向；
(3)对于漫反射Lumitexel，立方体分辨率为6×N d 2，对于镜面反射Lumitexel，立方体分辨率为6×N s 2，即每个面上均匀采样N d 2、N s 2个点作为光强为单位光强的虚拟点光源；
a.将采样点的镜面反射率ρ s设为0，生成在此Lumitexel空间下的漫反射特征向量
Figure PCTCN2021098576-appb-000027
b.将漫反射率ρ d设为0，生成在此Lumitexel空间下的镜面反射特征向量
Figure PCTCN2021098576-appb-000028
c.神经网络的输出为向量m d,m s，其中m d与
Figure PCTCN2021098576-appb-000029
长度相同，m s与
Figure PCTCN2021098576-appb-000030
长度相同，向量m d,m s分别为漫反射特征向量
Figure PCTCN2021098576-appb-000031
镜面反射向量
Figure PCTCN2021098576-appb-000032
的预测；
(4)材质特征部分的损失函数表达如下：
Figure PCTCN2021098576-appb-000033
其中，λ d和λ s分别表示m d,m s的损失权重，置信度β用来衡量镜面反射Lumitexel的损 失，log作用于向量的每个维度上；置信度β的确定如下：
Figure PCTCN2021098576-appb-000034
其中，
Figure PCTCN2021098576-appb-000035
项表示第j次采样所有单光源渲染值的最大值的对数，
Figure PCTCN2021098576-appb-000036
项表示第j次采样理论上可获得的单光源渲染值的最大值的对数，∈为比值调整因子。
进一步地，所述采集阶段中，在材质采集结束后进行几何对齐，之后再进行材质恢复，几何对齐具体为：使用扫描仪扫描物体得到几何模型，将其和三维重建几何模型进行对齐后替换三维重建的几何模型。
进一步地，对于有效纹理坐标，根据所述采集到的照片及采样点的位姿信息，依次取出照片中的像素，判断像素的有效性，结合对应的顶点位姿组成高维点云；对某个有效纹理坐标确定出的采样物体表面的一点p，第j次采样对于顶点p为有效的判断标准表达如下：
(1)顶点p位置
Figure PCTCN2021098576-appb-000037
在该采样下对于摄像机是可见的，且
Figure PCTCN2021098576-appb-000038
位于训练网络时定义的采样空间内；
(2)
Figure PCTCN2021098576-appb-000039
(·)为点积操作，θ为有效采样角度的下界，ω′ o表示世界坐标系下出射光方向，
Figure PCTCN2021098576-appb-000040
表示第j次顶点p的法向量；
(3)照片上像素的每个通道数值处于区间[a,b]，a，b为有效采样亮度的下界和上界；
当三个条件都满足时，认为第j次采样对于顶点p是有效的，将第j次采样的结果加入到高维点云中。
进一步地，恢复材质信息后可对材质参数进行拟合，分为两步：
(1)拟合局部坐标系及粗糙度：对某个有效纹理坐标确定出的采样物体表面的一点p，根据网络输出的单通道镜面反射向量，使用L-BFGS-B方法来拟合材质参数中的局部坐标系及粗糙度；
(2)拟合反射率：使用信赖域算法求解镜面反射率和漫反射率，求解时固定上一过程得到的局部坐标系及粗糙度，在采集所用视角下合成出观测值，使之与采集得到的观测值尽可能接近。
本发明的有益效果是：本发明方法提出将材质信息的学习转化为非结构化点云上的几何学习问题，将多个处于不同光照及观察方向下的采样结果组成高维点云，点云中的每个点为图像测量值和图像拍摄时物体的位姿信息组成的向量，本方法可以从无序、非规则、分布不均匀且精度受限的高维点云中有效地聚合非结构化视图的信息，恢复出高质量的材质属性。
附图说明
图1为本发明实施方式中的一种采集设备三维示意图；
图2为本发明实施方式中的一种采集设备正视图；
图3为本发明实施方式中的一种采集设备侧视图；
图4为本发明实施方式中的一种采集设备与采样空间关系示意图；
图5为本发明实施方式的采集方法流程图；
图6为本发明实施方式的神经网络结构示意图；
图7为本发明实施方式得到的光照图案的单通道展示，使用灰度值代表发光强度；
图8为使用本发明实施方式的系统恢复出的Lumitexel向量结果；
图9为使用本发明实施方式的系统恢复出的采样物体的材质属性结果。
具体实施方式
为使本发明的目的、技术方案和优点更加清楚，下面结合附图对本发明进行详细描述。
本发明提出的一种高维材质的自由式采集方法，具体实施可分为如下步骤：
一、训练阶段：
1.生成训练数据，获取采集设备的参数，包括光源到采样空间原点的距离及角度、光源的特性曲线、摄像机到采样空间原点的距离及角度、摄像机的内参和外参。利用这些参数生成模拟实际摄像机的采集结果，作为训练数据。生成训练数据时采用的渲染模型为GGX模型，生成公式如下：
Figure PCTCN2021098576-appb-000041
其中，f r(ω′ i,ω′ o；P)为关于ω′ i,ω′ o的四维反射函数，ω′ i表示世界坐标系下入射光方向，ω′ o表示世界坐标系下出射光方向，ω i为局部坐标系下的入射方向，ω o为局部坐标系下的出射方向，ω h为局部坐标系下的半路向量。P包含采样点的参数信息，包括采样点的材质参数n，t，α x，α y，ρ d，ρ s，其中n表示世界坐标系下的法向量，t表示世界坐标系下采样点局部坐标系的x轴方向，n与t用于将入射方向和出射方向从世界坐标系转换到局部坐标系。α x，α y表示粗糙度系数，ρ d表示漫反射率，ρ s表示镜面反射率，ρ d和ρ s在单通道下都为一个标量，在彩色情况下分别为三个标量
Figure PCTCN2021098576-appb-000042
和
Figure PCTCN2021098576-appb-000043
D GGX为微表面分布项，F为菲涅尔项，G GGX表示阴影系数函数。
当采集所用光源为彩色时，首先要对光源、采样物体和摄像机之间的光谱响应关系进行校正。校正方法如下：定义未知的彩色光源L的光谱分布曲线为
Figure PCTCN2021098576-appb-000044
λ表示波长，c 1表示RGB三通道中的一个，光强为{I R,I G,I B}的光源的光谱分布曲线L(λ)可以表示为：
Figure PCTCN2021098576-appb-000045
同样可以将任何采样点p的反射光谱分布曲线p(λ)表示为系数分别为p R，p G，p B的三个未知的基
Figure PCTCN2021098576-appb-000046
的线性组合，c 2表示RGB三通道中的一个：
Figure PCTCN2021098576-appb-000047
类似地，摄像机C的光谱分布曲线可以表示为
Figure PCTCN2021098576-appb-000048
的线性组合。所以对于在光强为{I R,I G,I B}的光源照射下，摄像机对于反射系数为{p R,p G,p B}的采样点在某一特定通道c 3的测量值如下式：
Figure PCTCN2021098576-appb-000049
Figure PCTCN2021098576-appb-000050
在光照条件为{I R,I G,I B}＝{1,0,0}/{0,1,0}/{0,0,1}下，对已知反射系数为{p R,p G,p B}的色彩测试卡进行拍摄，根据摄像机采集到的测量值建立线性方程组，可以求解出大小为3×3×3的颜色校正矩阵δ(c 1,c 2,c 3)，表示光源、采样物体和摄像机之间的光谱响应关系。
2.使用生成的训练数据，对图4所示神经网络进行训练。神经网络的特征如下：
(1)物体表面一采样点p在照片上的观测值B，反射函数f r和每个光源的光强的关系可以描述为：
Figure PCTCN2021098576-appb-000051
其中，I表示每个光源l的发光信息，包括：光源l的空间位置x l、光源l的法向量n l、光源l的发光强度I(l)，P包含采样点p的参数信息，包括：采样点的空间位置x p、材质参数n，t，α x，α y，ρ d，ρ s。Ψ(x l,·)描述了光源l在不同入射方向下的光强分布，V表示x l对于x p可见性的二值函数，(·) +为两个向量的点积操作，负值会被截断为0。f r(ω′ i；ω′ o,P)为ω′ o固定时关于ω′ i的二维反射函数。
神经网络的输入为k个无序、非规则采样下的Lumitexel，k为采样个数，Lumitexel为一个向量，记为m(l；P)，其中的每个值描述了采样点对来自每个光源的入射光沿着某个观察方向的反射光强；
Figure PCTCN2021098576-appb-000052
上式中B为在单通道下的表示，当光源为彩色光源时，将B拓展为如下形式：
Figure PCTCN2021098576-appb-000053
(ω′ i·n p) +(-ω′ i·n l) +δ(c 1,c 2,c 3)dx l
其中，f r(ω′ i；ω′ o,P,c 2)为f r(ω′ i；ω′ o,P)公式中
Figure PCTCN2021098576-appb-000054
的结果。B与光源发光强度成线性关系，可以用线性全连接层模拟。
(2)神经网络的第一层包括线性全连接层，线性全连接层的参数矩阵通过以下公式训练得到：
W l＝f W(W raw)
其中，W raw为待训练参数；W l为光照矩阵，对于单通道光源，大小为1×N，对于彩色光源，大小为3×N，N为Lumitexel的向量长度；f W为一个映射，用于对W raw进行变换，使得生成的光照矩阵能够对应到光源可能的发光强度，本实例映射f W选用Sigmoid函数，将第一层网络的光照矩阵W l的初始化取值限制在(0，1)，但f W不限于Sigmoid函数。
将W l作为光源发光强度，按照上述(1)所述关系，计算得到k个采样观测值B(I,P 1),B(I,P 2)…B(I,P k)。
(3)从第二层开始为特征提取网络，k个采样独立地进行特征提取得到特征向量，公式如下：
Figure PCTCN2021098576-appb-000055
其中，f为一维卷积函数，卷积核大小为1×1，
Figure PCTCN2021098576-appb-000056
分别为第j次采样时采样点的空间位置，采样点几何法向量和几何切向量，
Figure PCTCN2021098576-appb-000057
可通过三维重建或扫描仪扫描后得到的几何模型获得，
Figure PCTCN2021098576-appb-000058
为与
Figure PCTCN2021098576-appb-000059
正交的任意单位向量，通过第j次采样摄像机的位姿可以将
Figure PCTCN2021098576-appb-000060
转换得到
Figure PCTCN2021098576-appb-000061
V feature(j)为网络输出的第j次采样的特征向量。
(4)特征提取网络后为最大池化层。最大池化操作公式如下：
V feature＝max(V feature(1),V feature(2),…,V feature(k))
其中，最大池化操作在V feature(1),V feature(2),…,V feature(k)的每个维度上进行。
(5)最大池化层后为非线性映射网络；
Figure PCTCN2021098576-appb-000062
Figure PCTCN2021098576-appb-000063
其中，f i+1为第i+1层网络的映射函数，W i+1为第i+1层网络的参数矩阵，b i+1为第i+1层网络的偏移向量，y i+1为第i+1层网络的输出，d与s分别表示漫反射和镜面反射两个分支，输入
Figure PCTCN2021098576-appb-000064
和
Figure PCTCN2021098576-appb-000065
为V feature。
(6)神经网络的损失函数如下：
(6.1)虚拟一个Lumitextel空间，为一个中心在采样点空间位置x p的立方体，立方体中心坐标系x轴方向为
Figure PCTCN2021098576-appb-000066
z轴方向为
Figure PCTCN2021098576-appb-000067
(6.2)虚拟一个摄像机，观察方向为立方体z轴的正方向；
(6.3)对于漫反射Lumitexel，立方体分辨率为6×N d 2，对于镜面反射Lumitexel，立方体分辨率为6×N s 2，即每个面上均匀采样N d 2、N s 2个点作为光强为单位光强的虚拟点光源，本实施例中取N d＝8，N s＝32；
a.将采样点的镜面反射率ρ s设为0，生成在此Lumitexel空间下的漫反射特征向量
Figure PCTCN2021098576-appb-000068
b.将漫反射率ρ d设为0，生成在此Lumitexel空间下的镜面反射特征向量
Figure PCTCN2021098576-appb-000069
c.神经网络的输出为向量m d,m s，其中m d与
Figure PCTCN2021098576-appb-000070
长度相同，m s与
Figure PCTCN2021098576-appb-000071
长度相同，向量m d,m s分别为漫反射特征向量
Figure PCTCN2021098576-appb-000072
镜面反射向量
Figure PCTCN2021098576-appb-000073
的预测；
(6.4)材质特征部分的损失函数表达如下：
Figure PCTCN2021098576-appb-000074
其中，λ d和λ s分别表示m d,m s的损失权重，置信度β用来衡量镜面反射Lumitexel的损失，log作用于向量的每个维度上；
置信度β的确定如下：
Figure PCTCN2021098576-appb-000075
其中，
Figure PCTCN2021098576-appb-000076
项表示第j次采样所有单光源渲染值的最大值的对数，
Figure PCTCN2021098576-appb-000077
项表示第j次采样理论上可获得的单光源渲染值的最大值的对数，∈为比值调整因子，本实施例中取∈＝50％。
3.训练结束后，将网络的线性全连接层的参数W raw取出，通过公式W l＝f W(W raw)变换后作为光照图案。
二、采集阶段：采集阶段又可细分为材质采集阶段、几何对齐阶段(可选的)和材质恢复阶段。
1.材质采集阶段
采集设备按照光照图案依次对目标三维物体进行照射，摄像机获得一组非结构化视图下的照片，将照片作为输入，使用工业界公开的三维重建工具，可以获得采样物体几何模型和拍摄照片时摄像机的位姿。
2.几何对齐阶段(可选的)
(1)使用高精度扫描仪扫描物体得到几何模型；
(2)将扫描仪扫描得到的几何模型和三维重建几何模型进行对齐后替换三维重建的几何模型，对齐方法可使用领域内公开方法CPD(A.Myronenko and X.Song.2010.Point Set Registration:Coherent Point Drift.IEEE PAMI 32,12(2010),2262–2275.
https://doi.org/10.1109/TPAMI.2010.46)。
3.材质恢复阶段：
(1)根据材质采集步骤拍摄照片时摄像机的位姿得到拍摄第j张照片时采样物体上每个顶点的位姿
Figure PCTCN2021098576-appb-000078
(2)对三维重建得到的采样物体的几何模型或经对齐后的扫描仪扫所得的采样物体的几何模型，使用领域内公开工具Iso-charts，得到带有纹理坐标的几何模型；
(3)对于有效纹理坐标，根据上述一组采集到的照片r 1,r 2,…,r π及采样点的位姿信息，依次取出照片中的像素，判断像素的有效性，结合对应的顶点位姿
Figure PCTCN2021098576-appb-000079
组成高维点云，作为神经网络的第二层特征提取网络的输入向量，计算得到最后一层的输出向量m d和m s。
对某个有效纹理坐标确定出的采样物体表面的一点p，第j次采样对于点p为有效的判断标准表达如下：
1)
Figure PCTCN2021098576-appb-000080
在该采样下对于摄像机是可见的，且
Figure PCTCN2021098576-appb-000081
位于训练网络时定义的采样空间内；
2)
Figure PCTCN2021098576-appb-000082
(·)为点积操作，θ为有效采样角度的下界，本实施例中取θ＝0.3；
3)照片上像素的每个通道数值处于区间[a,b]，a，b为有效采样亮度的下界和上界，本实施例中取a＝32，b＝224；
当三个条件都满足时，认为第j次采样对于点p是有效的，将第j次采样的结果加入到高维点云中；
(4)拟合材质参数，分为两步：
1)拟合局部坐标系及粗糙度
对某个有效纹理坐标确定出的采样物体表面的一点p，根据网络输出的单通道镜面反射向量，使用L-BFGS-B方法来拟合材质参数中的局部坐标系及粗糙度，优化目标为：
Figure PCTCN2021098576-appb-000083
其中i为(6.3)中虚拟光源序号，m s(l)表示网络预测的镜面反射特征向量第l个维度上的值，ω′ i表示该序号为l的虚拟光源与采样点形成的入射角，ω′ o表示从采样点到(6.2)中虚拟摄像机形成的出射角，P包含该采样点的用于渲染的法向量n′、切向量t′和其他材质参数p′，因所选用模型不同而不同。如本工程所用GGX模型，则p′包括各向异性粗糙度、镜面反射率、漫反射率。上述优化目标中n′,t′,p′为可优化参数。
2)拟合反射率
该过程使用信赖域算法求解镜面反射率和漫反射率，拟合的目标为：
Figure PCTCN2021098576-appb-000084
其中，
Figure PCTCN2021098576-appb-000085
表示该像素在第j个用于拟合的视角下，被上述优化得到的光照图案照射后，摄像机的观测值。B j表示该像素在第j个用于拟合的视角下，使用上一过程得到的n′和t′及粗糙度合成出的观测值。对于彩色光源则合成参数还包括校正得到的颜色校正矩阵δ(c 1,c 2,c 3)。B j计算过程如下：
首先将漫反射率设为1，镜面反射率设为0，使用上一步得到的用于渲染的坐标系和粗糙度，渲染出第j个视角下的漫反射Lumitexel，
Figure PCTCN2021098576-appb-000086
再将漫反射率设为0，镜面反射率设为1，使用上一步得到的用于渲染的坐标系和粗糙度，渲染出第j个视角下的镜面反射Lumitexel，
Figure PCTCN2021098576-appb-000087
将两个Lumitexel连接起来，形成矩阵
Figure PCTCN2021098576-appb-000088
大小为N×2，其中N为采样设备的光源数。将采样时所用大小为3×N的光照图案矩阵W l与M j相乘，得到W lM j，大小为3×2。再与大小为2×3的可优化变量ρ d,s相乘，得到
T＝W lM jρ d,s
T的大小为3×3，将其拷贝连接形成张量
Figure PCTCN2021098576-appb-000089
在后两维上求和，最终得到三维向量B j。对于彩色光源，
Figure PCTCN2021098576-appb-000090
要与校准设备所得的颜色校正矩阵δ(c 1,c 2,c 3)逐元素相乘，再在后两维上求和。
以下给出一个具体的采集设备系统实例，如图1为系统实例三维展示，图2为正视图，图3为侧视图，该采集设备由1个灯板组成，上部固定有一个摄像头，用于采集图像。灯板上密集地排列了LED灯珠，共512个。灯珠由FPGA控制，可以调整发光亮度及发光时间。
以下给出一个应用本发明方法的获取系统实例，系统总体分为如下几个模块：
准备模块：为网络训练提供数据集，该部分使用GGX模型，输入一组材质参数及k个采样点的位姿信息，摄像机位置，可得到k个反射情况组成的高维点云。网络训练部分使用Pytorch开源框架，并使用Adam优化器进行训练。网络结构如图6所示，每个矩形表示一层神经元，矩形中的数字表示该层神经元个数。最左侧层为输入层，最右侧层为输出层。层与层之间实线箭头表示全连接，虚线箭头表示卷积。
采集模块：设备如图1、2、3所示，具体构成上文已描述，本系统定义的采样空间大小及采集设备和采样空间在空间上的位置关系如图4所示。
恢复模块：用三维重建得到的采样物体的几何模型或经对齐后的扫描仪扫所得的采样物体的几何模型计算得到带有纹理坐标的几何模型，加载训练好的神经网络，对带有纹理坐标的几何模型上的每个顶点，预测材质特征向量，拟合用于渲染的坐标系和材质参数。
图5为本实施例的工作流程。首先生成训练数据，随机采样得到2亿个Lumitexel，取80％作为训练集，其余作为验证集。训练网络时使用Xavier方法进行初始化参数，学习率为1e-4。光照图案为彩色，光照矩阵的大小为(3,512)，矩阵的三行分表表示红、绿、蓝三通道的光照图案。训练结束后，将光照矩阵取出，变换为光照图案，每列的参数指定了该位置处，光源的发光强度，图7展示了一个网络训练得到的红、绿、蓝三通道光照图案。接下来的流程为：1.手持设备，灯板按光照图案发光，摄像机同时对物体进行拍摄，得到一组采样结果。2.对于三维重建得到的采样物体的几何模型或经对齐后的扫描仪扫所得的采样物体的几何模型，使用Isochart得到带有纹理坐标的几何模型。3.对带有纹理坐标的几何模型上的每个顶点，根据采样时的位姿及采样照片的像素值找到对应的有效的实拍数据，组成高维点云输入网络，恢复出漫反射特征向量和镜面反射特征向量。4.根据网络输出的漫反射特征向量和镜面反射特征向量，对每个顶点使用LBFGS-B方法拟合用于渲染的坐标系及粗糙度，使用信赖域算法求解镜面反射率和漫反射率。
图8展示了两个使用上述系统恢复出验证集中的Lumitexel向量，左侧一列为
Figure PCTCN2021098576-appb-000091
右侧一列为对应的m s。
图9展示了使用上述系统对采样物体进行材质外观扫描恢复出的材质属性结果，第一行分别表示采样物体
Figure PCTCN2021098576-appb-000092
三个分量，第二行分别表示采样物体
Figure PCTCN2021098576-appb-000093
三个分量，第三行分别表示采样物体粗糙度系数α x，α y，灰度值代表数值大小。
以上所述，仅为较佳实施样例，本发明并不局限于上述实施方式，只要以相同手段达到本发明的技术效果，都应属于本发明的保护范围。在本发明的保护范围内，其技术方案和/或实施方式可以有各种不同的修改和变化。

Claims (10)

  1. 一种高维材质的自由式采集方法，其特征在于，该方法包括训练阶段和采集阶段；
    所述训练阶段包括以下步骤：
    (1)获取采集设备的参数，生成模拟实际摄像机的采集结果，作为训练数据；
    (2)使用生成的训练数据，对神经网络进行训练，神经网络的特征如下：
    (2.1)神经网络的输入为k个非结构化采样下的Lumitexel向量，k为采样个数，Lumitexel的每个值描述了采样点对来自每个光源的入射光沿着某个观察方向的反射光强，Lumitexel与光源发光强度成线性关系，用线性全连接层模拟；
    (2.2)神经网络的第一层包括线性全连接层，用于模拟实际采集时所用的光照图案，将所述k个Lumitexel变换为相机采集结果，这k个采集结果分别与对应的采样点的位姿信息结合组成高维点云；
    (2.3)从第二层开始为特征提取网络，从所述高维点云中每个点独立地进行特征提取得到特征向量；
    (2.4)特征提取网络后为最大池化层，用于聚合从k个非结构化视图中提取到的特征向量，得到全局特征向量；
    (2.5)最大池化层后为非线性映射网络，用于根据所述全局特征向量恢复出高维材质信息；
    所述采集阶段包括以下步骤：
    (1)材质采集：采集设备按照所述光照图案依次对目标三维物体进行照射，摄像机获得一组非结构化视图下的照片，将照片作为输入，获得采样物体带有纹理坐标的几何模型和拍摄照片时摄像机的位姿；
    (2)材质恢复：根据材质采集阶段拍摄照片时摄像机的位姿，得到拍摄每张照片时采样物体上每个有效纹理坐标对应顶点的位姿；根据所述采集到的照片及位姿信息，组成所述高维点云，作为神经网络的第二层特征提取网络的输入，计算得到高维材质信息。
  2. 根据权利要求1所述的一种高维材质的自由式采集方法，其特征在于，所述非结构化采样为非固定视角的自由随机采样，采样数据无序、非规则、分布不均，可采用固定物体，由人手持采集设备进行采集，或将物体放在转盘上旋转，固定设备采集。
  3. 根据权利要求1所述的一种高维材质的自由式采集方法，其特征在于，在训练数据生成过程中，当光源为彩色时，需要对光源、采样物体和摄像机之间的光谱响应关系进行校正，校正方法如下：
    定义未知的彩色光源L的光谱分布曲线为
    Figure PCTCN2021098576-appb-100001
    λ表示波长，c 1表示RGB三通道中的一 个，光强为{I R,I G,I B}的光源的光谱分布曲线L(λ)可以表示为：
    Figure PCTCN2021098576-appb-100002
    将任何采样点p的反射光谱分布曲线p(λ)表示为系数分别为p R，p G，p B的三个未知的基
    Figure PCTCN2021098576-appb-100003
    的线性组合，c 2表示RGB三通道中的一个：
    Figure PCTCN2021098576-appb-100004
    摄像机C的光谱分布曲线表示为
    Figure PCTCN2021098576-appb-100005
    的线性组合；对于在光强为{I R,I G,I B}的光源照射下，摄像机对于反射系数为{p R,p G,p B}的采样点在某一特定通道c 3的测量值如下式：
    Figure PCTCN2021098576-appb-100006
    Figure PCTCN2021098576-appb-100007
    在光照条件为{I R,I G,I B}＝{1,0,0}/{0,1,0}/{0,0,1}下，对已知反射系数为{p R,p G,p B}的色彩测试卡进行拍摄，根据摄像机采集到的测量值建立线性方程组，求解出大小为3×3×3的颜色校正矩阵δ(c 1,c 2,c 3)，表示光源、采样物体和摄像机之间的光谱响应关系。
  4. 根据权利要求3所述的一种高维材质的自由式采集方法，其特征在于，训练阶段步骤(2.1)中，物体表面一采样点p在照片上的观测值B，反射函数f r和每个光源的光强的关系可以描述为：
    Figure PCTCN2021098576-appb-100008
    其中，I表示每个光源l的发光信息，包括：光源l的空间位置x l、光源l的法向量n l、光源l的发光强度I(l)，P包含采样点p的参数信息，包括：采样点的空间位置x p、材质参数n，t，α x，α y，ρ d，ρ s。Ψ(x l,·)描述了光源l在不同入射方向下的光强分布，V表示x l对于x p可见性的二值函数，(·) +为两个向量的点积操作。f r(ω′ i；ω′ o,P)为ω′ o固定时关于ω′ i的二维反射函数。
    神经网络的输入为k个非结构化采样下的Lumitexel向量记为m(l；P)；
    Figure PCTCN2021098576-appb-100009
    上式中B为在单通道下的表示，当光源为彩色光源时，将B拓展为如下形式：
    Figure PCTCN2021098576-appb-100010
    (ω′ i·n p) +(-ω′ i·n l) +δ(c 1,c 2,c 3)dx l
    其中，f r(ω′ i；ω′ o,P,c 2)为f r(ω′ i；ω′ o,P)中
    Figure PCTCN2021098576-appb-100011
    的结果。
  5. 根据权利要求1所述的一种高维材质的自由式采集方法，其特征在于，训练阶段步骤(2.3)中，特征提取网络的公式如下：
    Figure PCTCN2021098576-appb-100012
    其中，f为一维卷积函数，卷积核大小为1×1，B(I,P j)表示第一层网络输出的结果或采集得到的测量值，
    Figure PCTCN2021098576-appb-100013
    分别为第j次采样时采样点的空间位置，采样点几何法向量和几何切向量，
    Figure PCTCN2021098576-appb-100014
    由几何模型获得，
    Figure PCTCN2021098576-appb-100015
    为与
    Figure PCTCN2021098576-appb-100016
    正交的任意单位向量，通过第j次采样摄像机的位姿可以将
    Figure PCTCN2021098576-appb-100017
    转换得到
    Figure PCTCN2021098576-appb-100018
    V feature(j)为网络输出的第j次采样的特征向量。
  6. 根据权利要求1所述的一种高维材质的自由式采集方法，其特征在于，训练阶段步骤(2.5)中，非线性映射网络形式化表达如下：
    Figure PCTCN2021098576-appb-100019
    Figure PCTCN2021098576-appb-100020
    其中，f i+1为第i+1层网络的映射函数，W i+1为第i+1层网络的参数矩阵，b i+1为第i+1层网络的偏移向量，y i+1为第i+1层网络的输出，d与s分别表示漫反射和镜面反射两个分支，输入
    Figure PCTCN2021098576-appb-100021
    和
    Figure PCTCN2021098576-appb-100022
    为最大池化层输出的全局特征向量。
  7. 根据权利要求1所述的一种高维材质的自由式采集方法，其特征在于，所述神经网络的损失函数设计如下：
    (1)虚拟一个Lumitextel空间，为一个中心在采样点空间位置x p的立方体，立方体中心坐标系x轴方向为
    Figure PCTCN2021098576-appb-100023
    z轴方向为
    Figure PCTCN2021098576-appb-100024
    为几何法向量，
    Figure PCTCN2021098576-appb-100025
    为与
    Figure PCTCN2021098576-appb-100026
    正交的任意单位向量；
    (2)虚拟一个摄像机，观察方向为立方体z轴的正方向；
    (3)对于漫反射Lumitexel，立方体分辨率为6×N d 2，对于镜面反射Lumitexel，立方体分辨率为6×N s 2，即每个面上均匀采样N d 2、N s 2个点作为光强为单位光强的虚拟点光源；
    a.将采样点的镜面反射率ρ s设为0，生成在此Lumitexel空间下的漫反射特征向量
    Figure PCTCN2021098576-appb-100027
    b.将漫反射率ρ d设为0，生成在此Lumitexel空间下的镜面反射特征向量
    Figure PCTCN2021098576-appb-100028
    c.神经网络的输出为向量m d,m s，其中m d与
    Figure PCTCN2021098576-appb-100029
    长度相同，m s与
    Figure PCTCN2021098576-appb-100030
    长度相同，向量m d,m s分别为漫反射特征向量
    Figure PCTCN2021098576-appb-100031
    镜面反射向量
    Figure PCTCN2021098576-appb-100032
    的预测；
    (4)材质特征部分的损失函数表达如下：
    Figure PCTCN2021098576-appb-100033
    其中，λ d和λ s分别表示m d,m s的损失权重，置信度β用来衡量镜面反射Lumitexel的损失，log作用于向量的每个维度上；
    置信度β的确定如下：
    Figure PCTCN2021098576-appb-100034
    其中，
    Figure PCTCN2021098576-appb-100035
    项表示第j次采样所有单光源渲染值的最大值的对数，
    Figure PCTCN2021098576-appb-100036
    项表示第j次采样理论上可获得的单光源渲染值的最大值的对数，ε为比值调整因子。
  8. 根据权利要求1所述的一种高维材质的自由式采集方法，其特征在于，所述采集阶段中，在材质采集结束后进行几何对齐，之后再进行材质恢复，几何对齐具体为：使用扫描仪扫描物体得到几何模型，将其和三维重建几何模型进行对齐后替换三维重建的几何模型。
  9. 根据权利要求1所述的一种高维材质的自由式采集方法，其特征在于，对于有效纹理坐标，根据所述采集到的照片及采样点的位姿信息，依次取出照片中的像素，判断像素的有效性，结合对应的顶点位姿组成高维点云；对某个有效纹理坐标确定出的采样物体表面的一点p，第j次采样对于顶点p为有效的判断标准表达如下：
    (1)顶点p位置
    Figure PCTCN2021098576-appb-100037
    在该采样下对于摄像机是可见的，且
    Figure PCTCN2021098576-appb-100038
    位于训练网络时定义的采样空间内；
    (2)
    Figure PCTCN2021098576-appb-100039
    (·)为点积操作，θ为有效采样角度的下界，ω′ o表示世界坐标系下出射光方向，
    Figure PCTCN2021098576-appb-100040
    表示第j次顶点p的法向量；
    (3)照片上像素的每个通道数值处于区间[a,b]，a，b为有效采样亮度的下界和上界；
    当三个条件都满足时，认为第j次采样对于顶点p是有效的，将第j次采样的结果加入到高维点云中。
  10. 根据权利要求1所述的一种高维材质的自由式采集方法，其特征在于，恢复材质信息后可对材质参数进行拟合，分为两步：
    (1)拟合局部坐标系及粗糙度：对某个有效纹理坐标确定出的采样物体表面的一点p，根据网络输出的单通道镜面反射向量，使用L-BFGS-B方法来拟合材质参数中的局部坐标系及粗糙度；
    (2)拟合反射率：使用信赖域算法求解镜面反射率和漫反射率，求解时固定上一过程得到的局部坐标系及粗糙度，在采集所用视角下合成出观测值，使之与采集得到的观测值尽可能接近。
PCT/CN2021/098576 2021-06-07 2021-06-07 一种高维材质的自由式采集方法 WO2022256962A1 (zh)

Priority Applications (3)

Application Number Priority Date Filing Date Title
JP2023558118A JP2024511083A (ja) 2021-06-07 2021-06-07 高次元マテリアルの自由式収集方法
PCT/CN2021/098576 WO2022256962A1 (zh) 2021-06-07 2021-06-07 一种高维材质的自由式采集方法
US18/493,831 US20240062460A1 (en) 2021-06-07 2023-10-25 Freestyle acquisition method for high-dimensional material

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2021/098576 WO2022256962A1 (zh) 2021-06-07 2021-06-07 一种高维材质的自由式采集方法

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/493,831 Continuation US20240062460A1 (en) 2021-06-07 2023-10-25 Freestyle acquisition method for high-dimensional material

Publications (1)

Publication Number Publication Date
WO2022256962A1 true WO2022256962A1 (zh) 2022-12-15

Family

ID=84424648

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/098576 WO2022256962A1 (zh) 2021-06-07 2021-06-07 一种高维材质的自由式采集方法

Country Status (3)

Country Link
US (1) US20240062460A1 (zh)
JP (1) JP2024511083A (zh)
WO (1) WO2022256962A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116188586A (zh) * 2023-04-25 2023-05-30 新乡学院 一种基于光分布的定位系统与方法

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102819864A (zh) * 2012-08-14 2012-12-12 北京航空航天大学 基于光度图的几何与表面材质的重建方法
US20160162760A1 (en) * 2014-12-08 2016-06-09 Canon Kabushiki Kaisha Devices, systems, and methods for learning and identifying visual features of materials
CN108985333A (zh) * 2018-06-15 2018-12-11 浙江大学 一种基于神经网络的材质获取方法和系统
CN109858462A (zh) * 2019-02-21 2019-06-07 武汉纺织大学 一种基于卷积神经网络的织物识别方法及系统
US20190347526A1 (en) * 2018-05-09 2019-11-14 Adobe Inc. Extracting material properties from a single image
CN110570503A (zh) * 2019-09-03 2019-12-13 浙江大学 基于神经网络的三维物体法向量、几何及材质获取方法
CN111831808A (zh) * 2020-07-16 2020-10-27 中国科学院计算机网络信息中心 一种数据驱动的人工智能材料预测系统
CN112256895A (zh) * 2020-10-16 2021-01-22 江南大学 一种基于多任务学习的织物图像检索方法
CN112649391A (zh) * 2020-12-03 2021-04-13 浙江大学 一种纤维质量等级在线检测系统及其应用

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102819864A (zh) * 2012-08-14 2012-12-12 北京航空航天大学 基于光度图的几何与表面材质的重建方法
US20160162760A1 (en) * 2014-12-08 2016-06-09 Canon Kabushiki Kaisha Devices, systems, and methods for learning and identifying visual features of materials
US20190347526A1 (en) * 2018-05-09 2019-11-14 Adobe Inc. Extracting material properties from a single image
CN108985333A (zh) * 2018-06-15 2018-12-11 浙江大学 一种基于神经网络的材质获取方法和系统
CN109858462A (zh) * 2019-02-21 2019-06-07 武汉纺织大学 一种基于卷积神经网络的织物识别方法及系统
CN110570503A (zh) * 2019-09-03 2019-12-13 浙江大学 基于神经网络的三维物体法向量、几何及材质获取方法
CN111831808A (zh) * 2020-07-16 2020-10-27 中国科学院计算机网络信息中心 一种数据驱动的人工智能材料预测系统
CN112256895A (zh) * 2020-10-16 2021-01-22 江南大学 一种基于多任务学习的织物图像检索方法
CN112649391A (zh) * 2020-12-03 2021-04-13 浙江大学 一种纤维质量等级在线检测系统及其应用

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116188586A (zh) * 2023-04-25 2023-05-30 新乡学院 一种基于光分布的定位系统与方法
CN116188586B (zh) * 2023-04-25 2023-06-27 新乡学院 一种基于光分布的定位系统与方法

Also Published As

Publication number Publication date
JP2024511083A (ja) 2024-03-12
US20240062460A1 (en) 2024-02-22

Similar Documents

Publication Publication Date Title
US11869139B2 (en) System and method for three-dimensional scanning and for capturing a bidirectional reflectance distribution function
Shi et al. A benchmark dataset and evaluation for non-lambertian and uncalibrated photometric stereo
Zhou et al. Color map optimization for 3d reconstruction with consumer depth cameras
WO2021042277A1 (zh) 基于神经网络的三维物体法向量、几何及材质获取方法
Wang et al. Modeling anisotropic surface reflectance with example-based microfacet synthesis
EP3664039B1 (en) Method for acquiring three-dimensional object by using artificial lighting photograph and device thereof
JP3575679B2 (ja) 顔照合方法と該照合方法を格納した記録媒体と顔照合装置
EP3382645B1 (en) Method for generation of a 3d model based on structure from motion and photometric stereo of 2d sparse images
US20220215266A1 (en) Systems and methods for synthesizing data for training statistical models on different imaging modalities including polarized images
CN107657656B (zh) 同名点匹配及三维重建方法、系统和光度立体摄像终端
Ciortan et al. A practical reflectance transformation imaging pipeline for surface characterization in cultural heritage
Huang et al. Deprocams: Simultaneous relighting, compensation and shape reconstruction for projector-camera systems
CN109816706A (zh) 一种平滑约束与三角网等比例剖分像对稠密匹配方法
US20240062460A1 (en) Freestyle acquisition method for high-dimensional material
CN108985333A (zh) 一种基于神经网络的材质获取方法和系统
CN115115776A (zh) 基于影子的单幅图像三维人体重建方法及装置
Pintus et al. Techniques for seamless color registration and mapping on dense 3D models
Hardy et al. Ms-ps: A multi-scale network for photometric stereo with a new comprehensive training dataset
Ono et al. Practical BRDF reconstruction using reliable geometric regions from multi-view stereo
CN113362466B (zh) 一种高维材质的自由式采集方法
CN114241059B (zh) 一种光度立体视觉系统中相机和光源的同步标定方法
Pintus et al. Practical free-form RTI acquisition with local spot lights
Gallardo et al. Using Shading and a 3D Template to Reconstruct Complex Surface Deformations.
US20220270321A1 (en) Image Processing for Generating Three-Dimensional Shape and Spatially-Varying Reflectance of the Object using a Deep Neural Network
WO2022175688A1 (en) Image processing

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21944475

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2023558118

Country of ref document: JP

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21944475

Country of ref document: EP

Kind code of ref document: A1