WO2020076026A1 - Method for acquiring three-dimensional object by using artificial lighting photograph and device thereof - Google Patents

Abstract

Disclosed are a method for acquiring a three-dimensional object by using an artificial lighting photograph and a device thereof. A three-dimensional object acquiring method according to an embodiment of the present invention comprises the steps of: receiving a plurality of images obtained by photographing a three-dimensional object by a camera; reconstructing spatially-varying bidirectional reflectance distribution functions for the three-dimensional object, on the basis of the received plurality of images; estimating a shading normals for the three-dimensional object on the basis of the reconstructed spatially-varying bidirectional reflectance distribution functions; and obtaining a three-dimensional geometry for the three-dimensional object on the basis of the estimated shading normals.

Description

3D object acquisition method and device using artificial lighting photo

The present invention relates to a three-dimensional object acquisition technology, and more specifically, artificial lighting, for example, the shape information and the surface reflectometer information (Bidirectional Reflectance Distribution Function, of the three-dimensional object from a photograph or image taken with the flash on) BRDF) is a method and apparatus for measuring or obtaining.

Acquiring and reproducing the appearance of a real object is one of the main goals of computer graphics, and various methods have been proposed, ranging from a method that relies on special hardware to a method that relies on mobile setup. However, there is an inevitable trade-off between the functionality and cost of this method. For example, the mobile method is inexpensive, but is almost limited to planar geometry.

Capturing spatially-varying bidirectional reflectance distribution functions (SVBRDF) of non-planar 3D objects with commercial cameras such as smartphone cameras has not been implemented due to the following technical problems.

First, in order to capture SVBRDF information, high-density optical view space sampling is required. These requirements are generally achieved using professional support structures such as light domes or four-dimensional gantry. Since the built-in flash of a conventional camera is placed almost the same as a lens, it severely limits the sampling of the light-view space. Also, near-field flash lighting (or flash light) illuminates the object non-uniformly.

Second, since most existing cameras do not have a depth sensor, restoring geometry requires relying on passive multi-view stereo technology that assumes only diffuse reflections. However, in the presence of spatially varying surface reflections, this leads to incorrect reconstruction.

Third, due to unstructured characteristics (or artificial characteristics) of handheld acquisition, pixel correspondence between 3D points of objects and 2D pixels of multiple images is not guaranteed. This correspondence is ensured in photometric stereo setups where a fixed camera and various light sources are used or the camera and light provide a structured input locked to a physical structure. In the handheld approach, both the intrinsic / extrinsic parameters, as well as the 3D geometry of the object, must be provided, which is information that is not possessed. The resulting imbalance is a major obstacle to high-quality reconstruction of 3D geometry and SVBRDF using unstructured capture setup.

Most acquisition methods using conventional cameras focus only on geometry or reflection capture, and generally require special hardware such as a mechanical gantry, optical stage or commercial 3D scanner to acquire both. .

Existing studies would be categorized into (a) reflection capture from known 3D geometry, (b) reflection capture limited to 2D planar geometry, (c) 3D reconstruction assuming only diffuse reflection, (d) simultaneous acquisition of reflection and 3D geometry, etc. You can.

One embodiment technique for capturing reflections from known 3D geometry has introduced a method of capturing SVBRDF of known 3D objects that contain clustered basis reflections. Per-texel reflection is progressively improved with linear blending. Another embodiment of the technique proposed an SVBRDF acquisition method that jointly optimizes the reflection basis and blend weights for known 3D geometry, which finds the smallest basis reflection and then gently blends them. However, these techniques require a commercial 3D scanner to accurately capture the input 3D geometry.

Conventional cameras, such as smartphone cameras, are also used to capture flat surface reflectometer information. One embodiment technique for reflection capture limited to 2D planar geometry is an efficient SVBRDF capture method that uses a LCD screen and camera to limit the range of angular reflection samples, as well as two flash / flashless reflections for specific cases of stationary material. An acquisition method was proposed, and a larger region can be synthesized from a small reconstruction through this method. Another embodiment technique uses a smartphone camera from various view-points to capture the appearance of adjacent planar objects, and the light source provides active illumination where normals and reflections are estimated. Another embodiment technology proposed a portable system consisting of a smart phone camera, a handheld linear light source and a custom BRDF chart, the technology taking a short video of the target object along the BRDF chart while moving the handheld light tube, Recover SVBRDF from a linear combination of reference BRDFs. Another embodiment technique presents an acquisition setup similar to a method focused on obtaining a planar art painting reflection. Other acquisition systems that capture high quality SVBRDF on a flat surface rely on more sophisticated hardware. For example, using a computer-controlled LED light to place a sample on a small dome to obtain both reflectance and normal at the microscope scale at the same time, different systems have many different light-camera combinations, linear light reflection measurements, or condenser lenses. Includes a 4-axis spherical gantry for sampling. These acquisition methods are limited to near-flat objects.

Shaded normals are often used to improve geometry detail by assuming reflectance, which only has diffusion throughout the object. One embodiment technique for 3D reconstruction assuming only diffuse reflection is obtained using base structure estimated using SfM (Structural From Motion) and Multiview Stereo (MVS), and then assuming diffuse reflection, using the estimated surface normal geometry Update it. Another embodiment technique uses Kinect Fusion to obtain the indicated distance function of the surface, and refines the distance function using surface shading cues. Another embodiment technique uses two mobile devices as a camera and a light source, respectively, and takes multiple images at fixed viewpoints under various light directions to reconstruct the surface from the metering stereo. Other recent methods have further demonstrated the use of smartphone cameras or large scenes to capture the 3D shape of the objects, which are based on SfM and MVS technologies. All of these methods cannot recover SVBRDF information, assuming that the surface reflection of the reconstructed object only diffuses.

Explaining the simultaneous acquisition of reflections and 3D geometry, previous studies that captured reflections and 3D geometry simultaneously relied on special hardware setups costing hundreds of thousands of dollars. For example, there are many optical stage designs that rely on discrete spherical illumination using polarized light sources. The technique of one embodiment has created a structure similar to an LED arm that orbits rapidly to generate a continuous spherical light in a harmonic pattern, and another embodiment technique uses two mechanical arms using a phase shift pattern for 3D geometry. In this case, a spherical gantry with a projector-camera pair was created, and similar dome structures of several cameras were presented. This structure uses structured light patterns for 3D geometry and exhibits reflections with bidirectional texture functions (BTF).

In addition, there are more applicable methods that require inexpensive hardware such as optical probes, multiple optical structures or RGB-D cameras. One embodiment technique further estimates SVBRDF based on 3D geometry by making a multi-light device consisting of 72 LED lights on a circle board to obtain 3D geometry by combining SfM and metering stereo, but using this information It does not refine the geometry or surface normals. Another embodiment technique solves the inverse rendering problem by using an environment map for estimating spherical illumination by capturing an optical probe. Another embodiment technique uses a mechanical rotating stage to capture a sequence of videos over a thousand frames, which requires high density sampling with at least two distinct changes in illumination per vertex (or vertex). Another embodiment technique uses spherical illumination or IR illumination from a depth camera, as well as relying on depth information using Kinect Fusion. Recently, we proposed an SVBRDF acquisition method that can capture the polarized appearance of diffuse reflections and specular reflections using high-resolution normals that rely on the input geometry of structured lighting.

These systems rely on special hardware, such as bulky, expensive to build, or hardware that may be difficult to secure, or use additional sources of information such as multiple light, spherical lighting or depth. In addition, existing methods generally require a time-consuming calibration process, while acquisition times often take hours.

Embodiments of the present invention, artificial lighting, for example, a method and apparatus for measuring or obtaining the shape information and surface reflectometer information (BRDF) of a three-dimensional object from a photograph or image taken with the flash on to provide.

A 3D object acquisition method according to an embodiment of the present invention includes receiving a plurality of images of a 3D object photographed by a camera; Reconstructing spatially changing surface reflectometer information for the 3D object based on the received plurality of images; Estimating a shadow normal for the 3D object based on the reconstructed spatially changing surface reflectometer information; And obtaining 3D geometry for the 3D object based on the estimated shading normal.

The reconstructing step estimates base surface reflectometer information for the 3D object based on the received plurality of images, and spatially changes the weight based on the estimated base surface reflectometer information and a spatially varying weight map. The changing surface reflectometer information can be reconstructed.

Furthermore. The method for acquiring a 3D object according to an embodiment of the present invention further includes acquiring a base geometry of the 3D object based on external parameters of the camera and the plurality of images, and the reconstructing step includes the plurality of Based on the image and the base geometry, it is possible to reconstruct the spatially changing surface reflectometer information.

The reconstructing step and the estimating step reconstruct the spatially changing surface reflectometer information using an inverse rendering technique that satisfies a preset image forming model for the camera, and estimate the shading normal can do.

The acquiring step may acquire the three-dimensional geometry using a Poisson surface reconstruction technique.

The reconstructing step updates the spatially changing surface reflectometer information based on the obtained 3D geometry and the plurality of images, and the estimating step is based on the updated spatially changing surface reflectometer information By updating the shading normal, the acquiring step may update the 3D geometry based on the updated shading normal.

In the receiving step, a plurality of images of the 3D object photographed while artificial lighting is on may be received.

Furthermore, a 3D object acquisition method according to an embodiment of the present invention includes obtaining a 3D point cloud for the 3D object from the plurality of images and the pose of the camera using multiview stereo; Generating a base mesh for the 3D object based on the 3D point cloud and Poisson surface reconstruction technique; And subdividing the base mesh to obtain initial geometry for the 3D object, and the reconstructing step includes the spatially changing surface reflectometer information based on the plurality of images and the initial geometry. Can be reconstructed.

A method for acquiring a 3D object according to another embodiment of the present invention includes receiving a plurality of images of a 3D object photographed by a camera while artificial lighting is on; And acquiring spatially changing surface reflectometer information and 3D geometry for the 3D object based on the received plurality of images and Poisson surface reconstruction technique.

The acquiring may include reconstructing the spatially changing surface reflectometer information based on the received plurality of images; Estimating a shadow normal for the 3D object based on the reconstructed spatially changing surface reflectometer information; And acquiring the 3D geometry based on the estimated shading normal and the Poisson surface reconstruction technique.

An apparatus for acquiring a 3D object according to an embodiment of the present invention includes a receiver configured to receive a plurality of images of a 3D object photographed by a camera; A reconstruction unit reconstructing spatially changing surface reflectometer information for the 3D object based on the received plurality of images; An estimator for estimating a shadow normal for the 3D object based on the reconstructed spatially changing surface reflectometer information; And an acquiring unit acquiring 3D geometry for the 3D object based on the estimated shadow normal.

The reconstruction unit estimates base surface reflectometer information for the 3D object based on the received plurality of images, and changes spatially based on the estimated base surface reflectometer information and a spatially changing weight map. The surface reflectometer information can be reconstructed.

The reconstruction unit acquires the base geometry of the 3D object based on the camera's external parameters and the plurality of images, and reconstructs the spatially changing surface reflectometer information based on the plurality of images and the basis geometry. You can.

The reconstructing unit and the estimating unit may reconstruct the spatially changing surface reflectometer information using an inverse rendering technique that satisfies a preset image forming model for the camera, and estimate the shading normal.

The acquiring unit may acquire the 3D geometry using a Poisson surface reconstruction technique.

The reconstructing unit updates the spatially changing surface reflectometer information based on the obtained three-dimensional geometry and the plurality of images, and the estimating unit is configured to shade the normal based on the updated spatially changing surface reflectometer information. And updating, the obtaining unit may update the 3D geometry based on the updated shadow normal.

The receiving unit may receive a plurality of images of the 3D object photographed while artificial lighting is on.

The reconstruction unit acquires a 3D point cloud for the 3D object from the poses of the plurality of images and the camera using multiview stereo, and based on the 3D point cloud and Poisson surface reconstruction technique, the 3D object To generate a base mesh for, and to subdivide the base mesh to obtain the initial geometry for the three-dimensional object, it is possible to reconstruct the spatially changing surface reflectometer information based on the plurality of images and the initial geometry have.

A method for acquiring a 3D object according to another embodiment of the present invention includes receiving a plurality of images of a 3D object taken by a camera; Reconstructing spatially changing surface reflectometer information for the 3D object based on the received plurality of images; And acquiring 3D geometry for the 3D object based on the reconstructed spatially changing surface reflectometer information.

The reconstructing may reconstruct spatially changing surface reflectometer information for the 3D object based on the received plurality of images and depth information for the 3D object.

According to embodiments of the present invention, it is possible to simultaneously measure shape information and surface reflectometer information (BRDF) of a 3D object that can be measured only by using special equipment using a camera equipped with a general flash.

According to embodiments of the present invention, since it is possible to measure high-quality 3D shape information and BRDF surface reflectometer information without relying on special equipment, it may affect the spread of 3D scanning technology.

The present invention is a technique required for extremely realistic rendering of a 3D model, and thus, it can be applied to virtual reality and augmented reality technologies and various applications using the same.

1 is a flowchart illustrating an operation of a 3D object acquisition method according to an embodiment of the present invention.

Figure 2 shows an exemplary view for explaining the outline of the three-dimensional object acquisition method according to the present invention.

Figure 3 shows an exemplary view of the geometry (b) of the present invention with a Rusinkiewicz parameter (a) of a light and camera and a smartphone.

4 illustrates an exemplary diagram for explaining a process of updating a 3D geometry.

FIG. 5 shows an exemplary diagram for a comparison result of the Nehab method and the screened Poisson reconstruction method for the unstructured capture setup of the present invention.

6 illustrates a configuration of a 3D object acquisition device according to an embodiment of the present invention.

Advantages and features of the present invention, and methods for achieving them will be clarified with reference to embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in various different forms, and only the present embodiments allow the disclosure of the present invention to be complete, and common knowledge in the art to which the present invention pertains. It is provided to completely inform the person having the scope of the invention, and the present invention is only defined by the scope of the claims.

The terminology used herein is for describing the embodiments, and is not intended to limit the present invention. In this specification, the singular form also includes the plural form unless otherwise specified in the phrase. As used herein, "comprises" and / or "comprising" refers to the components, steps, operations, and / or elements mentioned above of one or more other components, steps, operations, and / or elements. Presence or addition is not excluded.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings commonly understood by those skilled in the art to which the present invention pertains. In addition, commonly used terms defined in the dictionary are not ideally or excessively interpreted unless specifically defined.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The same reference numerals are used for the same components in the drawings, and duplicate descriptions for the same components are omitted.

It is difficult to capture spatially changing surface reflectometer information (SVBRDF) of a 3D object with only one handheld camera, for example, a typical smartphone or DSLR camera. Existing technologies are limited to planar geometry or rely on previously scanned 3D geometry, so their practicality is limited, and there are some technical challenges to overcome. The camera's built-in flash is positioned fixed at about the same position as the lens, which seriously interferes with the sampling process in the light-view space. Also, the near-field flash illuminates the object partially and non-uniformly. In terms of geometry, the conventional multi-view stereo technique only assumes diffuse reflection, so 3D reconstruction proceeds too smoothly.

Summary of the Invention Embodiments of the present invention are to measure shape information and surface reflectometer information (BRDF) of a 3D object that can be measured only by using special equipment using a camera equipped with a general flash.

That is, the present invention provides a simple yet powerful framework that eliminates the need for expensive dedicated hardware, so that the shape information and SVBRDF information of an actual 3D object can be measured simultaneously with a single camera with built-in flash.

The present invention can provide a high-quality 3D geometry reconstruction including more accurate high-frequency details than cutting-edge multi-view stereo technology by eliminating diffuse reflection assumptions and utilizing SVBRDF information.

The present invention can formulate co-reconstruction of SVBRDFs, shading normals, and 3D geometry into a multi-stage inverse-rendering pipeline, and can also be applied directly to existing multi-view 3D reconstruction techniques.

Here, co-reconstruction can be performed in a multi-step, eg, iterative and interactive optimization reverse-rendering pipeline that progressively enhances 3D to 2D correspondence, leading to high quality reconstruction of both SVBRDF and 3D geometry.

1 is a flowchart illustrating an operation of a 3D object acquisition method according to an embodiment of the present invention.

Referring to FIG. 1, a method for acquiring a 3D object according to an embodiment of the present invention receives a plurality of images of a 3D object photographed by a camera while artificial lighting is turned on (S110).

Here, step S110 may receive a plurality of images, for example, pictures, taken by two different exposures.

When a plurality of images photographed by artificial lighting is received in step S110, spatially changing surface reflectometer information (SVBRDF) for a 3D object is reconstructed based on the received plurality of images (S120).

Here, step S120 estimates the base surface reflectometer information (BRDF) for the 3D object based on the received plurality of images, and reconstructs the SVBRDF based on the estimated base surface reflectometer information and the spatially changing weight map. can do.

When the SVBRDF for the 3D object is reconstructed in step S120, the shadow normal for the 3D object is estimated based on the reconstructed SVBRDF, and the 3D geometry for the 3D object is obtained based on the estimated shadow normal (S130) , S140).

Here, step S140 may acquire a 3D geometry using a Poisson surface reconstruction technique.

When the 3D geometry is obtained in step S140, it is determined whether the error converges, and if the error is not converged, the update process is repeatedly performed to update the SVBRDF, the shadow normal, and the 3D geometry (S150, S160).

Although not shown in FIG. 1, the method according to an embodiment of the present invention acquires a 3D point cloud for a 3D object from a plurality of images and poses of a camera using multiview stereo, and the 3D point cloud Based on the Poisson surface reconstruction technique, the method further includes generating a base mesh for the 3D object, and subdividing the base mesh to obtain initial geometry for the 3D object, and step S120 is based on the plurality of images and the initial geometry. SVBRDF can be reconstructed.

The method according to the present invention will be described in detail with reference to FIGS. 2 to 5 as follows.

Table 1 below defines the variables used in the present invention.

x p	3D position of p-th vertex, p = 1,. . . , P
n p	geometric normal of x p
n to p	shading normal of x p
i p, k	light direction of x p at k-th camera
o p, k	view direction of x p at k-th camera
L (o p, k ; x p )	outgoing radiance from x p towards o p, k
L (-i p, k ; x p )	incoming radiance at x p from -i p, k
f (i, o; x, n)	BRDF at x
f b (i, o)	b-th basis BRDF, b = 1,. . . , B
ω p, b	blending weight of f b at x p
υ p, b	visibility of x p at I k
F b	set of basis BRDFs, {f b }
W	set of blending weights, {ω p, b }
N	set of shading normals, {
X	set of 3D points, {x p }

Figure 2 shows an exemplary view for explaining the outline of the three-dimensional object acquisition method according to the present invention.

As shown in Fig. 2, the method of the present invention uses an artificial lighting image photographed by a general camera, for example, a handheld camera.

At this time, the input can be composed of a set of K artificial light images I = {I _k } taken with two different exposures to expand the dynamic range, and considering the change in exposure, I _k is linear radiance (linear radiance) L _k .

Then, the present invention can be formulated by acquiring a set of surface reflectometer information (BRDF) F _b and SVBRDF information F defined by a corresponding weight map W, shaded normal N and 3D geometry X. First, the present invention acquires external parameters and rough base geometry of the camera using a commercial 3D reconstruction technique including SfM (Structural From Motion), MVS (Multiview Stereo), and mesh reconstruction in the initialization step. do. Then, the present invention reconstructs SVBRDF information simultaneously while improving the reconstructed 3D geometry using an iterative step. The iterative step begins with an inverse rendering step aimed at obtaining the first approximation for W, F _b and N.

The method according to the present invention reconstructs the SVBRDF information F, estimates the shadow normal N depending on the SVBRDF information, obtains W, F _b and N, and then inputs the deformation input of the shadow normal N through Poisson surface reconstruction. By updating the details of 3D geometry X using, it is possible to forcibly add photometric consistency during reconstruction.

Here, the present invention repeats the inverse-rendering process and the geometry reconstruction process until a predetermined error converges.

The present invention only needs to estimate the position of the flash relative to the camera and the camera optical parameters, and the position of the flash and the estimation of the camera optical parameters can be performed only once. Here, the present invention may use multiple chrome ball images and a checkerboard image.

Image formation model

The present invention uses a commercial camera with a built-in flash to capture a set of artificial lighting photos as input. Since the dynamic range of commercial cameras is insufficient to capture detailed specular reflections under flash lighting, the exposure time Δt is changed, for example, for a mobile phone with a fixed amount of flash light, the exposure time or flash intensity Δg is changed. Here, the flash intensity may correspond to the EV number of the DSLR camera. The image forming model for the pixel position u can be formulated as <Equation 1> below.

[Equation 1]

Here, I (u) means the captured image, and L (o; x) may mean the outgoing radiance emitted from the point x on the 3D geometry in the view direction o.

The radiance captured at the point x can be formulated with a reflection equation as shown in <Equation 2> below.

[Equation 2]

Here, f (i, o; x, n) means the reflection function at point x, n means the normal vector, and L (-i; x) at x for the light vector i It may mean the incident light (incident light).

Responsiveness  Finding correspondences

In a handheld setup, information about multiple exposures is stored in 3D points x units, not in 2D pixel u units. Therefore, the present invention needs to obtain a geometric relationship between x and u.

First, the present invention acquires internal parameters of the camera using a method capable of acquiring internal parameters. Here, the method for acquiring the internal parameter may be various methods, and such a method is obvious to those skilled in the art, and detailed description thereof is omitted.

In the step of obtaining the internal parameters of the camera, the initial relationship between the camera ² and the pixel u∈R In response to the captured surface x∈R point ³ is calculated in a perspective (perspective) projection matrix π∈R ^{3 × 3.} Rotation / translation transformation matrix [R | t] ∈R which defines the external relationship between the surface point and camera for each photo using SfM because the photo is captured without any supporting structure Obtain ^{3 × 4} . In addition, the present invention can gradually update internal parameters such as a focal length of each picture using SfM for more accurate geometric response by correcting a focus breathing effect due to autofocus. . The resulting internal and external parameters define the geometric relationship between pixel u and point x in Equation 1, as [u, 1] ^┬ = π [R | t] [x, 1] ^┬ .

Early Geometry (initial geometry)

The present invention acquires a high density 3D point cloud using MVS from captured images and camera poses. However, this initial point cloud generally violates the basic diffusion texture assumptions of the MVS method due to the high frequency noise caused by the specular reflection generated by flash lighting. Thus, the present invention uses a Poisson surface reconstruction screened to reduce this noise to produce a low resolution mesh, eg 2 ⁷ voxel grid mesh. Then, the present invention subdivides the low-resolution mesh to obtain a finer mesh, such as 2 ¹⁰ grid mesh, used as the initial geometry. Despite the high resolution of the initial geometry, fine geometrical details may be missing because it is removed along with the noise in the Poisson reconstruction step. Thus, the present invention recovers these details using an iterative geometry update algorithm.

SPATIALLY-VARYING REFLECTANCE AND NORMALS

The present invention acquires SVBRDF (W, F _b ) and normal information (N) from an input photo. Given a set of P surface (vertex) points X = {x _p } captured from different light / view directions i _k and o _k of the K photos, the captured radiance is L = {L (o _k ; x _p )}. The present invention satisfies the image forming model of Equation 2 by finding a set of two unknowns {f (i _{p, k} , o _{p, k} ; x _p , n _p ), n _p } that minimizes the objective function. The inverse-rendering problem can be formulated as <Equation 3> below.

[Equation 3]

Here, υ _{p, k} may mean the visibility function of the vertex x _p of the image k.

The inverse rendering problem of factoring reflections and shading is a very ill-posed and underdetermined problem. The present invention uses an iterative optimization method that updates four unknown elements W, F _b , N and X until the rendering result satisfies the input image. In addition, in order to prevent overfitting, the present invention separates the input picture into different training and test data sets to test optimized parameters with unused data sets. First, the present invention reconstructs the entire space of the SVBRDF, and obtains a normal using this information.

SVBRDF  (F) Reconstructing the SVBRDF )

In order to obtain SVBRDF F, the present invention estimates the set F _b = {f _b } of the base BRDF, and blends the spatially varying weight map W and the estimated set of the base BRDF. At this time, the reconstructed SVBRDF F can be formulated as <Equation 4> below.

[Equation 4]

Here, W = {w _{p , b} } may mean a set of per-point blending weights.

Flash pictures set up (Flash photography setup)

To obtain the BRDF, high-density sampling of θ _h , θ _d and φ _d is required as shown in FIG. 3A. This sampling is usually done using additional supporting hardware. The Fresnel effect strongly depends on θ _d = cos ^-1 (h · i), while the specular reflection changes rapidly as a function of θ _h = cos ^-1 (h · n). Here, h may mean a halfway vector.

On the other hand, the reflection is kept almost constant along the azimuth angle φ _d of the light i around h. The data set captured in the setup of the present invention includes high density sampling along the θ _h and φ _d dimensions. However, as shown in FIG. 3B, high-density sampling leads to a single sample for θ _d at about ˜5 ° because the light and camera are fixed and very close in the setup of the present invention. Here, the angle θ _h = 5 ° is an optimal angle for single shot capture.

Reflectance model

The present invention uses a Cook-Torrance (CT) model with a non-parametric normal distribution function (NDF) term for better representation of specular reflection. As described above, the present invention does not rely on an analytical function such as Beckmann that performs high-density sampling along θ _h and φ _d angles. The base reflection model f _b of the present invention can be expressed as <Equation 5> below.

[Equation 5]

Here, ρ _d and ρ _s denote diffusion and reflection albedo, D denotes a univariate NDF term for specularity, G denotes a geometric term, and F denotes Fresnel Term.

NDF is represented by the non-parametric table function D (θ _h ) ∈R ^M. In the present invention, by setting M = 90, each element stores the BRDF value in the corresponding θ _h using the square root mapping of the angle. The present invention can use a V-groove cavity model for shading / masking term G. Here, since the present invention cannot observe the Fresnel effect in the setup, the F term can be set as a constant. In practice, this setup helps reduce complexity in the optimization process. At this time, this approximation may be more effective than using a constant index of refraction (IOR) value for the Fresnel term.

SVBRDFs  Reconstructing SVBRDFs )

In the present invention, the base BRDF in Equation 5 can be expressed as a coefficient vector f _b = [ρ _d , ρ _s FD (θ _h )] ^┬ ∈R ^{M + 1} . The geometric factor G is excluded from the coefficient vector. The invention converts the reflection f _{^'p, k} captured the captured radiance from the equation (2). A measurement vector Φ _{p, k} ∈R ^{M + 1} specifying the sampled θ _h angle and the observed geometrical factor per pixel G / (4 (n · i) (n · o)) and the captured reflection f ^' _{p , k} given, the present invention obtains f _{^'p, k} = Φ _p, f _{k p.} The present invention

The base BRDFs f _b and the spatial weights w _{p , b} can be mixed to approximate. Then, the present invention can reform Equation 3 as an objective function as shown in <Equation 6> below to reconstruct base BRDFs and corresponding weights.

[Equation 6]

In order to separate diffusion and specular reflection more stably, the present invention can fix D (θ _h > 60) to 0 following conventional distribution functions such as Beckmann and GGX. In addition, the present invention can apply a non-negative monotonicity constraint to D (θ _h ). For example, D (θ _h ) may decrease monotonically as θ _h increases. In order to more accurately reproduce strong specular peaks, no softness limits are placed. In order to update F _b , Equation 6 is minimized while W is fixed. This becomes a quadratic programming problem in F _b with sparse input data. Here, the present invention may use a commercial rare secondary programming solver.

W Reconstructing W

First, the present invention estimates the set of diffusion components of the base BRDF F _b by averaging color observations around median brightness per vertex clustering using Kmean in CIELAB space. This produces an initial binary labeled weight set W. The underlying BRDFs are still a problem to be solved. As an example, an ad-hoc method is used to gradually determine the number of bases and increase it until the optimization converges. Here, the number of bases may be set using previous techniques, and the number of bases may vary depending on the situation.

In the next iteration, the present invention updates W using F _b estimated in the previous optimization. Updating W by using the fixed F _b of Equation 6 is the same as minimizing the objective function as shown in Equation 7 below at each point x _p .

[Equation 7]

The kth row of Q∈R ^{K + B} is [f ₁ (i _{p, k} , o _{p, k} ), ..., f _B (i _{p, k} , o _{p, k} )], r∈R ^K The kth element of is L (o _{p, k} ; x _p ) / L (-i _{p, k} ; x _p ) / (n _p · i _{p, k} ). Here, the present invention applies visibility υ _{p, k} and additional weights w _{p , k} = cos (θ _i ) sin (θ _h ) to each element of Q and r. Here, θ _i = cos ^-1 (n _p · i _{p, k} ) may be meant, and θ _h = cos ^-1 (n _p · h _{p, k} ) may be referred to. The cosine term compensates for unstable observations at grazing angles, but the sine term prevents bias to reflection observations. Minimization of Equation 7 is a standard secondary programming problem. Here, the present invention can solve this problem by using the convex quadratic programming solver.

Color

The present invention uses color based BRDF and monochromatic mixing weights. The present invention can independently optimize each color channel when updating F _b , and optimize the mixing weight of each vertex (vertex) using all color channels when updating W.

Reconstructing Normals

The present invention calculates SVBRDFs F by acquiring base BRDFs F _b = {f _b } and mixing weights W = {w _{p , b} }. The present invention should estimate the set of shaded normals N = {n ^- _p } per vertex. Predicting surface normals using BRDFs is a very serious problem, so the present invention applies an iterative optimization technique. First, the present invention feeds the initial surface normal n _p from the current geometry updated in the previous iteration as an input variable for BRDF f at point x _p in Equation 3 above. Since the incoming / outgoing radiance, the incoming and outgoing directions, and the reflection of the vertices are all known, we can factor the shading normals n ^to _p using standard linear least squares regression without any constraints. Geometric normal line at point x _p n _p is different from the normal shade n ^~ _p. The difference between n _p and x _p gradually converges while optimizing. Since the observation at the grazing angle and the mirror-reflection angle is not reliable, the present invention can introduce additional weights taking into account the illumination angle, such as w _p = cos (θ _i ) sin (θ _h ). Here, θ _i = cos ^{- 1} (n · i) may be used, and the sine term may prevent potential bias for misaligned surface normals and specular reflection of previous estimates.

Using metering consistency Geometry update

The present invention estimates the weight W, the base BRDFs F _b and the shadow normal N, and then reconstructs the geometry X to match the shadow observation. 4 illustrates an exemplary diagram for explaining a process of updating a 3D geometry. As illustrated in FIG. 4, the process of updating the 3D geometry is obtained by obtaining a rough base geometry (or base mesh) from the initial point cloud and subdividing it into a finer mesh. Then, the present invention estimates the shadow normal using the subdivided mesh, and then updates the geometry using the estimated shadow normal. Here, the geometry update may be performed using at least one of a Nehab method and a screened Poisson reconstruction method. Both methods can preserve low and high frequency details, but Nehab's method transmits the surface gradient directly to the target geometry, while the Poisson method uses coarse-to-fine reconstruction. Shading normal N can be assumed to contain high frequency noise, and the direct transmission in Nehab's method carries this noise, so this is an important difference between unstructured capture setups. Accordingly, the present invention preferably uses a screened Poisson method designed to reconstruct an implicit surface using a three-dimensional tri-quadratic B-spline basis function in a coarse-to-fine method voxel lattice. This results in robust performance when integrating noise-containing surface normals into 3D geometry.

FIG. 5 shows an exemplary diagram for a comparison result of the Nehab method and the screened Poisson reconstruction method for the unstructured capture setup of the present invention. As shown in FIG. 5, the result of the Nehab method is the unstructured capture setup While it contains the high frequency noise of the shadow normal estimated at, it can be seen that the geometry of the screened Poisson reconstruction method robustly processes the input noise through coarse-to-fine reconstruction, thereby providing more detailed details.

The screened Poisson reconstruction method implicitly surfaces from the input point cloud.

Can be reconstructed as <Equation 8> below.

[Equation 8]

Here, V: R ^3- > R ³ means a vector field derived from a set of shaded normals N, ▽ χ (x _p ) means the gradient of the implicit scalar surface function, χ: R ^3- > R, χ ² (x _p ) means the squared distance between the point x _p and the implicit surface χ, and α can mean the weight of the normalization term.

The present invention determines α∈ [0.1,4.0] according to the reliability of the initial geometry. To discretize the implicit surface function χ, the resolution of the voxel grating can be set to 2 ⁹ or 2 ¹⁰ for each dimension. This size corresponds to approximately 0.1 to 0.2 mm for the captured physical object.

The original algorithm uses geometric normals, but the present invention utilizes shaded normals n ^to _p to implicit surfaces whose gradients match n ^to _p.

Aim to seek. In other words, if different views and lighting directions are given for each vertex, a consistent shadow should appear. Once the implicit function is determined, the present invention converts the implicit surface to a polygonal mesh by applying marching cubes.

The present invention repeatedly updates W, F _b , N, and X until it finds the optimal 3D geometry and SVBRDF. The present invention evaluates the Hausdorff distance between the old mesh and the new X. The entire process shown in FIG. 2 is repeated until the test RMS error of the photometric difference of Equation 3 of the present invention starts to increase. In addition, to prevent overfitting, the present invention randomly separates captured images into training and test groups at a 9: 1 ratio.

As described above, the method for acquiring a 3D object according to an embodiment of the present invention uses the general flash-mounted camera to measure shape information of a 3D object and spatially changing surface reflectometer information (SVBRDF), which could only be measured using special equipment. ) Can be measured simultaneously.

In addition, the 3D object acquisition method according to an embodiment of the present invention can measure high quality 3D shape information and SVBRDF without relying on special equipment.

That is, the method for acquiring a 3D object according to an embodiment of the present invention requires only a single commercial camera with a built-in flash, for example, a smartphone camera, because multiple artificial lighting photos are used as input.

As described above, the core of the 3D object acquisition method according to an embodiment of the present invention is a new co-reconstruction of SVBRDF, shading normals, and 3D geometry, and the co-reconfiguration is a multi-step e.g. iteratively enhancing 3D to 2D correspondence And can be performed in an interactive optimization inverse-rendering pipeline to induce high-quality reconstruction of both SVBRDF and 3D geometry.

Furthermore, the method according to embodiments of the present invention reconstructs spatially changing surface reflectometer information for a 3D object based on a plurality of images photographed by a camera, and thus reconstructed spatially changing surface reflection Acquiring 3D geometry for a 3D object based on system information, and estimating the shading normal for a 3D object using the reconstructed spatially changing surface reflectometer information, and then based on the estimated shading normal It is not limited to obtaining 3D geometry for a 3D object. That is, the method according to another embodiment of the present invention can be applied to a variety of methods for acquiring 3D geometry for a 3D object based on spatially changing surface reflectometer information. Accordingly, a method according to another embodiment of the present invention reconstructs spatially changing surface reflectometer information for a 3D object based on a plurality of images photographed by a camera, and reconstructed spatially changing surface reflectometer information Based on the information, it is possible to obtain 3D geometry for a 3D object.

Furthermore, in the method according to the embodiments of the present invention described above, the technical configuration of reconstructing spatially changing surface reflectometer information for a 3D object is not limited to being reconstructed only by a plurality of images photographed by a camera. In addition, additional depth information, that is, depth information for a 3D object may be reflected to improve the weight and speed of the algorithm. For example, the method according to embodiments of the present invention can reconstruct spatially changing surface reflectometer information for a 3D object based on depth information for a 3D object as well as a plurality of images photographed by a camera. have.

Here, the depth information for the 3D object may be obtained separately using a depth camera, or may be obtained using various measurement techniques. For example, the present invention uses a multi-view camera to depth a 3D object. Information may be obtained, or depth information on a 3D object may be obtained using stereo matching. Of course, the method for obtaining depth information in the present invention is not limited to the above-described method, and any method applicable to the present invention can be used.

A method of reconstructing spatially changing surface reflectometer information for a 3D object by additionally using the depth information may be applied to all of the above-described contents.

6 illustrates a configuration of a 3D object acquisition device according to an embodiment of the present invention, and shows a conceptual configuration of a device performing the method of FIGS. 1 to 5.

Referring to FIG. 6, a 3D object acquisition apparatus 600 according to an embodiment of the present invention includes a reception unit 610, a reconstruction unit 620, an estimation unit 630, and an acquisition unit 640.

The reception unit 610 receives a plurality of images of a 3D object photographed by a camera.

At this time, the receiving unit 610 may receive a plurality of images of the 3D object photographed with artificial lighting turned on.

The reconstruction unit 620 reconstructs spatially changing surface reflectometer information for a 3D object based on the received plurality of images.

At this time, the reconstruction unit 620 estimates the base surface reflectometer information for the 3D object based on the plurality of images, and spatially changes based on the estimated base surface reflectometer information and the spatially changing weight map. The surface reflectometer information can be reconstructed.

At this time, the reconstruction unit 620 acquires the base geometry of the 3D object based on the camera's external parameters and a plurality of images, and reconstructs spatially changing surface reflectometer information based on the plurality of images and the base geometry. You can.

At this time, the reconstruction unit 620 may reconstruct spatially changing surface reflectometer information using an inverse rendering technique that satisfies a preset image forming model for the camera.

Furthermore, the reconstruction unit 620 may update spatially changing surface reflectometer information based on the 3D geometry and a plurality of images acquired by the acquisition unit 640.

Furthermore, the reconstruction unit 620 acquires a 3D point cloud for a 3D object from a pose of a plurality of images and cameras using multiview stereo, and 3D based on a 3D point cloud and a Poisson surface reconstruction technique It is possible to generate a base mesh for an object, to subdivide the base mesh to obtain initial geometry for a 3D object, and reconstruct spatially changing surface reflectometer information based on a plurality of images and the initial geometry.

The estimator 630 estimates the shadow normal for the 3D object based on the reconstructed spatially changing surface reflectometer information.

At this time, the estimator 630 may estimate the shadow normal using an inverse rendering technique that satisfies a preset image forming model for the camera.

Furthermore, when the spatially changing surface reflectometer information is updated, the estimator 630 may update the shading normal based on the updated spatially changing surface reflectometer information.

The acquisition unit 640 acquires 3D geometry based on the estimated shadow normal and Poisson surface reconstruction technique.

At this time, the acquisition unit 640 may acquire a 3D geometry using a Poisson surface reconstruction technique.

Furthermore, when the shadow normal is updated, the acquirer 640 may update the 3D geometry based on the updated shadow normal.

Although the description is omitted in the apparatus of FIG. 6, each of the constituent elements constituting FIG. 6 may include all the contents described in FIGS. 1 to 5, which is apparent to those skilled in the art.

The device described above may be implemented with hardware components, software components, and / or combinations of hardware components and software components. For example, the devices and components described in the embodiments include, for example, a processor, controller, arithmetic logic unit (ALU), digital signal processor (micro signal processor), microcomputer, field programmable array (FPA), It may be implemented using one or more general purpose computers or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may run an operating system (OS) and one or more software applications running on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For convenience of understanding, a processing device may be described as one being used, but a person having ordinary skill in the art, the processing device may include a plurality of processing elements and / or a plurality of types of processing elements. It can be seen that may include. For example, the processing device may include a plurality of processors or a processor and a controller. In addition, other processing configurations, such as parallel processors, are possible.

The software may include a computer program, code, instruction, or a combination of one or more of these, and configure the processing device to operate as desired, or process independently or collectively You can command the device. Software and / or data may be interpreted by a processing device, or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. Can be embodied in The software may be distributed on networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer-readable medium may include program instructions, data files, data structures, or the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiments or may be known and usable by those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs, DVDs, and magnetic media such as floptical disks. -Hardware devices specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language code that can be executed by a computer using an interpreter, etc., as well as machine language codes produced by a compiler.

As described above, although the embodiments have been described by a limited embodiment and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques are performed in a different order than the described method, and / or the components of the described system, structure, device, circuit, etc. are combined or combined in a different form from the described method, or other components Alternatively, even if replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (20)

1. Receiving a plurality of images of a 3D object photographed by a camera;

   Reconstructing spatially changing surface reflectometer information for the 3D object based on the received plurality of images;

   Estimating a shadow normal for the 3D object based on the reconstructed spatially changing surface reflectometer information; And

   Obtaining 3D geometry for the 3D object based on the estimated shading normal

   3D object acquisition method comprising a.
2. According to claim 1,

   The reconstructing step

   Estimating base surface reflectometer information for the 3D object based on the received plurality of images, and spatially changing surface reflectometer based on the estimated base surface reflectometer information and a spatially changing weight map 3D object acquisition method characterized by reconstructing information.
3. According to claim 1,

   Obtaining the base geometry of the 3D object based on the camera's external parameters and the plurality of images.

   Further comprising,

   The reconstructing step

   And reconstructing the spatially varying surface reflectometer information based on the plurality of images and the underlying geometry.
4. According to claim 1,

   The reconstructing step and the estimating step are

   A method for acquiring a 3D object, characterized by reconstructing the spatially changing surface reflectometer information and estimating the shading normal using an inverse rendering technique that satisfies a preset image forming model for the camera. .
5. According to claim 1,

   The obtaining step

   A method for acquiring a 3D object, characterized by acquiring the 3D geometry using a Poisson surface reconstruction technique.
6. According to claim 1,

   The reconstructing step

   Update the spatially changing surface reflectometer information based on the obtained 3D geometry and the plurality of images,

   The estimating step

   Update the shading normal based on the updated spatially changing surface reflectometer information,

   The obtaining step

   And updating the 3D geometry based on the updated shading normal.
7. According to claim 1,

   The receiving step

   A method for acquiring a 3D object, characterized by receiving a plurality of images of the 3D object photographed while artificial lighting is on.
8. According to claim 1,

   Obtaining a 3D point cloud for the 3D object from the poses of the plurality of images and the camera using multiview stereo;

   Generating a base mesh for the 3D object based on the 3D point cloud and Poisson surface reconstruction technique; And

   Subdividing the base mesh to obtain initial geometry for the 3D object

   Further comprising,

   The reconstructing step

   And reconstructing the spatially varying surface reflectometer information based on the plurality of images and the initial geometry.
9. Receiving a plurality of images of a 3D object photographed by a camera while artificial lighting is on; And

   Obtaining spatially changing surface reflectometer information and 3D geometry for the 3D object based on the received plurality of images and Poisson surface reconstruction technique

   3D object acquisition method comprising a.
10. The method of claim 9,

    The obtaining step

    Reconstructing the spatially changing surface reflectometer information based on the received plurality of images;

    Estimating a shadow normal for the 3D object based on the reconstructed spatially changing surface reflectometer information; And

    Obtaining the three-dimensional geometry based on the estimated shadow normal and the Poisson surface reconstruction technique

    3D object acquisition method comprising a.
11. A receiver configured to receive a plurality of images of a 3D object photographed by a camera;

    A reconstruction unit reconstructing spatially changing surface reflectometer information for the 3D object based on the received plurality of images;

    An estimator for estimating a shadow normal for the 3D object based on the reconstructed spatially changing surface reflectometer information; And

    An acquiring unit acquiring 3D geometry for the 3D object based on the estimated shadow normal

    3D object acquisition device comprising a.
12. The method of claim 11,

    The reconstruction unit

    Estimating base surface reflectometer information for the 3D object based on the received plurality of images, and spatially changing surface reflectometer based on the estimated base surface reflectometer information and a spatially changing weight map 3D object acquisition device, characterized in that to reconstruct the information.
13. The method of claim 11,

    The reconstruction unit

    Based on the external parameters of the camera and the plurality of images, the basis geometry of the 3D object is obtained, and the spatially changing surface reflectometer information is reconstructed based on the plurality of images and the basis geometry. 3D object acquisition device.
14. The method of claim 11,

    The reconstruction unit and the estimation unit

    3D object acquiring apparatus characterized by reconstructing the spatially changing surface reflectometer information and estimating the shading normal using an inverse rendering technique that satisfies a preset image forming model for the camera.
15. The method of claim 11,

    The acquisition unit

    A device for acquiring 3D objects, characterized by acquiring the 3D geometry using a Poisson surface reconstruction technique.
16. The method of claim 11,

    The reconstruction unit

    Update the spatially changing surface reflectometer information based on the obtained 3D geometry and the plurality of images,

    The estimation unit

    Update the shading normal based on the updated spatially changing surface reflectometer information,

    The acquisition unit

    And updating the 3D geometry based on the updated shadow normal.
17. The method of claim 11,

    The receiving unit

    A device for obtaining a 3D object, characterized by receiving a plurality of images of the 3D object taken while artificial lighting is on.
18. The method of claim 11,

    The reconstruction unit

    Multi-view stereo is used to obtain a three-dimensional point cloud for the three-dimensional object from the plurality of images and the pose of the camera, and a basis for the three-dimensional object based on the three-dimensional point cloud and Poisson surface reconstruction technique Generating a mesh, subdividing the base mesh to obtain initial geometry for the 3D object, and reconstructing the spatially changing surface reflectometer information based on the plurality of images and the initial geometry. 3D object acquisition device.
19. Receiving a plurality of images of a 3D object photographed by a camera;

    Reconstructing spatially changing surface reflectometer information for the 3D object based on the received plurality of images; And

    Obtaining 3D geometry for the 3D object based on the reconstructed spatially changing surface reflectometer information

    3D object acquisition method comprising a.
20. The method of claim 19,

    The reconstructing step

    And reconstructing spatially changing surface reflectometer information for the 3D object based on the received plurality of images and depth information for the 3D object.

Images