JP2023032793A - Texture acquisition system, texture acquisition device and texture acquisition program - Google Patents

Abstract

To acquire detailed information of a three-dimensional model necessary for photorealistic CG rendering capable of expressing the texture of the surface of a subject in real time.SOLUTION: A texture acquisition device 1 of a texture acquisition system 1000 comprises: a subject shape acquisition unit 120 that acquires shape information of a subject from a depth measurement device 20; a reflection component separation unit 141 that separates reflected light from the subject irradiated with polarized light of multiple wavelengths into a diffuse reflection component and a specular reflection component; a diffuse reflection calculator 142 that calculates a diffuse reflection coefficient and normal information using the diffuse reflection component; a specular reflection calculation unit that calculates the roughness and reflectance of the surface of the object using a predetermined specular reflection model, and calculates a refractive index using the Fresnel equation; and a three-dimensional model detailed information output unit 150 that outputs the calculated information as three-dimensional model detailed information.SELECTED DRAWING: Figure 1

Description

The present invention relates to a texture acquisition system, a texture acquisition device, and a texture acquisition program for acquiring shape, texture, and the like as object information, which is material data handled in computer graphics (CG).

With the recent spread of AR (Augmented Reality) / VR (Virtual Reality) technology, there is a need to convert real-world subjects into subject information (hereinafter sometimes referred to as "three-dimensional models") that can be handled by CG technology. rising. This is a technique generally called volumetric capture, in which the surface pattern and shape of the entire subject are acquired by cameras and depth measuring devices arranged to surround the subject (see Non-Patent Documents 1 and 2).

With conventional volumetric capture, especially when it is necessary to change the lighting conditions after shooting, it is possible to create a uniform image on the surface of the subject by taking measures such as arranging the subject under a flat lighting environment. The subject is illuminated and shot without shadows. Since it is not possible to directly measure the reflectance of the subject, the amount of light incident on the surface of the subject is made uniform and the lighting effect is non-directional. In this method, the amount of light incident on each part of the image is normalized, and the luminance on the obtained photographed image can be treated as a pseudo diffuse reflection coefficient. As a result, pseudo shading is added at the time of rendering to express the appearance of the three-dimensional model as expected under the lighting conditions in the virtual space.

O. Schreer, et al., “Capture and 3D Video Processing of Volumetric Video,” 2019 IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan, 2019, pp. 4310-4314. O. Schreer, et al., “ADVANCED VOLUMETRIC CAPTURE AND PROCESSING,” [online], IBC2018, [searched July 1, 2021], Internet <URL: https://www.ibc.org/download? ac=6559＞

However, in the method using the conventional volumetric capture technology, the diffuse reflected light component of the reflected light is obtained in a pseudo manner. Also, the specular reflection light component is not handled, or the specular reflection coefficient is manually set uniformly over the entire subject or for each part of the subject. When assigning coefficients (specular reflection coefficients) manually, when the subject is non-rigid and changes over time, assigning coefficients to each part of the subject takes a lot of effort, so it is not realistic. is difficult, and it is even more difficult to adapt to live video.

Further, when observing the surface of an object microscopically, the surface of many objects has fine unevenness, and therefore the appearance of the object is greatly different when illuminated. The unevenness of the surface of the subject is a factor that greatly affects the appearance, such as gloss and texture, that is, the impression of texture. Therefore, in order to create this texture, it is necessary to describe the behavior of the specular reflection in units of parts of the subject or in units smaller than the pixel unit of the imaged pixels, but this cannot be expressed at present.

In an AR application, when using the obtained subject information as content, rendering is performed using CG technology based on the subject information and superimposed on the real space using AR glasses or the like. At that time, if the lighting conditions do not match the real space, the image will give a sense of discomfort. Furthermore, while lighting in real space has complex spectral characteristics, the surface pattern of a general CG model is represented by RGB three-color spectrum, which makes matching difficult and unnatural. may result in a composite image.

These factors make it difficult to match the environment of each object when a plurality of pieces of object information are arranged and rendered simultaneously in one scene in AR/VR. As a result, not only discomfort due to the environment mismatch between the real space and the virtual space, as in AR, but also discomfort occurs between subjects in the virtual space, which adversely affects the sense of immersion and presence.

The present invention has been made in view of the above points, and enables real-time acquisition of detailed information of a three-dimensional model that is necessary for photorealistic CG rendering that can express the texture of the surface of an object. An object of the present invention is to provide a texture acquisition system, a texture acquisition device, and a texture acquisition program.

In order to solve the above-mentioned problems, the texture acquisition system of the present invention includes: a photographing booth having a depth measuring device, a polarized lighting device, and a polarized photographing device; a texture acquisition device that is communicatively connected to the photographing device and that acquires the texture indicated by the unevenness of the surface of the object.
Also, the texture acquisition device includes a storage unit, a subject shape acquisition unit, a measurement control unit, a reflection component separation unit, a diffuse reflection calculation unit, a specular reflection calculation unit, and a three-dimensional model detailed information output unit. It was configured.

According to this configuration, the texture acquisition device acquires the shape information of the subject by receiving the depth information measured by the depth measurement device by the subject shape acquisition unit, and the measurement control unit detects the deflection of the plurality of wavelengths. The polarized lighting device controls the irradiation of the light onto the subject, and the reflection component separation unit acquires the reflected light from the subject irradiated with the polarized light of multiple wavelengths as information captured by the polarized imaging device, and diffuses the reflected light. It can be separated into reflective and specular components.
As a result, the texture acquisition device of the texture acquisition system can separate the reflected light of each of the polarized lights of a plurality of wavelengths irradiated onto the subject into diffuse reflection components and specular reflection components.

In addition, the texture acquisition device uses the diffuse reflection calculation unit to irradiate the object with polarized light of a plurality of wavelengths from the polarized lighting devices arranged in three directions, and the reflected light captured by the polarization imaging device is analyzed by the reflected light component separating unit. Lambert's cosine law can be applied to the diffuse reflection component separated by and the illumination position and illumination intensity of the polarized illumination device to calculate the diffuse reflection coefficient and normal information of the object.
As a result, the texture acquisition device of the texture acquisition system can calculate the diffuse reflection coefficient and normal line information of the subject using only the diffuse reflection component excluding the specular reflection component. Diffuse reflection coefficient and normal information can be obtained accurately.

In addition, the texture acquisition device includes the specular reflection components separated by the reflection component separation unit for each of the polarized lights of a plurality of wavelengths, the illumination position of the polarization illumination device, the camera position of the polarization imaging device, and the calculated The obtained normal information is applied to a predetermined specular reflection model, and the angle at which the normal distribution function obtained by the specular reflection model has a half value of the maximum value is calculated as a parameter of the surface roughness of the subject, Applying the illumination position of the polarizing illumination device, the camera position of the polarizing imaging device, the normal information, and the normal distribution function to a predetermined specular reflection model, the reflectance of the object for each of the p-wave and s-wave polarized light of multiple wavelengths is calculated. is calculated, and the calculated reflectance is applied to Fresnel's equation regarding reflection in media having different refractive indices, whereby the refractive index of the object can be calculated.
As a result, the texture acquisition device of the texture acquisition system can calculate the roughness as a parameter that indicates the detailed normal distribution of the surface of the object, and also calculate the reflectance of the object for each of the p-wave and s-wave for each wavelength, and can be calculated.

In addition, the texture acquisition device outputs information including any one of the diffuse reflection coefficient, normal line information, surface roughness of the subject, reflectance, and refractive index calculated by the three-dimensional model detailed information output unit. In order to calculate the information, it is possible to output as three-dimensional model detailed information in accordance with the time when the polarization photographing device takes the photograph.
As a result, the texture acquisition device of the texture acquisition system can output more detailed information of the three-dimensional model to, for example, an external device in real time.

According to the present invention, there is provided a texture acquisition system, a texture acquisition device, and a texture acquisition program for acquiring detailed information of a three-dimensional model necessary for photorealistic CG rendering capable of expressing the texture of the surface of an object in real time. can be done.

1 is a functional block diagram showing the overall configuration of a texture acquisition system including a texture acquisition device according to this embodiment; FIG. FIG. 4 is an explanatory diagram relating to texture acquisition processing by the texture acquisition device of the present embodiment; It is explanatory drawing which shows the example which comprises a photography booth of this embodiment by a regular octahedron. It is a figure which shows an example of the illumination pattern in the polarization|polarized-light illumination device which concerns on this embodiment. It is an explanatory view showing the RGB Bayer arrangement of the camera imaging plate of the polarization imaging device according to the present embodiment. FIG. 4 is an explanatory diagram for separating a diffuse reflection component and a specular reflection component of reflected light according to the embodiment; FIG. 3 is an explanatory diagram relating to Lambert's cosine law in the present embodiment; FIG. 4 is an explanatory diagram relating to calculation of a diffuse reflection coefficient and normal information based on Lambert's cosine law in the present embodiment; FIG. 4 is an explanatory diagram relating to calculation of a diffuse reflection coefficient and normal information based on Lambert's cosine law in the present embodiment; FIG. 4 is an explanatory diagram of a reflection model regarding specular reflection in this embodiment; FIG. 4 is an explanatory diagram relating to reflection and refraction on the surface of an object in this embodiment; It is a figure which shows an example which comprises a photography booth of this embodiment by a multispectral illuminator and a multispectral camera. It is a figure which shows an example which comprises a photography booth of this embodiment by a multispectral illuminator and a multispectral camera. FIG. 11 is an explanatory diagram relating to a modification of the three-dimensional model detailed information output unit according to the present embodiment; 5 is a flow chart showing the flow of processing of the texture acquisition device according to the present embodiment; FIG. 5 is a diagram showing calculation results of a diffuse reflection coefficient and normal vector information using only a diffuse reflection component in the embodiment;

EMBODIMENT OF THE INVENTION Hereafter, the form (it is called "embodiment" hereafter) for implementing this invention is demonstrated with reference to drawings.
First, with reference to FIG. 1, an outline of a texture acquisition system 1000 including the texture acquisition device 1 will be described first, and then each configuration will be described.

As shown in FIG. 1, the texture acquisition system 1000 includes a depth measuring device 20, a polarized lighting device 30, and a polarized lighting device 40, and the depth measuring device 20 and the polarized lighting device, which are arranged in a photography booth BH for photographing a subject. 30 and a texture acquisition device 1 that is communicatively connected to the polarizing imaging device 40 . The texture acquisition device 1 is connected for communication with one or more external devices 5 via a network or the like.

The texture acquisition device 1 according to the present embodiment is useful as a method for measuring subject information (three-dimensional model) used in AR/VR and capturing subject information in three-dimensional television, and acquires the texture of a subject. The processing (hereinafter referred to as "texture acquisition processing") includes (1) processing for separating diffuse reflection components and specular reflection components, (2) processing for calculating diffuse reflection coefficients and normal information using the diffuse reflection components, (3) Calculation processing of the “roughness” of the surface of the subject and the specular reflection coefficient (calculated as “refractive index of the subject” in this embodiment) using the specular reflection component (hereinafter “roughness and refractive index calculation processing” ), is realized.

(1) In the process of separating the diffuse reflection component and the specular reflection component, the texture acquisition device 1, as shown in FIG. The subject S is irradiated with polarized light (illumination light) by the green (B) illumination and the blue (B) illumination. Then, the reflected light from the irradiated object S is photographed by a polarizing photographing device 40 that selectively transmits the polarized light of 0, 45, 90, and 135 degrees (see FIGS. 5 and 6 to be described later). ). The photographed image P is a combination of the specular reflection component I _s that appears as a highlight that looks like the light source is reflected, and the diffuse reflection component I _d that is diffused by the incident light entering the subject that absorbs the color components of the object. becomes an image. The texture acquisition device 1 separates the diffuse reflection component I _d and the specular reflection component I _s by using the difference in the reflection behavior of the polarized light on the object surface (details will be described later).

(2) In the diffuse reflection coefficient and normal information calculation process, the texture acquisition apparatus 1 uses the separated diffuse reflection component I _d to obtain the normal information N (normal vector N) of the surface of the object for each wavelength and , diffuse reflection coefficient K _d (see FIG. 2). In this embodiment, for each wavelength (for example, red, green, and blue), reflected light from three directions is observed using a method of photometric stereo, and the surface normal vector of the object surface and the diffuse reflection coefficient K _d and are calculated (details will be described later).

(3) In the roughness and refractive index calculation process, the texture acquisition apparatus 1 uses the separated specular reflection component I _s to calculate the surface roughness γ of the subject and the refractive index n _j of the subject. The texture acquisition device 1 can also obtain the specular reflection coefficient K _s as a single parameter that does not consider wavelength dependence. However, when Fresnel reflection is taken into account, the specular reflection coefficient K _s depends on the angle of incidence and, strictly speaking, behaves differently for p-waves and s-waves. As information, the refractive index n _j of the subject for each wavelength band is calculated. In addition, Fresnel reflection and wavelength dependence can be obtained by obtaining Ro (reflection coefficient of light incident parallel to the normal) of the Fresnel coefficient approximation formula proposed by Schlick et al. , can be obtained as a coefficient considering the polarization.

In this way, the texture acquisition apparatus 1 can obtain accurate diffuse reflection coefficient K _d and normal information on the surface of the subject, parameters indicating the state of unevenness of the surface of the subject, which cannot be obtained by a method using conventional volumetric capture technology. and the refractive index n _j of the subject. Then, the texture acquisition device 1 acquires detailed information of the three-dimensional model thus acquired (“three-dimensional model detailed information” to be described later), and the performance of each external device 5 (see FIG. 1) such as a CG rendering device. Output according to (spec).
Details of the texture acquisition system 1000 including the texture acquisition device 1 according to the present embodiment will be described below.

<Configuration of Texture Acquisition Device>
A texture acquisition device 1 according to the present embodiment is configured by a general computer, and includes a control unit 10, an input/output unit 11, and a storage unit 12, as shown in FIG.
The input/output unit 11 includes a communication interface for transmitting and receiving information, and an input/output interface for transmitting and receiving information to and from an input device such as a keyboard and an output device such as a monitor.

The storage unit 12 is configured by a flash memory, a hard disk, a RAM (Random Access Memory), or the like. Initial setting information 200 (details of which will be described later) for texture acquisition processing by the texture acquisition device 1 is stored in the storage unit 12 .
The storage unit 12 also temporarily stores a program (texture acquisition program) for executing each function of the control unit 10 and information necessary for processing of the control unit 10 .

The control unit 10 controls overall control of the texture acquisition device 1, and as shown in FIG. It is configured including an output unit 150 .
The control unit 10 is implemented, for example, by a CPU (Central Processing Unit) (not shown) expanding a program (texture acquisition program) stored in the storage unit 12 into a RAM and executing the program.

When the texture acquisition device 1 executes texture acquisition processing, the initial setting information acquisition unit 110 preliminarily acquires information including the lighting position, light distribution characteristics, and camera orientation of each device installed in the photography booth BH as initial setting information 200. It is acquired and stored in the storage unit 12 . An example of how the initial setting information acquisition unit 110 acquires the initial setting information 200 will be described with reference to FIG.
FIG. 3 shows an example in which the photography booth BH is configured with a regular octahedron. The depth measuring device 20, the polarizing lighting device 30 and the polarizing imaging device 40 are arranged to surround the subject. As shown in FIG. 3, each polarized illumination device 30 is arranged as a point light source at the vertex of the regular octahedron. Also, the depth measuring device 20 and the polarizing device 40 are placed at the center of the plane with the line of sight directed to the center of the regular octahedron.

A plurality of wavelength bands and wavelength patterns irradiated by each light source (polarized illumination device 30) are set by the measurement control unit 130 in a combination that does not cause crosstalk. The number of combinations is at least three, and more are possible (see FIGS. 12 and 13 to be described later). Here, by using a multispectral polarized light source and a multispectral polarized camera, it is possible to set three or more axes. , and the XYZ three axes in which the R, G, and B light sources are installed in the plus and minus directions of each XYZ axis will be described.

As shown in FIG. 3, each light source (polarized illumination device 30) emits the same light from above and below (e.g., G (green)), front and back (e.g., R (red)), and left and right (e.g., B (blue)) of the subject. Irradiate the illumination of the wavelength band. In other words, polarized lights of multiple wavelengths are emitted from the polarized illumination device 30 .
In this embodiment, the depth measurement device 20 makes the shape of the object known, and also makes the lighting position known by initial setting in advance. Therefore, it is possible to obtain information on the front and back of the object surface with respect to the installed lighting, and it is possible to simultaneously irradiate illumination light of the same wavelength band or wavelength pattern from the plus and minus directions. This is because it is possible to determine which illumination on the XYZ axis is on the plus side or the minus side.
Furthermore, the positional relationship between each light source (polarized illumination device 30) and each adjacent camera is symmetrical with respect to the center of the photography booth installed inside the regular octahedron. This is to allow consideration of Fresnel reflections to be ignored when determining the "roughness" of the surface of the subject. ). Note that the regular octahedron is an example and is not limited to this.

Each light source can be composed of a surface light source (such as a flat panel display). Installation is easy because there is no need to install a flat panel display. In addition, a point light source that does not surround the subject with a display makes it easier to record the voice of the subject than a shooting booth BH that is surrounded by a surface light source (flat panel display).

The initial setting information acquisition unit 110 acquires the illumination position (X, Y, Z coordinates) where the polarized illumination device 30 as each light source is arranged, the light distribution characteristics obtained by measuring the luminous intensity in each direction in the space, and the depth measurement device. 20 and the camera position (X, Y, Z coordinates) and the camera orientation (roll, pitch, yaw) where the polarizing imaging device 40 is arranged are acquired as the initial setting information 200 from the system management device or the like, and the storage unit 12 memorize to
It is assumed that the set of the depth measurement device 20 and the polarization photography device 40 arranged at one location in the photography booth BH has the same camera position and the same camera posture.

Returning to FIG. 1, the description of the configuration is continued. The subject shape acquisition unit 120 transmits instruction information for executing measurement to each depth measurement device 20 arranged in the photography booth BH via the measurement control unit 130 . Then, the subject shape acquisition unit 120 acquires a (rough) subject shape by receiving measurement information from each depth measurement device 20 .
The depth measurement device 20 may use, for example, a ToF (Time of Flight) camera that calculates the depth by measuring the time it takes for near-infrared rays to be emitted from the camera, reflected by an object, and returned. can be done.
In addition, without installing the depth measurement device 20 shown in this embodiment, using a shape measurement method of stereo matching for a camera image (for example, the image of the polarization imaging device 40) installed so as to surround the subject, the shape of the entire subject may be obtained.

The measurement control unit 130 cooperates with the subject shape acquisition unit 120 and the texture calculation unit 140 to transmit instruction information to the depth measurement device 20, the polarized illumination device 30, and the polarized photography device 40 placed in the photography booth BH. Each device is controlled by
Specifically, the measurement control unit 130 causes the depth measurement device 20 to measure the shape of the subject by transmitting instruction information.

In addition, when the texture calculation unit 140 executes texture acquisition processing, the measurement control unit 130 provides instruction information to the polarized illumination device 30 so as to control the illumination state of a predetermined pattern as shown in FIG. 4, for example. Send. FIG. 4 is an example of RGB spectroscopy with the polarization imaging device 40 to be measured, one sequence is composed of 6 frames (FIGS. 4A to 4F), and the wavelength band of illumination by the polarization illumination device 30, And, the pattern is changed in polarization direction. Here, each illumination is lit so that the wavelength band and the polarization direction of the adjacent illumination are not the same, and is sequentially changed from R → G → B with the passage of time, and the polarization direction is also set to 0 degree. Change by 90 degrees.
For example, the polarized illumination device 30 positioned at the top of the regular octahedron changes the wavelength band from G→B→R→G→B→R as shown in FIGS. 4(a) to 4(f). , the polarization direction is also changed as follows: 90 degrees→0 degrees→90 degrees→0 degrees→90 degrees→0 degrees.

Here, the reason for changing the polarization direction is that depending on the angle of incidence of the p-wave (p-polarized light) of the illumination light on the surface of the object, the Brewster's angle may occur and the reflected light may disappear. However, this Brewster's angle is a rare case that occurs only when the surface is not rough and mirror-finished. Therefore, if there is no problem in terms of application, the change accompanying the time transition of the polarization direction can be omitted, and one sequence can be composed of three frames. In this case, for example, as shown in FIG. 4(a)→FIG. 4(c)→FIG. 4(e), only the wavelength band of the illumination may be changed by three patterns of RGB without changing the polarization direction.
When considering the disappearance of the reflected light at the Brewster angle, it may be solved by irradiating illumination light whose polarization direction is sequentially changed and interpolating information before and after the disappearance time. It is not limited to this.

In addition, the measurement control unit 130 causes the polarization photographing device 40 to capture the reflected light in response to the texture calculation unit 140 executing the texture acquisition process and the polarization lighting device 30 irradiating the subject with illumination light. Sends instruction information to execute

A camera imaging plate of the polarization imaging device 40 (here, referred to as a “polarization RGB camera”) has an RGB Bayer arrangement as shown in FIG. 5, for example. Further, each pixel of RGB is composed of four divided pixels, which selectively transmit components of polarization directions of 0 degrees, 45 degrees, 90 degrees, and 135 degrees, respectively. In FIG. 5, the hatched direction indicates the direction of the transmitted polarized light component.

Returning to FIG. 1, the description of the configuration is continued. As texture acquisition processing, the texture calculation unit 140 performs the above-described (1) diffuse reflection component/specular reflection component separation processing, (2) diffuse reflection coefficient and normal information calculation processing using the diffuse reflection component, ( 3) Perform processing for calculating the "roughness" of the object surface and the refractive index of the object using the specular reflection component. The texture calculation unit 140 includes a reflection component separation unit 141 , a diffuse reflection calculation unit 142 and a specular reflection calculation unit 143 .

The reflection component separating unit 141 separates the diffuse reflection component and the specular reflection component in the reflected light. Here, FIG. 6 shows an example in which the polarized illumination device 30 is used as a point light source, and the subject is illuminated with RGB three-color illumination under the control of the measurement control unit 130 . At that time, the polarized RGB camera 40a is used as the polarized light photographing device 40, and the object is photographed by the polarized RGB camera 40a.
The camera imaging plate of this polarized RGB camera 40a has an RGB Bayer array, as indicated by reference numeral 101 in FIG. , 45 degrees, 90 degrees, and 135 degrees are selectively transmitted. Note that the 4-divided pixels and the polarization directions of 0 degrees, 45 degrees, 90 degrees, and 135 degrees are just examples, and the present invention is not limited to these.

The observed brightness I is represented by the following equation (1) based on the dichroic reflection model. The dichroic reflection model states that the spectrum of reflected light can be represented by a linear sum of the spectra of the diffuse reflection component and the specular reflection component.

I=I _d +a(1+cos(2Θ-β)) Equation (1)
Here, I _d represents the diffuse reflection component and a(1+cos(2Θ−β)) represents the specular reflection component I _s . Also, a indicates the amplitude, and Θ indicates the polarization direction of the pixel. β is the phase.

The luminance values obtained from pixels with polarization directions of 0, 45, 90, and 135 degrees (I ₀ , I ₄₅ , I ₉₀ , and I ₁₃₅ , respectively) are averaged as shown in FIG. It exhibits sinusoidal behavior with respect to the value I _ave , but becomes sinusoidal at double the polarization angle. Therefore, using the trigonometric function formula sin(θ−β) ² +cos(θ−β) ² =1 (β considers the phase), for example, I ₀ ² +I ₄₅ ² =a (a is the amplitude of the trigonometric function ). Thereby, the amplitude a can be calculated. The minimum value I _min of the observed brightness I is the diffuse reflection component I _d , and 2a, which is twice the amplitude a, is the specular reflection component I _s .
In this way, the reflection component separating section 141 can separate the diffuse reflection component I _d and the specular reflection component I _s of the reflected light.

Note that the reflection component separation unit 141 separates, for example, one sequence (6 frames) (see FIG. 4) that has been photographed into a diffuse reflection component and a specular reflection component. At this time, the images in one sequence have different photographing times. When the subject is non-rigid and moves in one sequence, motion compensation is performed on the frame before and after the frame at the reference time, and the same subject is identified even if the time is different. Processing is performed with the body part as the target.
In addition, the reflection component separation unit 141 does not limit the method of separating the diffuse reflection component and the specular reflection component to the separation using the difference in the reflection behavior of the surface of the object of the polarized light as described above. (For example, a specular reflection component is calculated and removed by deep learning) may be used.

Returning to FIG. 1, the description of the configuration is continued. The diffuse reflection calculator 142 uses the diffuse reflection component I _d calculated by the reflection component separator 141 to calculate the diffuse reflection coefficient K _d and normal information. Here, the diffuse reflection calculator 142 uses Lambert's cosine law, which states that the intensity of reflected light is proportional to the cosine of the incident angle, and the method of photometric stereo. This photometric stereo technique obtains images illuminated in the same wavelength band from three directions. Therefore, _normal information (surface normal vector ) and the diffuse reflection coefficient _Kd .

Here, the uneven state of the object surface is defined as the surface normal vector of the object surface in shooting pixel units. do. However, since the surface of an actual subject has surfaces in even finer units, the specular reflection calculator uses the normal distribution of the finer surfaces as the parameter "γ" (roughness parameter of the surface of the subject) of the normal distribution function. 143 (details below).

On the other hand, Lambert's cosine law is expressed by Equation (2) when the diffuse reflection coefficient is K _d on the object surface in the state shown in FIG.
Here, the observed light intensity I is assumed to be the diffuse reflection component I _d obtained by removing the specular reflection component _Is from the reflected light components. Also, the diffuse reflection coefficient _Kd and the normal vector N are unknown.

Then, the diffuse reflection calculation unit 142 calculates the normal vector N and the diffuse reflection coefficient K _d from the image of the diffuse reflection component I _d obtained by photographing with the polarization imaging device 40, based on the following derivation method, the simultaneous equations is obtained by solving Here, the normal vector is a unit vector, the surface normal vector is obtained by normalizing the calculated vector N, and the Euclidean norm is the diffuse reflection coefficient _Kd .

Here, as shown in FIG. 8(a), three illuminations (L ₁ , L ₂ , L ₃ ) are photographed by the polarizing imaging device 40. As a result, according to Lambert's cosine law shown in the above equation (2), Equation (3) shown in FIG. 8(b) is derived.
Further, as shown in FIG. 8(c), the normal vector is represented by Equation (4), and the unit normal vector is represented by Equation (5).

By substituting the equations (4) and (5) into the equation (3), the equation (6) in FIG. 9A is obtained. Constants known from the illumination position and illumination intensity indicated by the initial setting information 200 and the observed light intensity (diffuse reflection component I _d ) obtained by the camera are represented by constants (Ca, Cb, Cc) (see FIG. 9(b)). As a result, the simultaneous linear equations shown in Equation (8) are established (see FIG. 9(c)).
By solving this simultaneous equation and finding that the norm of the normal vector n becomes the diffuse reflection coefficient _Kd , the diffuse reflection coefficient _Kd and the normal vector (surface normal vector) are expressed by the equation shown in FIG. 9(c) ( 9).

In this way, the diffuse reflection calculation unit 142 irradiates the object with polarized light of multiple wavelengths (RGB) from the polarized lighting devices (three lights (L ₁ , L ₂ , L ₃ )) arranged in three directions. The diffuse reflection component I _d separated by the reflection component separation unit 141 and the illumination position and illumination intensity of the polarized illumination device 30 are applied to Lambert's cosine law to determine the diffuse reflection of the subject. The coefficient _Kd and normal information (normal vector) can be calculated.

Returning to FIG. 1, the description of the configuration is continued. The specular reflection calculation unit 143 uses the specular reflection component I _s separated by the reflection component separation unit 141 to calculate a parameter (γ) indicating the degree of roughness of the object surface (hereinafter referred to as “roughness (γ)”). ) and the refractive index n _j of the subject.

<Roughness (γ) calculation process>
First, calculation processing of the surface roughness (γ) of the subject by the specular reflection calculator 143 will be described. In the present embodiment, as a predetermined specular reflection model for physically based rendering, the Cook-Torrance specular reflection model represented by the following equation (10) will be described.

In this embodiment, as shown in FIG. 10, a part of an object is observed at different angles of incidence (θ _i ) from illumination (polarized illumination device 30) at different positions and with polarized light imaging devices 40 arranged at different positions. Let the angle (θ _r ) and the observed light intensity V be the specular reflection component I _s . Here, the incident light and the reflected light used in the processing are treated independently as p-waves and s-waves so as not to consider the influence of the Fresnel term (F). In addition, the camera (polarized photographing device 40) adjacent to the lighting (polarized lighting device 30) and the position of the lighting adjacent to the camera are both symmetrical with respect to the photography booth BH. If so, the angle between the illumination vector (L) and the line-of-sight vector (V) can be assumed to be the same. This makes it possible to ignore the influence of Fresnel reflection. Note that the geometrical attenuation coefficient (G) is set to a constant 1 and is not considered in this embodiment because it has no effect on determining the brightness during rendering.

Moreover, in this embodiment, the normal distribution function D is expressed by the following equation (11). This expresses the normal distribution function as a Gaussian distribution function. For details, refer to reference 1 (Norihiro Tanaka, Shoji Tominaga, “Analysis and Estimation of 3D Reflection Model”, Information Processing Society of Japan Transactions on Computer Vision and Image Media (CVIM), vol.41, pp.1-11, Dec. 2000).

Here, the roughness (γ) of the surface of the subject is a parameter representing how much the reflection of the illumination light is dispersed, and represents how much the surface normal is dispersed.
As shown in FIG. 10, let (ρ) be the angle between the half angle (half vector H) of the angle formed by the light source and the line of sight and the surface normal vector, and the normal distribution function D (ρ ) is the half value (1/2) of the maximum value is defined as (γ). In this embodiment, from the positions of three light sources (polarized lighting device 30) and the position of one camera (polarized imaging device 40), the lighting position, orientation, intensity, camera orientation, etc. are calibrated in advance. Since (ρ) is known, the unknown (γ) can be obtained by Gaussian fitting (Gaussian curve fitting).

In this way, the specular reflection calculation unit 143 calculates the specular reflection component I _s separated by the reflection component separation unit 141, the illumination position of the polarization lighting device 30, the camera position of the polarization imaging device 40, and the normal information (normal vector) is applied to a predetermined specular reflection model (Cook-Torrance specular reflection model), and the angle at which the normal distribution function D(ρ) is half the maximum value is the roughness of the object surface (γ) can be calculated as
Note that the calculated object surface roughness (γ) is susceptible to noise due to the dynamic range of the camera and the like. Therefore, (γ) may be stabilized by averaging the values of (γ) obtained with different illumination and cameras.
In addition, the expression of the normal distribution function in this embodiment is not limited to the Gaussian distribution function described above, and it is also possible to derive the roughness (γ) by applying it to the GGX distribution function, Beckman function, etc. .

<Calculation processing of object refractive index>
Next, calculation processing of the subject refractive index n _j by the specular reflection calculator 143 will be described.
In the Cook-Torrance specular reflection model shown in Equation (10), the geometrical attenuation coefficient (G) is regarded as a constant 1 because it has little effect, as described above. Also, the normal distribution function (D), normal vector (N), illumination vector (L), and line-of-sight vector (V) are known, and π is a constant.
Here, since the p-wave and s-wave act independently, the total reflectance based on the above Cook-Torrance specular reflection model is R _{s_spec} and R _{p_spec} respectively, and the following equations (12) and ( 13). Note that i _rs represents the amount of reflected s-wave light, i _ps represents the amount of reflected p-wave light, i _s represents the amount of incident s-wave light, and _ip represents the amount of incident p-wave light.

Also, the following equations (14) and (15) can be derived from the Cook-Torrance specular reflection model (equation 10).

By solving equations (12) and (14) and equations (13) and (15), r _s,i,j and r _p,i,j are obtained. Note that F=(r _s,i,j +r _p,i,j )/2 because the illumination is normally non-polarized.
In this way, the specular reflection calculator 143 converts the illumination position of the polarizing illumination device 30, the camera position of the polarizing imaging device 40, the normal information (normal vector), and the normal distribution function (D) into a predetermined specular reflection model. (Cook-Torrance specular reflection model) can be applied to obtain the reflectance of each subject for p-wave and s-wave for the number of light sources observed.

On the other hand, the behavior of reflection and refraction on the object surface in different media is as shown in _FIG . ) is the refractive index of “n _j ”, according to the Fresnel equation, the reflectance of s-polarized light (r _s,i,j ) is represented by equation (16), and the reflectance of p-polarized light (r _{p,i , j} ) is represented by equation (17).

From the equations (16) and (17), the subject refractive index (n _j ) can be summarized as the following equation (18).

Here, since the medium 1 is air, it can be known based on the following (Reference 2: Junichiro Nakanishi, Refractive index of air and propagation characteristics of electromagnetic waves, Bulletin of Sagami Institute of Technology, 10 (1), pp:23-31, 3/31/1976). The refractive index n _i of dry air containing 0.03% carbon dioxide CO ₂ at 15° C. and 760 mmHg with respect to vacuum is given by the following equation (19) at wavelength λ ₀ (μ)=0.2 to 1.25.

As described above, the specular reflection calculation unit 143 obtains the reflectance (r _s,i,j , r _p,i,j ) considering the Fresnel reflection of the s-wave and p-wave for each incident angle, and the formula By substituting the air refractive index n _i calculated from (19) into the equation (18), the object refractive index (n _j ) can be obtained.

In the above-described embodiment, as described with reference to FIGS. 3 and 4, it is assumed that the RGB three-color light source is lit with the polarization direction changed between 0 degrees and 90 degrees. However, other than that, it is possible to acquire detailed spectral reflectance by multi-spectrum. For example, as shown in FIG. 12, the polarization direction is changed to 0 degrees and 90 degrees in light sources of six color spectra (B+, B-, G+, G-, R+, R-) with different wavelengths, and one sequence is 12 You may make it comprise by a frame. By doing so, it becomes possible to measure the spectral reflectance in detail.

As another example, as shown in FIG. 13, 12 color spectrum (B+, B'+, B-, B'-, G+, G'+, G-, G'-, R+ , R'+, R-, R'-) and a corresponding multispectral polarization camera, more detailed spectral reflectance can be observed without increasing the number of frames required for one sequence. You can also As a result, it is possible to reduce the blind area of the lighting that occurs in a complex-shaped object. Furthermore, in addition to the wavelength band, for example, by coding the light source wavelength at one location using a comb filter or multispectral illumination that can control the spectral distribution, the number of light sources required for one sequence can be reduced without increasing the number of light source points. can be reduced.

In an implementation example in which this embodiment is actually applied, one side of the regular octahedron of the photography booth BH shown in FIG. 3 is rotated and placed on the bottom as shown in FIGS. do. This is because, in the arrangement shown in FIG. 3, lighting from the bottom of the photography booth BH is difficult in terms of construction, and it is better to set the lighting on the bottom of the photography booth as shown in FIGS. This is because it is easy to install the photographing stage ST on which the .

Returning to FIG. 1, the description of the configuration is continued. The three-dimensional model detailed information output unit 150 outputs the diffuse reflection coefficient K _d , normal information (surface normal vector), object surface roughness (γ), and object refractive index (n _j ) calculated by the texture calculation unit 140. , the reflectance of the object (three-dimensional model detailed information) is drawn as predetermined three-dimensional model detailed information according to the time when the polarizing imaging device 40 captures each piece of information. Output to an external device 5 such as a device. The predetermined three-dimensional model detailed information includes the diffuse reflection coefficient K _d calculated by the texture calculation unit 140, normal information (surface normal vector), surface roughness of the subject (γ), subject refractive index (n _j ), and the reflectance information of the object may be included. It is set as dimensional model setting information and output to each external device 5 .
By acquiring this three-dimensional model detailed information, the CG rendering device (external device 5) can express in real time the state of fine unevenness on the surface of the subject as a texture.

Note that the subject spectral reflectance (s-wave reflectance (r _s,i,j ), p-wave reflectance (r _p,i,j )) calculated by the specular reflection calculator 143 is calculated using a low-dimensional linear model , and may be expressed by, for example, 7 to 8 basis functions and output. By approximating the calculated subject spectral reflectance with the basis functions and their weights, complex spectral reflectance can be expressed only with the weights and basis functions. If 7 to 8 basis functions are used and 7 to 8 weighting factors are used, the data size becomes large and the load in data storage and transmission increases, but the accuracy increases. On the other hand, if the number of basis functions is reduced to 3, the data size is reduced and the accuracy is reduced, but it is advantageous when the transmission path and storage capacity are limited.

In addition, the three-dimensional model detailed information output unit 150 outputs the object refractive index (n _j ) to the external device 5, so that rendering taking p-waves and s-waves into account is possible. However, there are cases where the CG renderer of the external device 5 does not consider polarization, and where it is preferable not to increase the load on the external device 5 . In this case, the three-dimensional model detailed information output unit 150 may convert and output the Fresnel reflection coefficient approximation R _o proposed by Schlick et al. instead of the refractive index. Note that R _o in the Fresnel reflection coefficient approximation is the specular reflection coefficient of light incident parallel to the normal, and is detailed in (Reference 3) below. By outputting this R _o to the external device 5, it is possible to perform rendering without significant deterioration in quality.
(Reference 3) “Schlicks approximation,” Wikipedia, <URL: https://en.wikipedia.org/wiki/Schlick's_approximation>

<Modified example of 3D model detailed information output unit>
Next, a modification of the three-dimensional model detailed information output unit 150 (see FIG. 1) will be described with reference to FIG. The 3D model detailed information output unit 150a according to the modification outputs predetermined 3D model detailed information from the 3D model detailed information output unit 150 according to the present embodiment, and the external device 5 shown in FIG. Information ("optimization condition information 51" to be described later) including processing capacity and presentation conditions for each rendering is acquired from each external device 5, and optimal information for each external device 5 (three-dimensional It has a function to generate and output model detailed information).

The three-dimensional model detailed information output unit 150a includes a transmission content determination unit 151, a detailed information output unit 152, and a 3DCG rendering unit 153, as shown in FIG.
Here, as an example, the external device 5, which is a CG drawing device, is divided into an external device 5A with high 3DCG drawing ability ("High" in FIG. 14) and an external device 5A with low 3DCG drawing ability ("Low" in FIG. 14). Notation) Description will be made assuming that the device is the external device 5B.

The transmission content determination unit 151 acquires the optimization condition information 51 from each external device 5 (here, the external devices 5A and 5B). The optimization condition information 51 includes, for example, device type information (stationary 3D TV, HMD, smart phone, PC, etc.), hardware resources (memory capacity, hardware capacity , CPU processing speed, etc.), information on the type of 3D data to be used such as point cloud, mesh, NURBS curve, etc., diffuse reflection coefficient K _d calculated by the texture acquisition device 1, normal information (surface normal 3DCG of each external device 5 such as information (presentation conditions) requested by the external device 5 among information such as object surface roughness (γ), object refractive index (n _j ), object reflectance, etc. Contains information needed to optimize rendering.

Using the optimization condition information 51 acquired from each external device 5, the transmission content determination unit 151 first outputs the three-dimensional model detailed information 52 or the two-dimensional image 53 as the output information. judge. Here, for example, when the 3DCG drawing ability is high like the external device 5A (for example, a dedicated stereoscopic television equipped with a CPU with high processing ability), the transmission content determination unit 151 determines the detailed 3D model information 52. is determined as an output. Then, the transmission content determination unit 151 instructs the detailed information output unit 152 to generate the three-dimensional model detailed information 52 according to the specifications and settings of the external device 5 . For example, the transmission content determination unit 151 performs reduction (thinning) of point groups, polygons, and control points according to the specifications of the external device 5 and issues an instruction to generate the three-dimensional model detailed information 52 .

On the other hand, the transmission content determination unit 151, for example, when the 3DCG drawing ability is low like the external device 5B (for example, a PC having a CPU with low 3DCG drawing ability processing ability), the 3D model detailed information 52 is It is determined that the texture information is to be sent as a two-dimensional image 53 instead of being sent, and information to that effect is output to the 3DCG rendering section 153 .
In this case, the transmission content determination unit 151 performs resampling of the two-dimensional image for the external device 5B, acquires information such as resolution, and outputs instruction information including the information to the 3DCG rendering unit 153. .

Based on the instruction from the transmission content determination unit 151, the detailed information output unit 152 selects the diffuse reflection coefficient K _d , normal line Information selected from information (surface normal vector), object surface roughness (γ), object refractive index (n _j ), and object reflectance is used as three-dimensional model detailed information 52 corresponding to each external device. It generates and outputs to the external device 5 (external device 5A in this case).

The 3DCG rendering unit 153 has a 3DCG rendering function, and executes rendering processing on behalf of the external device 5 that does not have the rendering function or the external device 5 (for example, the external device 5B) that has low processing power in 3D drawing. A two-dimensional image 53 corresponding to the external device 5 is converted and output.

By doing so, the 3D model detailed information output unit 150a can optimize the information to be output according to the hardware resources and specifications of each external device 5 . Note that the 3D model detailed information output unit 150a does not divide the 3DCG drawing capability into two stages such as high and low as shown in FIG. According to the level of ability (multiple levels of three or more levels), shape data may be reduced and sent, or the number of basis functions expressing spectral reflectance may be reduced and sent.

Further, when the three-dimensional model detailed information output unit 150a transmits the three-dimensional model detailed information 52 to the external device 5 and causes the external device 5 to perform rendering, a delay occurs due to data distribution and an increase in processing paths. expected to occur. Therefore, the transmission content determination unit 151 determines whether or not the delay time defined in advance as the latency requirement is satisfied based on the performance of the external device 5 and the network state indicated by the obtained optimization condition information 51. Whether to output as the dimensional model detailed information 52 or as the two-dimensional image 53 may be dynamically controlled. This latency requirement may be included in the optimization condition information 51 and acquired from each external device 5 by the three-dimensional model detailed information output unit 150a.
By doing so, the texture acquisition device 1 outputs the three-dimensional model detailed information 52 based on whether or not the delay time (latency requirement) required for establishment of interaction with the external device 5 is satisfied. It is possible to automatically control whether to output as a two-dimensional image 53 or to output as a two-dimensional image 53 .

<Operation of texture acquisition device>
Next, operations performed by the texture acquisition device 1 according to the present embodiment will be described with reference to FIG. 15 .

First, the initial setting information acquisition unit 110 of the texture acquisition device 1 acquires the camera position, camera orientation, and illumination position of each device (the depth measurement device 20, the polarized lighting device 30, and the polarized photography device 40) installed in the photography booth BH. , information such as light distribution characteristics is acquired as the initial setting information 200 from the system management device or the like, and stored in the storage unit 12 (step S1).

Next, the subject shape acquisition unit 120 of the texture acquisition device 1 transmits instruction information for executing measurement to the depth measurement device 20 placed in the photography booth BH via the measurement control unit 130 . Then, the subject shape acquisition unit 120 acquires the (rough) subject shape by receiving the measurement information from the depth measurement device 20 (step S2).

Next, the texture calculation unit 140 (reflection component separation unit 141) of the texture acquisition device 1 separates the reflected light from the surface of the object into diffuse reflection component I _d and specular reflection component I _s (step S3).
Specifically, the texture calculation unit 140 (reflection component separation unit 141), via the measurement control unit 130, emits RGB three-color illumination in a pattern such as that shown in FIG. Illuminate the subject. Then, the reflected light is photographed by the polarization photographing device 40 .

Then, as shown in FIG. 6, the reflection component separating unit 141 generates a sine wave from the brightness values obtained from the pixels with the polarization directions of 0 degrees, 45 degrees, 90 degrees, and 135 degrees. Then, the reflection component separation unit 141 sets the minimum value I _min of the luminance I of the sine wave as the diffuse reflection component I _d and sets 2a, which is twice the amplitude a, as the specular reflection component I _s .

Next, the texture calculation unit 140 (diffuse reflection calculation unit 142) calculates a diffuse reflection coefficient _Kd and normal line information using the diffuse reflection component _Id calculated by the reflection component separation unit 141 (step S4).
Here, the diffuse reflection calculation unit 142 applies Lambert's cosine law and the photometric stereo method to derive the equation (3) shown in FIG. I _d ) and information such as the illumination position indicated by the initial setting information 200, formula (8) shown in FIG. 9(c) is derived. Then, the diffuse reflection calculator 142 calculates the diffuse reflection coefficient K _d and the normal information (normal vector) by solving the simultaneous linear equations of Equation (8).

Subsequently, the texture calculation unit 140 (specular reflection calculation unit 143) calculates the roughness (γ) of the object surface using the specular reflection component I _s calculated by the reflection component separation unit 141 (step S5).
Here, as shown in FIG. 10, the specular reflection calculation unit 143 sets the angle between the half angle (half vector H) of the angle formed by the light source and the line of sight to the surface normal vector (ρ), and the roughness parameter Regarding (γ), let (γ) be the angle at which the normal distribution function D(ρ) is half the maximum value (1/2). Then, the specular reflection calculator 143 obtains the unknown (γ) of the normal distribution function expressed by the above equation (11) by Gaussian fitting.

Next, the texture calculation unit 140 (specular reflection calculation unit 143) calculates the reflectance for each of the s-wave and the p-wave based on the amount of incident light and the amount of reflected light (step S6).
Here, the specular reflection calculator 143 calculates the reflectance of each of the s-wave and the p-wave based on the above equations (12) and (13).

Subsequently, the specular reflection calculation unit 143 calculates the object refractive index (n _j ) from the above equation (18) using the calculated p-wave and s-wave reflectances. It is assumed that the refractive index (n _i ) of air is stored in advance in the initial setting information 200 (step S7).

Then, the three-dimensional model detailed information output unit 150 of the texture acquisition device 1 outputs the diffuse reflection coefficient K _d , the normal information (surface normal vector), the roughness (γ) of the object surface, Of the object refractive index (n _j ) and the object reflectance, predetermined information set in advance is generated as three-dimensional model detailed information, and in order to calculate each piece of information, according to the time when the polarization photographing device 40 photographed , to an external device 5 such as a CG drawing device (step S8).
At this time, as described above, the three-dimensional model detailed information output unit 150 may express the reflectance by basis functions and their weights and include them in the three-dimensional model detailed information. It may be included in the three-dimensional model detailed information as _Ro of the Fresnel reflection coefficient approximation formula.

14, the optimization condition information 55 is acquired from each external device 5, and according to the specifications and settings of the external device 5, Alternatively, the 3D model detailed information output unit 150a (3DCG rendering unit 153) renders the 3D model detailed information 52, converts it into a 2D image 53, and outputs it to the external device 5. You may do so.

≪Example of implementation≫
An implementation example is explained. Here, as shown in FIG. 3, the photography booth BH has a regular octahedron with polarized illumination devices 30 arranged at the vertices of a regular octahedron, and a depth measurement device 20 and a polarized photography device 40 arranged at the center of each plane. Install so as to photograph the inside of the mask. Also, as shown in FIGS. 12 and 13, the arrangement is such that one surface of the regular octahedron is rotated as the bottom surface.
Here, it is assumed that a linear polarizing filter is arranged in the polarizing illumination device 30, the polarization direction is not polarized, and shooting is performed with one sequence of three frames.
A high-speed polarization RGB camera of 240 Hz, for example, is used as the polarization imaging device 40, and motion detection and motion cancellation processing by optical flow are performed between sequences. This suppresses the influence of the movement of the subject within one sequence. In this implementation example, since it is a 240 Hz high-speed polarized RGB camera, it is possible to shoot 3 frames in one sequence at 240/3=80 Hz.
With this configuration of the photography booth BH, texture acquisition processing is performed by the texture acquisition device 1 according to the present embodiment, and metadata such as the acquisition time and other information (shooting conditions, camera configuration information, etc.) is added to the three-dimensional model detailed information. output the attached file.

Here, a specific example of the effect of calculating the diffuse reflection coefficient _Kd and the normal information by the texture calculation unit 140 (diffuse reflection calculation unit 142) of the texture acquisition device 1 of the present embodiment will be described.
In the conventional method, the diffuse reflection coefficient _Kd and the normal information (normal vector) are calculated without separating the diffuse reflection component _Id and the specular reflection component _Is of the reflected light. On the other hand, the texture acquisition apparatus 1 according to this embodiment separates the diffuse reflection component I _d and the specular reflection component I _s of the reflected light, and uses only the diffuse reflection component I _d (diffuse reflection light) to obtain the diffuse reflection coefficient Calculate _Kd and normal information (normal vector).

As a result, as shown in FIG. 16A, the result of the diffuse reflection coefficient K _d of this embodiment using only the diffusely reflected light (line A) is the diffuse reflection coefficient The value is closer to the true value than the result of _Kd .
Further, as shown in FIG. 16B, the result of the normal line information (normal vector) of this embodiment using only the diffusely reflected light (line A) is the normal line information (normal line vector) obtained by not separating the reflected light (line B). The value is closer to the true value than the line information (normal vector).
According to the texture acquisition apparatus 1 according to the present embodiment, by calculating an accurate value of the normal information (normal vector) in this way, the roughness of the surface of the object calculated using this information (γ) and object refractive index (n _j ) values are also more accurate.
As described above, the texture acquisition device 1 acquires in real time the detailed information of the three-dimensional model necessary for photorealistic CG rendering capable of expressing the texture of the surface of the subject, and outputs it to each external device 5 as optimum information. It becomes possible.

1 texture acquisition device 5 external device 10 control unit 11 input/output unit 12 storage unit 20 depth measurement device 30 polarization lighting device 40 polarization photography device 40a polarization RGB camera 51 optimization condition information 52 three-dimensional model detailed information 53 two-dimensional image 110 initial Setting information acquisition unit 120 Subject shape acquisition unit 130 Measurement control unit 140 Texture calculation unit 141 Reflection component separation unit 142 Diffuse reflection calculation unit 143 Specular reflection calculation unit 150, 150a Three-dimensional model detailed information output unit 151 Transmission content determination unit 152 Detailed information Output unit 153 3DCG rendering unit 200 Initial setting information 1000 Texture acquisition system BH Photography booth ST Photography stage

Claims (5)

a photography booth having a depth measuring device, a polarized lighting device, and a polarized photography device arranged around a subject;
a texture acquisition device that is communicatively connected to the depth measurement device, the polarized lighting device, and the polarized imaging device, and acquires a texture that is indicated by the uneven state of the surface of the subject, wherein
The texture acquisition device includes:
a storage unit for storing initial setting information including the illumination position and illumination intensity of the polarized illumination device and camera positions and camera orientations of the depth measurement device and the polarized imaging device;
a subject shape acquisition unit that acquires shape information of the subject by receiving depth information measured by the depth measurement device;
a measurement control unit that controls the polarized illumination device to irradiate the subject with polarized light of multiple wavelengths;
a reflection component separation unit that acquires the reflected light from the subject irradiated with the polarized light of each of the plurality of wavelengths as information photographed by the polarization imaging device, and separates the reflected light into a diffuse reflection component and a specular reflection component; ,
The diffuse reflection component separated by the reflection component separation unit for the reflected light photographed by the polarization imaging device by irradiating the subject with the polarized light of the plurality of wavelengths by the polarized lighting devices arranged in three directions. and a diffuse reflection calculation unit that applies the illumination position and illumination intensity of the polarized illumination device to Lambert's cosine law to calculate the diffuse reflection coefficient and normal information of the subject;
The specular reflection component separated by the reflection component separation unit for each of the polarized lights of the plurality of wavelengths, the illumination position of the polarized illumination device, the camera position of the polarized imaging device, and the calculated normal line information are stored in a predetermined manner. is applied to the specular reflection model, and the angle at which the normal distribution function obtained by the predetermined specular reflection model has a half value of the maximum value is calculated as a parameter of the surface roughness of the subject,
Applying the illumination position of the polarized illumination device, the camera position of the polarized imaging device, the normal information, and the normal distribution function to the predetermined specular reflection model, the p-wave and s-wave of the polarized light of the plurality of wavelengths a specular reflection calculation unit that calculates the reflectance of the subject at each point, and applies the calculated reflectance to Fresnel's formula for reflection in media having different refractive indices to calculate the refractive index of the subject;
the calculated diffuse reflection coefficient, the normal information, the roughness of the surface of the subject, the reflectance, and the refractive index, the polarizing apparatus for calculating the information; and a three-dimensional model detailed information output unit for outputting three-dimensional model detailed information in accordance with photographing time. A texture acquisition device that is communicatively connected to a depth measurement device, a polarized lighting device, and a polarized imaging device that are arranged surrounding a subject in a photography booth, and that acquires a texture indicated by the unevenness of the surface of the subject,
a storage unit for storing initial setting information including the illumination position and illumination intensity of the polarized illumination device and camera positions and camera orientations of the depth measurement device and the polarized imaging device;
a subject shape acquisition unit that acquires shape information of the subject by receiving depth information measured by the depth measurement device;
a measurement control unit that controls the polarized illumination device to irradiate the subject with polarized light of multiple wavelengths;
a reflection component separation unit that acquires the reflected light from the subject irradiated with the polarized light of each of the plurality of wavelengths as information photographed by the polarization imaging device, and separates the reflected light into a diffuse reflection component and a specular reflection component; ,
The diffuse reflection component separated by the reflection component separation unit for the reflected light photographed by the polarization imaging device by irradiating the subject with the polarized light of the plurality of wavelengths by the polarized lighting devices arranged in three directions. and a diffuse reflection calculation unit that applies the illumination position and illumination intensity of the polarized illumination device to Lambert's cosine law to calculate the diffuse reflection coefficient and normal information of the subject;
The specular reflection component separated by the reflection component separation unit for each of the polarized lights of the plurality of wavelengths, the illumination position of the polarized illumination device, the camera position of the polarized imaging device, and the calculated normal line information are stored in a predetermined manner. is applied to the specular reflection model, and the angle at which the normal distribution function obtained by the predetermined specular reflection model has a half value of the maximum value is calculated as a parameter of the surface roughness of the subject,
Applying the illumination position of the polarized illumination device, the camera position of the polarized imaging device, the normal information, and the normal distribution function to the predetermined specular reflection model, the p-wave and s-wave of the polarized light of the plurality of wavelengths a specular reflection calculation unit that calculates the reflectance of the subject at each point, and applies the calculated reflectance to Fresnel's formula for reflection in media having different refractive indices to calculate the refractive index of the subject;
the calculated diffuse reflection coefficient, the normal information, the roughness of the surface of the subject, the reflectance, and the refractive index, the polarizing apparatus for calculating the information; and a three-dimensional model detailed information output unit for outputting predetermined three-dimensional model detailed information in accordance with photographing time. The texture acquisition device is connected for communication with one or more external devices,
Instead of outputting the predetermined 3D model detailed information, the 3D model detailed information output unit acquires optimization condition information indicating information about 3DCG rendering processing capability of each of the external devices, information to be included in the three-dimensional model detailed information is selected from the diffuse reflection coefficient, the normal line information, the surface roughness of the object, the reflectance, and the refractive index according to the processing capability of the external 3. The texture acquisition device according to claim 2, wherein the three-dimensional model detailed information corresponding to each device is generated and output. The texture acquisition device has a 3DCG rendering function,
The three-dimensional model detailed information output unit
When it is determined that the predetermined latency requirement acquired as the optimization condition information is not satisfied even if the three-dimensional model detailed information corresponding to each of the external devices is output to the external device, the external 4. The texture acquisition device according to claim 3, wherein 3DCG rendering is executed instead of the device, converted into a two-dimensional image corresponding to the external device, and output. A texture acquisition program for causing a computer to function as the texture acquisition device according to any one of claims 2 to 4.

Images