JP6575999B2 - Lighting information acquisition device, lighting restoration device, and programs thereof - Google Patents

Description

  The present invention relates to an illumination information acquisition device that acquires illumination information at a place where a video or the like is captured, an illumination restoration device that restores illumination based on the illumination information, and a program thereof.

  When producing video content, a technique for synthesizing a live-action video and computer graphics (CG) may be used. Using such a technique, it is expected that the range of video expression will be widened by synthesizing live-action video and CG to produce video.

  Patent Document 1 describes a method of generating a diffuse reflected light component image, a specular reflected light component image, and light source information using two imaging units. Specifically, the diffuse reflected light component image generation unit selects the smaller one of the corresponding pixels in the two captured images as the minimum luminance value, and diffuse reflection indicating the minimum luminance value for each corresponding pixel. A light component image is generated. Further, the specular reflection light component image generation unit selects the larger of the corresponding pixels in the two captured images as the maximum luminance value, and the difference between the maximum luminance value and the minimum luminance value for each corresponding pixel. A specular reflection light component image is generated. The light source information generation unit obtains the direction of the light source based on the incident angle of the reflected light from the subject, and calculates the light amount of the light source from the specular reflection component image and the light source direction.

  Patent Document 2 describes a technical configuration for camera calibration that is required when a live-action image and CG are combined. By using this prior art, it is possible to calibrate the posture of the camera, the zoom state of the zoom lens, and the like. As a result, it can be expected to improve the accuracy of geometric synthesis of the live-action video and CG.

JP 2012-173916 A JP 2010-0747430 A

The method described in Patent Document 1 has a problem that information on the shape of the light source and information on the accurate light amount cannot be obtained.
In addition, the method described in Patent Document 2 has a problem that information on the light source cannot be obtained although the camera posture, the zoom state of the zoom lens, and the like can be calibrated.

  In order to synthesize a live-action video and CG naturally, it is necessary to reflect not only the camera posture and the like, but also information related to the lighting situation at the time of shooting the real-time video at the time of CG generation. Specifically, the information regarding the lighting state at the time of photographing is information indicating from what direction what component of light reaches to the photographing location with what intensity. This is hereinafter referred to as illumination information. On the other hand, in consideration of storing and using such illumination information as an archive together with a live-action video, it is desired that the data amount of the illumination information does not become too large.

  The present invention has been made in consideration of the above-described circumstances. An illumination information acquisition apparatus for rendering CG in a more natural lighting situation so that a live-action image and CG can be synthesized. , A lighting restoration apparatus, and these programs.

  In order to solve the above problems, an illumination information acquisition device according to an aspect of the present invention is included in an image acquisition unit that acquires images captured by cameras provided at a plurality of positions in a three-dimensional space, and the acquired image An illumination area separation unit that identifies an illumination area in the three-dimensional space based on a pixel value of the pixel to be displayed, a representative position, a luminance value, and a size for each of the illumination areas identified by the illumination area separation unit; Is provided as an illumination information for diffuse reflection component, and an image output unit for outputting the image acquired by the image acquisition unit as specular reflection component illumination information.

  According to another aspect of the present invention, in the illumination information acquisition apparatus, the image output unit outputs the image after performing a process of reducing a dynamic range of luminance of the image acquired by the image acquisition unit. It is characterized by that.

  One embodiment of the present invention is the above-described illumination information acquisition device, wherein the illumination area separation unit is configured to detect the pixel based on a pixel value of a pixel included in the image for each position of the camera that captures the image. Whether or not the illumination is reflected, and based on the determination result, vote for the area in the direction corresponding to the pixel included in the three-dimensional space, and according to the number of votes obtained by the area It is characterized by determining whether an area | region is the area | region of the said illumination.

  According to another aspect of the present invention, a low dynamic range image having a predetermined dynamic range is acquired as illumination information for specular reflection components, and based on a high dynamic range image having a relatively higher dynamic range than the low dynamic range image. The information of the representative position, the luminance and the size of the illumination area in the three-dimensional space estimated as described above is acquired as the diffuse reflection component illumination information, and the portion where the luminance is saturated in the low dynamic range image, A luminance information compensation unit that performs a process of restoring the luminance in a pseudo manner based on the information on the luminance and the size included in the diffuse reflection component illumination information having a representative position corresponding to the portion, and the luminance information compensation In computer graphics using illumination information for specular reflection components whose luminance is restored by the And specular reflection component drawing unit for drawing the specular reflection component that is an illumination restoring apparatus characterized by comprising.

  One embodiment of the present invention is a program for causing a computer to function as any one of the illumination information acquisition apparatuses described above.

  One embodiment of the present invention is a program for causing a computer to function as any one of the above-described illumination restoration devices.

  According to the present invention, it is possible to synthesize a live-action image and a CG in a state where there is no sense of incongruity regarding illumination conditions. In addition, it is possible to efficiently accumulate or transmit illumination information necessary for that purpose.

It is a block diagram which shows schematic structure of the illumination information acquisition apparatus by 1st Embodiment of this invention. It is the schematic which shows the external appearance structure of the imaging device for acquiring illumination information in the embodiment. In the embodiment, it is the schematic which showed typically the relationship between the voxel in the three-dimensional space of imaging | photography object, and the camera of the imaging device for acquiring illumination information. In the process in which the illumination information estimation unit according to the embodiment estimates illumination information based on the acquired image, a process of assigning a label corresponding to a camera (an imaging device for acquiring illumination information) to some voxels It is the schematic which shows the outline | summary. It is the schematic which shows the outline | summary of the method in which the illumination information estimation part by the embodiment uses means other than the number of voxels in order to obtain | require the size of the group of voxels. It is a flowchart which shows the procedure of the process which acquires the illumination information for diffuse reflection components by the embodiment. FIG. 3 is a block diagram illustrating a schematic functional configuration of an illumination restoration device that restores an illumination state using illumination information and a synthesis device that synthesizes a CG and a live-action image according to the embodiment. FIG. 10 is a schematic diagram showing an example of a method for obtaining a luminance value of a region divided by Voronoi division, which is a part of a method for acquiring illumination information for diffuse reflection components according to the second embodiment.

Next, embodiments of the present invention will be described with reference to the drawings.
[First Embodiment]
FIG. 1 is a block diagram illustrating a schematic functional configuration of the illumination information acquisition apparatus according to the present embodiment. In the same figure, the code | symbol 1 is an illumination information acquisition apparatus. As illustrated, the illumination information acquisition apparatus 1 includes a video acquisition unit 21 (image acquisition unit), a spatial information setting unit 22, an illumination region separation unit 23, an illumination information estimation unit 24, and an LDR image separation unit 41 ( An image output unit), a compression unit 50, a storage unit 52, and a transmission unit 53.

  The video acquisition unit 21 acquires images (videos) captured by cameras provided at a plurality of positions in the three-dimensional space. Normally, cameras are installed at a plurality of positions in a three-dimensional space, and these cameras acquire images simultaneously.

  The spatial information setting unit 22 sets a coordinate system in the three-dimensional space. That is, the spatial information setting unit 22 assumes a voxel space in which voxels are arranged without gaps based on predetermined coordinate axes (usually, three coordinate axes orthogonal to each other). The voxel will be described later. Then, the spatial information setting unit 22 sets the coordinate value of the position where the plurality of cameras are installed. Then, each pixel included in the image captured from each camera is associated with the above voxel.

  The illumination area separation unit 23 specifies an illumination area in the three-dimensional space based on the pixel values of the pixels included in the image acquired by the video acquisition unit 21. Note that the illumination area separation unit 23 determines, for each position of the camera that captured the image, whether or not illumination is reflected in the pixel based on the pixel value of the pixel included in the image, and based on the determination result. Voting is performed on a region in a direction corresponding to the pixel included in the three-dimensional space, and it is determined whether or not the region is an illumination region according to the number of votes obtained by the region. Here, the minimum unit of the region in the three-dimensional space is the voxel. Voting in the illumination area separation unit 23 is also performed for voxels. Whether or not the voxel region is illuminated by voting to a voxel when the pixel value of a pixel included in an image taken from a certain camera position satisfies a predetermined condition (and only in that case) Estimate.

  The illumination information estimation unit 24 determines the representative position (for example, the center of gravity position), the luminance value of the region, and the size of the region for each of the illumination regions specified by the illumination region separation unit 23, and diffused reflection component illumination information. Estimate as

  The LDR image separation unit 41 outputs the image acquired by the video acquisition unit 21 as specular reflection component illumination information. Note that the LDR image separation unit 41 outputs the image after performing processing for reducing the dynamic range of the luminance of the image acquired by the video acquisition unit 21. That is, the LDR image separation unit 41 generates and outputs a low dynamic range image from the high dynamic range image. By reducing the dynamic range, the information amount (data amount) of the specular reflection component illumination information to be output can be reduced.

The compression unit 50 compresses and encodes the diffuse reflection component illumination information output from the illumination information estimation unit 24 and the specular reflection component illumination information output from the LDR image separation unit 41 by an appropriate method.
The storage unit 52 stores the information encoded by the compression unit 50 in a recording medium or the like.
The transmission unit 53 transmits the information encoded by the compression unit 50 to an external device through communication.

Next, more detailed functions and operations of each unit included in the illumination information acquisition apparatus 1 will be described.
The video acquisition unit 21 acquires video signals from a plurality of external imaging devices 14, 15, and 16.

The illumination information acquisition device 1 according to the present embodiment performs processing by capturing images captured by the three imaging devices 14, 15, and 16. Note that the number of imaging devices is arbitrary as long as it is two or more, but it is particularly desirable that the number is three or more. In addition, when the number of imaging devices is three or more, it is desirable to avoid an arrangement in which three or more of them are arranged in a straight line. This is to prevent an error in specifying the position of illumination using the principle of triangulation or to reduce the error.
In the following description, description is made on the assumption that there are three imaging devices. When the implementation is performed in such a way that the number of imaging devices is different, it is necessary to appropriately read the “three units” as the actual number of imaging devices.

Next, the outline of the imaging devices 14, 15, and 16 described above will be described.
FIG. 2 is a schematic diagram showing the external appearance of each of the imaging devices 14, 15, 16. As shown in the figure, each of the imaging devices 14, 15, and 16 includes HDR cameras 81 and 82 and support means 83. HDR is an abbreviation for high dynamic range, and represents that high dynamic range imaging is performed.
Each of the HDR cameras 81 and 82 has a wide dynamic range of about 140 decibels (dB) to 170 decibels, and can be procured from a video equipment manufacturer. The HDR cameras 81 and 82 are provided back to back. Thereby, the imaging optical axes of the HDR cameras 81 and 82 are opposite to each other by 180 degrees. Each of the HDR cameras 81 and 82 includes an imaging lens (such as a fisheye lens) having a viewing angle of 180 degrees or more, and images a wide range of images. Therefore, the imaging range of such two cameras covers the whole celestial sphere. In addition, in order to clarify the relative arrangement of each camera, calibration processing is performed in advance. Video signals taken by the HDR cameras 81 and 82 are supplied to the illumination information acquisition apparatus 1 via a cable.
The support means 83 is, for example, a tripod. Since the support unit 83 supports the HDR cameras 81 and 82, the HDR cameras 81 and 82 can be installed at desired positions at desired positions.

  Instead of using a camera that can shoot HDR video by itself as the HDR cameras 81 and 82, HDR video may be generated by integrating images shot by changing the exposure amount in time division. .

  Next, details of a method for acquiring illumination information for each reflection component will be described. Here, the reflection component is two types of specular reflection component and diffuse reflection component in computer graphics (CG). These two types of reflection components are used in different ways when the CG is drawn later using the acquired illumination information. The specular reflection is a reflection in which light incident from one direction is emitted in one direction with respect to an object. Diffuse reflection is reflection in which light incident from one direction is emitted in various directions with respect to an object. Usually, the reflection of light incident on an object is a superposition of a specular reflection component and a diffuse reflection component.

  First, the specular reflection component is a component that is specularly reflected on an object or the like in CG, and is used to express the surrounding reflection on the object or the like. By appropriately acquiring the information of the specular reflection component, it is possible to express a reflection without a sense of incongruity. The specular reflection component acquired in the present embodiment is an omnidirectional image captured by each imaging device (14, 15, 16).

  The diffuse reflection component is a component that diffusely reflects an object or the like in CG. The information of the diffuse reflection component is information such as the position of the light source (lighting or the like) in the three-dimensional space, the luminance, the size of the light source, and the like. Note that, in an environment for photographing video or the like, when a picture is normally taken using an LDR (low dynamic range) camera (not an HDR camera), a pixel value in a portion where the light source is reflected is saturated. Therefore, in that case, the necessary luminance information cannot be obtained correctly as part of the diffuse reflection component information. Therefore, in order to obtain diffuse reflection component information, the above-described HDR camera image is used.

  Hereinafter, the acquisition of the diffuse reflection component illumination information will be described in detail, and then the acquisition of the specular reflection component illumination information will be described.

(1) Acquisition of Diffuse Reflection Component Illumination Information The illumination information acquisition device 1 uses three omnidirectional HDR images acquired from the imaging devices 14, 15, and 16, respectively, and uses the images to describe the following method To obtain the illumination information for the diffuse reflection component.

  The spatial information setting unit 22 sets a voxel in a three-dimensional space for video shooting. The space in which the spatial information setting unit 22 sets voxels includes the imaging devices 14, 15, and 16 and also includes illumination (light source). Note that a voxel is a three-dimensional lattice unit composed of a minimum unit of length in each coordinate axis when the coordinate axis is set in a three-dimensional space. In this embodiment, a three-dimensional orthogonal coordinate system is used, and the units of lengths in these three coordinate axes are the same. The spatial information setting unit 22 sets voxels with a size that can sufficiently obtain a spatial resolution in the processing described below. In this voxel setting operation, the spatial information setting unit 22 specifically determines the directions of three orthogonal coordinate axes at the video shooting site, determines the origin of the coordinate system, and sets the unit length in the coordinate system. And the coordinates of the positions of the imaging devices 14, 15, and 16 in the coordinate system are grasped (for example, by external input). Then, for each of the two HDR cameras (reference numerals 81 and 82) included in each of the imaging devices 14, 15, and 16, the spatial information setting unit 22 includes a pixel on the imaging surface, a lens principal point of the camera from the pixel. Corresponds to all voxels ahead of the extended straight line. At this time, it is not always necessary to store all the coordinate values of the voxels corresponding to a certain pixel. For example, it is sufficient to store information for calculating the coordinate values of the voxel corresponding to a certain pixel. As an example, the spatial information setting unit 22 stores, for a certain pixel, coefficients of a linear equation connecting the pixel and the lens principal point. Alternatively, the optical characteristics of the camera (for example, information on the focal length or angle of view of the camera lens) and the installation status of the camera (the coordinates of the position of the principal point of the camera lens and the vector value representing the orientation of the camera) ) As data, and the coefficient of the linear equation may be calculated.

  FIG. 3 is a schematic diagram schematically showing the relationship between the voxel in the three-dimensional space and the camera of the imaging device. The lattice shown in the figure shows a large number of voxels in a three-dimensional space. An actual voxel is a space in the form of a three-dimensional cube, but is projected and represented as a two-dimensional square in FIG. Also, Cam1, Cam2, and Cam3 in the figure are one of the HDR cameras (either the HDR camera 81 or the HDR camera 82) provided in the imaging devices 14, 15, and 16, respectively. In the figure, the lens portion is schematically shown in the configuration of each camera. A set of straight lines extending from the principal point of the lens of each camera indicates a portion in the three-dimensional space corresponding to one pixel in the image sensor in the camera. This partial space has the shape of a cone having a sufficient length (height) in theory. In other words, the space corresponding to one pixel in the image sensor in the camera expands in accordance with the distance from the lens of the camera (that is, the farther away it is). In the figure, the partial spaces (cone shapes) extending from the HDR cameras Cam1, Cam2, and Cam3 intersect at one of the voxels indicated by the lattice. That is, a certain voxel in the three-dimensional space corresponds to a certain pixel in each of the three HDR cameras. The pixels in each HDR camera and the voxels in the three-dimensional space can be obtained in association with the information set by the spatial information setting unit 22 as described above.

Here, variations regarding the luminance value and the pixel value described in the following processing will be described. Images taken by the imaging devices 14, 15, and 16 are color images. There are typically the following three methods for obtaining illumination information based on the video. Any of these three methods is used in the following processing.
(1) For the color space represented by the three primary colors of RGB (green, blue, red), the processing described below is performed for each of RGB, and illumination information for diffuse reflection components is acquired for each color of RGB.
(2) An image represented by RGB or the like is converted into a gray scale value by a predetermined calculation formula, and illumination information for diffuse reflection components is acquired based on the gray scale value. It should be noted that the formula for converting a color image into a grayscale image is based on an existing technique. There are several types of such expressions, but an appropriate one is selected and used.
(3) In the color space represented by the luminance signal and the color difference signal, the processing described below is performed only for the value representing the luminance, and the diffuse reflection component illumination information is acquired for each of the RGB colors. The color space represented by the luminance signal and the color difference signal is a color space such as YUV, YCbCr, or YPbPr, for example.

The illumination region separation unit 23 estimates a portion that is illumination from the images obtained by the imaging devices 14, 15, and 16. Moreover, the illumination area separation unit 23 appropriately groups the parts estimated to be illumination. For this purpose, the illumination area separation unit 23 performs the following process.
In other words, the illumination area separation unit 23 identifies pixels that are part of an image that can be regarded as illumination among the pixels in the images based on the images obtained from the three imaging devices 14, 15, and 16. To do. Then, the illumination area separation unit 23 performs voting on the voxels included in the partial space (cone shape) corresponding to the pixel considered to be illumination. Specifically, the illumination region separation unit 23 exceeds the threshold Th when the luminance value of the voxel corresponding to the pixel included in one frame of the acquired video exceeds a predetermined threshold Th. Vote one vote for those voxels with luminance values.

  Here, the luminance value of the voxel corresponding to the pixel is a function f (x) that is a characteristic when the HDR camera operates from the pixel value of the pixel in the captured image data, and the lens of the HDR camera. This value can be obtained according to the distance from the principal point to the subject. The relationship between the pixel value of the pixel in the image data and the luminance value of the voxel is expressed by the following equation (1).

  In this equation, L is the luminance value of the subject (voxel). R is the distance from the HDR camera to the subject (voxel). Further, f (x) is a function that receives the amount of light incident on the HDR camera and outputs a pixel value of the light. P is a pixel value (a pixel signal value of the pixel, that is, a luminance value) proportional to the illuminance of the pixel. The function f (x) is a monotonically increasing function. That is, in the above equation (1), the value of P increases as the value of (L / r ^ 2) increases. That is, under the condition where the distance r does not change, the pixel value P of the pixel increases as the value of the luminance L of the voxel increases. The function f (x) is a function that can be obtained in advance by measuring the amount of light incident on the lens of the HDR camera and the pixel value, and is a function that represents the relationship between the amount of incident light and the pixel value. In order to obtain the voxel luminance value L from the pixel value P of the pixel on the basis of the equation (1), the illumination region separation unit 23 performs the calculation according to the following equation (2).

Here, the function f ⁻¹ (x) is an inverse function of f (x). Similarly to the above, f ⁻¹ (x) is obtained in advance by measuring the amount of light incident on the lens of the HDR camera and the pixel value.

An example of how to determine the threshold Th is as follows. For example, the pixel value obtained by the HDR camera is set to 0 to 16777215. 16777215 is 2 ^ 24-1. And the range corresponding to the pixel value by a normal camera (LDR camera) shall be 256-65535 of said 0-16777215. 65535 is 2 ^ 16-1. In such a case, for example, (r ^ 2) · f ⁻¹ (65535) is set as the threshold Th. Note that the method of determining the threshold values described here is merely an example, and other threshold values may be determined as appropriate.

The illumination area separation unit 23 evaluates the voting result after the voting process to the voxel is completed for all the pixels of the celestial sphere HDR image by each of the three imaging devices 14, 15, and 16. The illumination area separation unit 23 extracts voxels having a vote count equal to or greater than a predetermined threshold as belonging to the voxel set v _l . Since the number of imaging devices is 3, here, the threshold for the number of votes (hereinafter sometimes referred to as “voting number threshold”) is 3. It is assumed that the number of votes from a single imaging device (any one of 14, 15, 16) to a certain voxel is 1 at maximum. Therefore, when the calculation result is such that voting from the plurality of pixels in the single imaging device to the same voxel occurs, the upper limit of the number of votes from within the single imaging device is set to 1 at the time of voting. Processing that restricts may be performed.
Here, the vote threshold value is set to 3, that is, the voxels voted by all the imaging devices are estimated to be illumination, but the vote threshold value is a value other than 3 (a numerical value other than the algebra of the imaging device). May be determined as appropriate.

Next, the illumination area separation unit 23 groups voxels belonging to the voxel set v _l . Specifically, the illumination area division section 23, the voxel _{v s (v s ∈v l)} , whether an element of an adjacent voxel _{v n} also set _{v l,} is namely _{v n} ∈ V _l It is determined whether or not. If v _n ∈v _l, it is determined that the voxels v _s and v _n belong to the same group. The illumination area separation unit 23 makes such a determination for all elements of the voxel set v _l . As a result, the voxels belonging to the voxel set v _l are classified into groups. Then, the illumination area separation unit 23 assigns a unique index to each voxel belonging to the voxel set v _l for each group. It is highly likely that such a group of voxels corresponds to one illumination (light source) in a three-dimensional space.

That is, the illumination area separation unit 23 specifies an illumination area in the three-dimensional space based on the pixel values of the pixels included in the acquired image. The process for specifying the illumination area is first performed by the above-mentioned voting in units of voxels. As a result, it is specified whether each voxel is an illumination area. Then, when the voxels identified as the illumination area are adjacent to each other, the illumination area separation unit 23 groups both as voxels belonging to a group corresponding to the same illumination. As a result, a group of consecutive voxels is identified as one illumination region.
In the process so far, it is estimated whether or not each voxel belongs to illumination, and the voxels belonging to illumination are grouped. Next, the illumination information estimation unit 24 estimates illumination information.

First, the illumination information estimation unit 24 assigns a luminance value corresponding to the voxel. The illumination information estimation unit 24 according to the present embodiment is calculated based on the pixel values of the pixels in each HDR camera (imaging devices 14, 15, and 16) that voted for all voxels belonging to the same group. All the luminance values of the voxels are averaged, and the average value is set as the luminance value of each voxel belonging to the group. That is, all voxels belonging to the same group have the same luminance value. Here, the illumination information estimation unit 24 stores the obtained luminance value in association with each voxel.
In addition, the illumination information estimation part 24 is based on the pixel value of the pixel in each HDR camera (imaging device 14, 15, 16) which voted instead of averaging the luminance value about all the voxels which belong to the same group. The luminance values of the calculated voxels may be averaged, and the average value may be used as the luminance value of the voxel. Also in this case, the illumination information estimation unit 24 stores the obtained luminance value in a form associated with each voxel.

Next, the illumination information estimation unit 24 adds a label for identifying each HDR camera only to the voxels located in front of the principal point of each HDR camera lens among the voxels belonging to each group. To do.
FIG. 4 is a schematic diagram showing an outline of processing for assigning a label corresponding to the HDR camera to some of the voxels belonging to a group with the illumination information estimation unit 24. The lattice shown in the figure shows a large number of voxels in a three-dimensional space. An actual voxel is a space in the form of a three-dimensional cube, but is projected and represented as a two-dimensional square in FIG. Also, Cam1 and Cam2 in the figure are one of the HDR cameras (HDR camera 81 or HDR camera 82) provided in the imaging devices 14 and 15, respectively. In the figure, the imaging device 16 is omitted. Of the voxels shown in the figure, the voxels marked with “◯”, the numbers “1” or “2”, or the alphabet “B” constitute one group as a whole. That is, voxels with these characters or symbols belong to a single group. Further, the voxel indicated by the numeral “1” or the alphabetic character “B” is the voxel existing in the foreground in this group as viewed from the camera Cam1. The illumination information estimation unit 24 assigns a label corresponding to the camera Cam1 to these voxels. Further, the voxel indicated by the numeral “2” or the alphabetic character “B” is the voxel existing in the foreground in this group when viewed from the camera Cam2. The illumination information estimation unit 24 assigns a label corresponding to the camera Cam2 to these voxels. That is, the voxel indicated by the letter “B” is the voxel that is present in the foreground, as viewed from the camera Cam1 and from the camera Cam2. The illumination information estimation unit 24 assigns both a label corresponding to the camera Cam1 and a label corresponding to the camera Cam2 to the voxels indicated by the letter “B”. A voxel marked with “◯” is a voxel belonging to the group, but is not present at the forefront when viewed from the camera Cam1 or the camera Cam2. Therefore, the illumination information estimation unit 24 does not assign a label corresponding to the camera Cam1 or a label corresponding to the camera Cam2 to the voxel marked with “◯”.

  Next, the illumination information estimation part 24 calculates | requires the following attribute value for every label (label corresponding to the imaging device 14,15,16) of each group. The attribute values are (A) barycentric position, (B) average luminance value, and (C) group size (number of voxels). In other words, the illumination information estimation unit 24 determines each label (this label is the (estimated) illumination of each group (this group is estimated to be one illumination (light source) as described above)). These attribute values are estimated as the diffuse reflection component illumination information.

The center-of-gravity position in (A) is the center of gravity of a set of voxels that belong to the group and have a specific label (that is, closest to the specific camera) in three-dimensional coordinates. Position. When obtaining the center of gravity, the weights of the voxels belonging to the set may be obtained as the same weight, or when different brightness values are assigned to the voxels, the brightness values may be obtained as weights. This barycentric position is a position representing the group. Therefore, this barycentric position may be referred to as a “representative position”. Although the representative position does not necessarily need to be the center of gravity position, it is desirable to set the center of gravity position as the representative position in the subsequent processing.
The average luminance value in (B) is an average luminance value of a set of voxels that belong to the group and are assigned a specific label. The average luminance value is calculated based on the luminance value given by the illumination information estimation unit 24 to each voxel.
The number of voxels in (C) is the number of voxels belonging to the group and given a specific label. The number of voxels is a value corresponding to the size of the group.

Note that the size of the group may be obtained by another method instead of the number of voxels in (C) above.
FIG. 5 is a schematic diagram showing an outline of a method for obtaining a group size by a method other than the number of voxels. The lattice shown in FIG. 5 is the same as the lattice shown in FIG. Further, the position of the camera Cam1 shown in FIG. 5 is the same as the position of the camera Cam1 shown in FIG. In FIG. 5, the camera Cam2 is omitted. Further, in FIG. 5, among the voxels shown in FIG. 4, only the voxels with the numeral “1” or the alphabet “B” are left, and the numerals “2” and “O” are marked. There is nothing written about the voxels that showed. That is, FIG. 5 shows a case in which the letter “1” or “B” is written only in the voxel existing closest to the camera Cam1 among the voxels belonging to the group. In FIG. 5, point C is the position of the center of gravity of the voxel in which the characters “1” or “B” are marked. A plane P (a straight line P because it is projected in the figure) is a plane perpendicular to a straight line connecting the camera Cam1 and the center of gravity C and passes through the center of gravity C. Further, in this plane P, the range from the point P1 to the point P2 is a plane image obtained by parallel projecting a set of voxels marked “1” or “B” on the plane P. And this method Then, the number of pixels occupied by the planar image in the image obtained by photographing the planar image (the image shown in the range from P1 to P2 and having the depth in FIG. 5) by the camera Cam1 is the size value of the group. To do.

The illumination information estimation unit 24 uses (A) the position of the center of gravity, (B) the average luminance value, and (C) the size of the group, obtained as above for each group and each camera label, as the diffuse reflection component illumination information. Output.
That is, the primary key of the diffuse reflection component illumination information is a composite key in which the group index and the camera label are combined. The attribute values for the combination of the group index and the camera label are (A) the barycentric position, (B) the average luminance value, and (C) the size of the group.

The procedure for acquiring the diffuse reflection component illumination information described above is summarized as follows.
FIG. 6 is a flowchart illustrating a procedure of processing for acquiring the diffuse reflection component illumination information. This flowchart shows processing corresponding to one frame period of the video imaged by the imaging devices 14, 15 and 16. That is, the illumination information acquisition apparatus 1 repeats the process of this flowchart for every frame period. Hereinafter, the processing procedure will be described along the flowchart.

  First, in step S <b> 11, the illumination area separation unit 23 performs voting for voxels whose luminance values exceed a threshold value based on the pixel values of the pixels of the camera included in the imaging device. Next, in step S12, the illumination area separation unit 23 evaluates the result of the vote performed in the previous step. That is, the illumination area separation unit 23 determines (estimates) whether each voxel is a voxel belonging to illumination. Next, in step S13, the illumination region separation unit 23 groups the voxels estimated to be illumination, and assigns an index to each group. Specifically, in this step, when the voxels estimated to be illumination are adjacent to each other, the illumination region separation unit 23 determines that these voxels belong to the same group.

  Next, in step S14, the illumination information estimation unit 24 calculates the luminance value of the voxel. Next, in step S15, the illumination information estimation unit 24 applies the camera label to the voxel according to the appearance of the illumination from each camera (that is, each group to which the voxel estimated to be illumination belongs). Is granted. Next, in step S16, the illumination information estimation unit 24 calculates the attribute value of each group. The estimated value of each group is the diffuse reflection component illumination information acquired in the present embodiment. Next, in step S <b> 17, the illumination information estimation unit 24 outputs the diffuse reflection component illumination information obtained by calculation.

(2) Acquisition of Specular Reflection Component Illumination Information Next, a method for acquiring specular reflection component illumination information will be described.
The LDR image separation unit 41 extracts a low dynamic range (LDR) image from the three omnidirectional HDR images corresponding to each imaging device acquired by the video acquisition unit 21. At this time, the LDR image separation unit 41 applies the above-described function f ⁻¹ (x) to each pixel value in the HDR image in order to extract an LDR image with pixel values excluding the camera characteristics. Then, the LDR image separation unit 41 processes a pixel whose value as a result of applying the function f ⁻¹ (x) exceeds the range that can be represented as an LDR image as saturated. In the following description regarding extraction of an LDR image, the pixel value is a value after applying the function f ⁻¹ (x).

  For example, when the range of pixel values by the HDR camera is 0 to 16777215 and the range that can be captured without being saturated by the LDR camera is 256 to 65535 of the above range 0 to 16777215, the LDR image separation unit 41 Perform the following process. That is, for a pixel whose pixel value is 0 to 255 in the HDR camera, the LDR image separation unit 41 processes the pixel as if the pixel value in the HDR camera is 256. In addition, regarding the pixels having pixel values of 65536 to 16777215 in the HDR camera, the LDR image separation unit 41 processes the pixels assuming that the pixel value of the HDR camera is 65535. Similarly, when the range of pixel values of the HDR camera and the LDR camera is different from the example given here, the LDR image separation unit 41 can also express gradation information that can be expressed as an HDR image but cannot be expressed as an LDR image. Cut. Also, the LDR image separation unit 41 appropriately adjusts the gradation quantization width as necessary to generate an LDR image.

  The LDR image separation unit 41 outputs the omnidirectional LDR image corresponding to each imaging device by performing the above processing on each frame of each omnidirectional image captured by the imaging devices 14, 15, and 16. This omnidirectional LDR image is illumination information for specular reflection components.

The compression unit 50 compresses and encodes the diffuse reflection component illumination information and the specular reflection component illumination information obtained by the above method. The compression unit 50 compresses the diffuse reflection component illumination information and the specular reflection component illumination information by a method corresponding to each. Specifically, the compression unit 50 always performs lossless compression on the diffuse reflection component illumination information. Further, the specular reflection component illumination information may be reversibly compressed or irreversibly compressed. The compression unit 50 uses, for example, LZSS encoding when performing information compression of the diffuse reflection component illumination information output by the illumination information estimation unit 24. Note that LZSS encoding itself is a process that can be realized by existing technology. In addition, the compression unit 50 performs compression processing by, for example, JPEG (Joint Photographic Experts Group) when performing information compression of the specular reflection component illumination information output by the LDR image separation unit 41.
Further, the compression unit 50 is, for example, H.264. The specular reflection component illumination information is encoded as a moving image by a moving image encoding method such as H.264, and the diffuse reflection component illumination information is embedded as metadata, and both can be compressed and encoded together. .

The illumination information compressed and encoded by the compression unit 50 is stored in a recording medium or transmitted to an external device for use in CG rendering processing.
The storage unit 52 includes a hard disk drive that stores information using a magnetic recording medium or an SSD (solid state drive) using a semiconductor memory, and sequentially stores illumination information output from the compression unit 50.
The transmission unit 53 includes a communication unit, and transmits illumination information output from the compression unit 50 to an external device.
The illumination information accumulated in the accumulation unit 52 or transmitted by the transmission unit 53 is used by an illumination restoration device or the like described later. The lighting restoration device uses the acquired lighting information to perform rendering of CG that naturally matches the lighting situation when the video is shot.

  FIG. 7 is a block diagram illustrating a schematic functional configuration of an illumination restoration device that restores the illumination status using illumination information, and a synthesis device that synthesizes a CG and a camera video (actual video). As illustrated, the illumination restoration device 3 includes an illumination information acquisition unit 71, a decoding unit 72, a diffuse reflection component illumination information extraction unit 75, a diffuse reflection component CG drawing unit 76, an LDR image extraction unit 78, An interpolation image generation unit 79 (luminance information compensation unit), a specular reflection component CG drawing unit 80 (specular reflection component drawing unit), a camera parameter supply unit 85, and an addition unit 86 are included. The synthesizing device 5 includes a camera video input unit 87 and a video synthesizing unit 88.

  The illumination restoration device 3 renders the object in a three-dimensional CG space based on the data of the CG object prepared separately. At this time, the illumination restoration device 3 performs CG rendering using the above-described illumination information for each of the diffuse reflection component and the specular reflection component. The function of each part will be described below.

  The illumination information acquisition unit 71 acquires illumination information (for diffuse reflection component and specular reflection component) from the illumination information acquisition device 1. Specifically, the illumination information acquisition unit 71 reads the illumination information stored in the recording medium or the like in the storage unit 52 of the illumination information acquisition device 1 or is transmitted from the transmission unit 53 of the illumination information acquisition device. Receive. When receiving the illumination information transmitted from the transmission unit 53, the illumination information acquisition unit 71 has a communication function for performing communication with the transmission unit 53. The illumination information acquired by the illumination information acquisition unit 71 is information compressed and encoded by the compression unit 50 of the illumination information acquisition device 1.

  The decoding unit 72 decodes the illumination information acquired by the illumination information acquisition unit 71. Thereby, the decoding unit 72 expands the compressed illumination information.

(1) Diffuse reflection component image The diffuse reflection component illumination information extraction unit 75 extracts the diffuse reflection component illumination information from the illumination information decoded by the decoding unit 72.

  The diffuse reflection component CG drawing unit 76 performs CG rendering (drawing) on the diffuse reflection component, and outputs a diffuse reflection component image. Based on the illumination information extracted by the diffuse reflection component illumination information extraction unit 75, the diffuse reflection component CG drawing unit 76 generates the original three-dimensional space (the illumination information acquisition device 1 obtains the illumination information) in the three-dimensional space of the CG. A point light source or a surface light source is arranged at the same position as the acquired three-dimensional space. The rendering process itself is performed using existing technology.

When the illumination is arranged as a point light source, the diffuse reflection component CG drawing unit 76 sets the position of the center of gravity of each illumination (the group of voxels described above) as the position of the point light source. Further, the diffuse reflection component CG rendering unit 76 sets the brightness of the point light source to a value obtained by multiplying the average luminance value of the illumination (the group) by the number of voxels of the group.
On the other hand, when the illumination is arranged as a surface light source, the diffuse reflection component CG drawing unit 76 sets a sphere centered on the center of gravity of each illumination as the surface light source. At this time, the diffuse reflection component CG drawing unit 76 sets the radius of the sphere that is the light source based on the size (for example, the number of voxels) of the (C) group obtained from the illumination information. The radius is calculated by the following equation (3) based on the size of the group included in the illumination information.

  In this equation, S is a value representing the size of the group and is the number of voxels. When the size of the group is expressed by the number of voxels, the average value of the number of voxels by each of the three cameras is taken for the group, and the average value is S. Further, π is a circumference ratio. R is the radius of the surface light source to be set. The unit of r is the length of one side of the voxel. Even when the group size is not represented by the number of voxels, the radius of the sphere is obtained by an appropriate method.

  The diffuse reflection component CG rendering unit 76 renders only the diffuse reflection component of the CG object using the diffuse reflection component of illumination set as described above, and outputs the result as a diffuse reflection component image.

(2) Specular Reflection Component Image The LDR image extraction unit 78 extracts an LDR image (specular reflection component illumination information) from the illumination information decoded by the decoding unit 72. As described above, this LDR image corresponds to each imaging device and is an image for each frame period.

  The interpolated image generation unit 79 restores the original three-dimensional space environment by performing EPI (Epipolar-Plane Image) analysis based on the LDR images captured by the three cameras. . The EPI analysis itself is an existing technology. Then, the interpolated image generation unit 79 obtains the centroid position of the CG model to be rendered, and obtains the omnidirectional image at the centroid position by interpolation processing based on the three LDR images (omnidirectional images). Generate. This interpolation process itself can be performed using an existing technique (such as View Morphing). Image interpolation processing using this technique should be taken by a virtual camera placed at an arbitrary position by geometric processing based on images taken by multiple cameras placed at different positions. The image is generated.

In addition, the interpolated image generation unit 79 performs a process of supplementing the luminance information using the diffuse reflection component illumination information with respect to an area where the luminance is saturated in the image generated by the above interpolation process. Specifically, the interpolated image generation unit 79 acquires the diffuse reflection component illumination information, and based on the diffuse reflection component illumination information, obtains a luminance distribution that has a Gaussian distribution with the center of gravity of the light source as the center point. Generate. Then, the interpolated image generation unit 79 cuts the luminance distribution in the saturated region, then scales the luminance distribution to match the value obtained by multiplying the average luminance by the number of pixels, and obtains the luminance distribution obtained thereby. A process of fitting into the saturation region is performed. That is, the interpolated image generation unit 79 acquires a low dynamic range image having a predetermined dynamic range as specular reflection component illumination information, and based on a high dynamic range image having a relatively higher dynamic range than the low dynamic range image. Information on the representative position, luminance, and size of the illumination area in the estimated three-dimensional space is acquired as diffuse reflection component illumination information, and the portion where the luminance is saturated in the low dynamic range image Based on the information on the luminance and the size included in the diffuse reflection component illumination information having a representative position corresponding to the portion, a process of restoring the luminance in a pseudo manner is performed.
As described in Expression (1), the function f (x) is a monotonically increasing function. That is, under the condition that the distance from the illumination to the camera does not change, the increase / decrease of the pixel value of the pixel in the image data and the increase / decrease of the luminance of the subject have a monotonous relationship. Therefore, the luminance information compensation processing performed by the interpolated image generation unit 79 here is processing for compensating luminance information of the subject estimated to be illumination. That is, the interpolated image generation unit 79 has a function of supplementing (pseudo-reconstructing) illumination luminance information.

  By this processing of the interpolated image generating unit 79, the luminance information of the saturated region that is missing because the LDR image is used as the specular reflection component illumination information is restored in a pseudo manner. Therefore, a more natural specular reflection component image can be generated as compared with a case where an LDR image in which the saturated region is simply saturated is used.

  The specular reflection component CG drawing unit 80 performs CG rendering (drawing) on the specular reflection component. Specifically, the specular reflection component CG rendering unit 80 uses the specular reflection component obtained by the processing by the LDR image extraction unit 78 and the interpolated image generation unit 79 as a global illumination, and uses the specular reflection of the CG object. Render only the components. That is, the specular reflection component CG drawing unit 80 draws the specular reflection component by computer graphics using the specular reflection component illumination information whose luminance is restored by the interpolated image generation unit 79.

  The camera parameter supply unit 85 acquires parameters relating to the video camera and supplies the parameters to the diffuse reflection component CG drawing unit 76 and the specular reflection component CG drawing unit 80 that perform the rendering process. This video camera is a camera (not shown in FIG. 1) and takes a video. The video itself shot by this video camera is input to the camera video input unit 87 of the synthesizer 5. Here, the camera parameters are the position of the camera in the aforementioned three-dimensional space, the posture of the camera (such as the pan angle and tilt angle in the coordinate system of the aforementioned three-dimensional space), and lens information (such as field angle information). These parameters are grasped in the environment where the video is shot, and transmitted to the camera parameter supply unit 85 through a transmission path (not shown). When a plurality of video cameras are used and videos shot by each video camera are switched and used, the camera parameter supply unit 85 acquires camera parameters related to each of the video cameras and performs switching (switching). ) Information.

  The adding unit 86 simply adds the CG drawn by the diffuse reflection component CG drawing unit 76 and the CG drawn by the specular reflection component CG drawing unit 80 and outputs the result.

  The camera video input unit 87 acquires the video shot by the video camera described above and supplies it to the video synthesis unit 88.

  The video composition unit 88 synthesizes and outputs the CG video output from the illumination restoration device 3 and the camera video supplied from the camera video input unit 87 as a video. As a result, the video output from the video composition unit 88 is a video as if a CG object exists in the camera video (actual shooting). Since the lighting restoration device 3 performs CG rendering in accordance with the lighting condition of the camera video, the video and the CG object coexist in the synthesized video without any sense of incongruity with respect to light rays or the like.

[Second Embodiment]
Next, a second embodiment of the present invention will be described. In the following, description of items similar to those of the above-described embodiment will be omitted, and items specific to the present embodiment will be mainly described.
The functional configuration of the illumination information acquisition apparatus according to the present embodiment is substantially the same as that shown in FIG. However, in the present embodiment, a part of the method for obtaining the diffuse reflection component illumination information is different from the method for obtaining in the first embodiment. Specifically, the method for obtaining the average luminance value (B) for each group and for each camera label, which is obtained by the illumination information estimation unit 24 in the present embodiment, is different from the method for obtaining in the first embodiment.

  The method by which the illumination information estimation unit 24 according to the present embodiment obtains (B) average luminance value is as follows. When the (A) barycentric position for each camera label of each voxel group (a group of a plurality of voxels estimated to be a single illumination) is obtained, the illumination information estimation unit 24 then determines the total sky from the camera. A sphere image (HDR image) is divided into Voronoi. When performing Voronoi division, the center of gravity of each voxel group is used as a generating point. The Voronoi division is a division in which, when a plurality of generating points arbitrarily arranged in a space are given, the region is divided depending on which generating point is close to points other than the generating point in the space. That is, in the plane space, the boundary line of the region by Voronoi division is a vertical bisector connecting a certain segment and a neighboring segment.

FIG. 8 is a schematic diagram showing an example of the above Voronoi division.
FIG. 6A shows an image (a part of the omnidirectional image) taken by a camera of a certain imaging device (14, 15, or 16) in a three-dimensional space (video recording studio) that is a video shooting site. An example in which Voronoi division is performed using the center of gravity of the illumination as a generating point is shown. In FIG. 6A, the part marked with a circle is the position estimated to be the center of gravity of the illumination. The result of dividing the region with these positions as a reference is shown in FIG.

  FIG. 6B shows one region obtained by Voronoi division extracted from FIG. In the figure, reference numeral 101 denotes one of the boundary lines surrounding the area. The area is surrounded by four straight lines (boundary lines). Reference numeral 102 denotes a line surrounding the area estimated to be illumination in the area. This area shows one group formed by a plurality of voxels estimated to be illumination. Reference numeral 103 denotes the barycentric position of the voxel assigned the same label (camera label) in the group.

And the illumination information estimation part 24 performs the following process about each area | region by which Voronoi division was carried out. That is, the illumination information estimation unit 24 provides a plane that is perpendicular to the straight line connecting the principal point of the camera lens and the centroid position and passes through the centroid position. Then, the illumination information estimation unit 24 applies the function f ⁻¹ (x) to the pixel value of each pixel included in the Voronoi region, and further calculates the distance r from the principal point of the lens to the centroid position. The value multiplied by the square is obtained as the luminance. And the average luminance value about the voxel whose vote number in the above-mentioned vote was more than a vote number threshold value is calculated | required. The illumination information estimation unit 24 sets the average luminance value obtained as described above as the average luminance value of (B) regarding the camera of the group.

The illumination information acquisition apparatus according to the present embodiment outputs information on the average luminance value obtained by the above method as a part of the diffuse reflection component illumination information. The illumination restoration device according to the present embodiment restores illumination using such diffuse reflection component illumination information and performs CG rendering.
According to the present embodiment, more accurate illumination reproducibility can be obtained.

  In addition, you may make it implement | achieve the function of the illumination information acquisition apparatus in each embodiment mentioned above, an illumination decompression | restoration apparatus, and a synthetic | combination apparatus with a computer. In that case, the program for realizing these functions may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible disk, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, a “computer-readable recording medium” dynamically holds a program for a short time, like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system serving as a server or a client in that case may be included, and a program that holds a program for a certain period of time. The program may be a program for realizing a part of the above-described functions, or may be a program that can realize the above-described functions in combination with a program already recorded in a computer system.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

  The present invention is particularly applicable to industries that produce video content. It can also be used in video games, devices or computer programs for realizing virtual reality.

DESCRIPTION OF SYMBOLS 1 Illumination information acquisition apparatus 3 Illumination restoration apparatus 5 Composition apparatus 14, 15, 16 Imaging device 21 Image | video acquisition part (image acquisition part)
22 Spatial information setting unit 23 Illumination region separation unit 24 Illumination information estimation unit 41 LDR image separation unit (image output unit)
DESCRIPTION OF SYMBOLS 50 Compression part 52 Accumulation part 53 Transmission part 71 Illumination information acquisition part 72 Decoding part 75 Diffuse reflection component illumination information extraction part 76 Diffuse reflection component CG drawing part 78 LDR image extraction part 79 Interpolation image generation part (luminance information compensation part)
80 Specular reflection component CG drawing unit (Specular reflection component drawing unit)
85 Camera parameter supply unit 86 Addition unit 87 Camera image input unit 88 Image composition unit

Claims (6)

An image acquisition unit for acquiring images taken by cameras provided at a plurality of positions in a three-dimensional space;
An illumination area separation unit that identifies an illumination area in the three-dimensional space based on pixel values of pixels included in the acquired image;
An illumination information estimation unit that estimates a representative position, a luminance value, and a size as illumination information for diffuse reflection components for each of the illumination regions specified by the illumination region separation unit;
An image output unit that outputs the image acquired by the image acquisition unit as specular reflection component illumination information;
An illumination information acquisition apparatus comprising: The image output unit outputs the image after performing a process of reducing a dynamic range of luminance of the image acquired by the image acquisition unit.
The illumination information acquisition apparatus according to claim 1. The illumination area separation unit determines, for each position of the camera that captured the image, whether or not illumination is reflected in the pixel based on a pixel value of the pixel included in the image, and based on the determination result Voting on a region in the direction corresponding to the pixel included in the three-dimensional space, and determining whether the region is the illumination region according to the number of votes obtained by the region,
The illumination information acquisition apparatus according to claim 1, wherein the illumination information acquisition apparatus is an illumination information acquisition apparatus. A low dynamic range image with a predetermined dynamic range is acquired as illumination information for specular reflection components, and in a three-dimensional space estimated based on a high dynamic range image with a higher dynamic range than the low dynamic range image Information on the representative position, luminance, and size of the illumination area is acquired as diffuse reflection component illumination information, and a portion where the luminance is saturated in the low dynamic range image has a representative position corresponding to the portion. A luminance information compensation unit that performs a process of restoring luminance in a pseudo manner based on the information on the luminance and the size included in the illumination information for diffuse reflection component;
A specular reflection component drawing unit that draws a specular reflection component by computer graphics using the illumination information for specular reflection component whose luminance is restored by the luminance information compensation unit;
An illumination restoration apparatus comprising:   A program for causing a computer to function as the illumination information acquisition device according to any one of claims 1 to 3.   A program for causing a computer to function as the illumination restoration device according to claim 4.

Images