JP5795556B2 - Shadow information deriving device, shadow information deriving method and program - Google Patents

Description

  The present invention relates to a technique for deriving shadow information.

  In augmented reality, for example, when a CG (computer graphics) object such as a virtual object is combined with a scene image, a shadow is added to the CG object. In this way, by adding a shadow corresponding to the CG object, it is possible to make it appear as if the CG object actually exists in the scene image.

  As described above, shadow information is used to add a shadow. The shadow information is information indicating how the shadow generated when the entity of the CG object is located in the actual scene in correspondence with the synthesis position of the CG object in the scene image is expressed in the scene image. is there.

The derivation of the shadow information is realized, for example, by estimating information (light source information) about the light source in the scene and rendering the CG object based on the light source information.
In estimating the light source information, for example, an omnidirectional camera that captures an omnidirectional image is used (see, for example, Non-Patent Document 1 and Non-Patent Document 2). This method is based on the premise that a light source is imaged in any part of an image obtained by imaging all directions. In addition, this method estimates the position, color, position, intensity, and the like of the light source corresponding to the scene image based on the image captured by the omnidirectional camera.

Further, a technique using a metal sphere is known (for example, see Non-Patent Document 3). This method is based on the idea according to the above-mentioned omnidirectional camera. That is, this method is based on the premise that a light source is also included in an image of a surrounding scene reflected on a metal sphere. In addition, this method estimates the position, color, position, intensity, and the like of the light source based on the image of the metal sphere in which the surrounding scene is reflected.
In addition, a method using a white sphere having a diffuse reflection surface is known (for example, see Non-Patent Document 4). In this method, the color, intensity, direction, and the like of the light source around the white sphere are estimated based on the shadow appearing in the white sphere image.
Furthermore, a method of estimating a light source distribution based on a shadow of an object whose reflection characteristics are known on a known plane is also known (see, for example, Non-Patent Document 5).

Imari Sato, Yoichi Sato, and Katsushi Ikeuchi, Acquiring a radiance distribution to superimpose virtual objects onto a real scene, IEEE Transactions on Visualization and Computer Graphics, Volume 5, Issue 1, pages 1-12, 1999. Imari Sato, Morihiro Hayashida, Kaiyo, Yoichi Sato, Katsushi Ikeuchi, Image Generation under Real Light Environment: High-speed rendering method based on linear sum of basic images, IEICE Transactions. D-II, Information / System, II-Pattern processing J84-D-II (8), pages 1864-1872, 2001. Kusuma Agusanto, Li Li, Zhu Chuangui, and Ng Wan Sing, Photorealistic rendering for augmented reality using environment illumination, Proceedings of the Second IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR), pages 208-216, 2003. Miika Aittala, Inverse lighting and photorealistic rendering or augmented reality, The Visual Computer, Volume 26, Issue 6, pages 669-678, 2010. Imari Sato, Yoichi Sato, Katsushi Ikeuchi, Estimation of light source environment based on object shadow, Journal of Information Processing Society of Japan: Computer Vision and Image Media, Volume 41, Number SIG 10 (CVIM 1), pages 31-40, 2000.

An omnidirectional camera is required when using the methods described in Non-Patent Documents 1 and 2. This omnidirectional camera is a special device including, for example, a fisheye lens, and is not easily available to a general end user called a consumer.
In the case of the methods according to Non-Patent Documents 3 and 4, it is necessary to prepare a metal sphere or a white sphere having a diffuse reflection surface. Such metal spheres and white spheres are also special as instruments and are not readily available to end users.
In the case of the method according to Non-Patent Document 5, the shape of the object and the reflection characteristics of the plane need to be known. In this case, the user must measure the shape and reflection characteristics of these objects, and this requires a certain level of expertise.
As described above, any method according to Non-Patent Documents 1 to 5 requires, for example, an uncommon apparatus or instrument, or requires specialized knowledge. Therefore, it is difficult to provide an end user with an environment in which, for example, a CG object having a shadow in augmented reality can be combined with a scene image by any of the methods of Non-Patent Documents 1 to 5.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide a technique that can provide a user with an environment in which a shaded CG object can be easily combined with a scene image.

  In order to solve the above-described problem, a shadow information deriving device according to one aspect of the present invention provides a normal map of a scene indicated by the scene image based on a depth map corresponding to a scene image having image content as a scene. A normal map deriving unit for deriving, a reference object region extracting unit for extracting a reference object region that is a region portion of a reference object based on at least one of the scene image, the depth map, and the normal map; A shadow information deriving unit for deriving shadow information based on the scene image, the normal map, and the reference object region.

  The shadow information deriving device of the present invention further includes a shadow information regularization unit that corrects a predetermined parameter used to derive the shadow information so that the regularized shadow information is derived.

  The shadow derivation method as one aspect of the present invention includes a normal map derivation step of deriving a normal map of a scene indicated by the scene image based on a depth map corresponding to a scene image having image content as a scene; A reference object region extracting step of extracting a reference object region that is a region portion of a reference object based on at least one of the scene image, the depth map, and the normal map; and the scene image and the normal vector A shadow information deriving step for deriving shadow information based on the map and the reference object region.

  The shadow information deriving device of the present invention further includes a shadow information regularization step for correcting a predetermined parameter used for deriving the shadow information so that the regularized shadow information is derived.

  A program as one aspect of the present invention is for causing a computer to function as the above-described shadow information deriving device.

  As described above, according to the present invention, it is possible to provide the user with an environment in which a shaded CG object can be easily combined with a scene image.

It is a figure which shows the structural example of the shadow information derivation | leading-out apparatus in the 1st Embodiment of this invention. It is a figure which shows an example of a scene image. It is a figure which shows an example of a depth map. It is a figure which visualizes and shows a normal map. It is a figure which shows the extraction result of a reference object area | region. It is a figure which visualizes and shows the spherical harmonic function utilized in this embodiment. It is a figure which visualizes and shows the shadow information in this embodiment. It is a figure which shows an example of the CG object which provided the shadow by the shadow information. It is a figure which shows an example of the scene image with which the CG object which provided the shadow by the shadow information was synthesize | combined. It is a figure which shows the example of a process sequence which the shadow information derivation | leading-out apparatus in 1st Embodiment performs. It is a figure which shows the structural example of the shadow information derivation | leading-out apparatus in 2nd Embodiment. It is a figure which shows an example of a scene image. It is a figure which shows an example of a depth map. It is a figure which visualizes and shows the shadow information which is not regularized by the shadow information regularization part of this embodiment. It is a figure which visualizes and shows the shadow information regularized by the shadow information regularization part of this embodiment. It is a figure which shows the example of a process sequence which the shadow information derivation | leading-out apparatus in 2nd Embodiment performs.

<First Embodiment>
[Configuration example of shadow information deriving device]
The shadow information deriving device according to the first embodiment of the present invention will be described below. The shadow information deriving device of this embodiment derives shadow information by inputting a scene image and a depth map for the scene image.

  In the present embodiment, the shading information represents in the scene image the shading that occurs when the entity of the CG object is located in the actual scene in correspondence with the synthesis position of the CG (computer graphics) object in the scene image. It is information for. This shadow information is used to give a shadow according to the position of the CG object and the illumination of the scene in the scene image obtained by combining the CG objects. By giving the shadow in this way, the CG object looks natural in the scene image, and the realism of the scene image combined with the CG object is improved.

  FIG. 1 shows a configuration example of a shadow information deriving device 100 in the first embodiment. The shadow information deriving device 100 shown in this figure includes a scene image input unit 101, a depth map input unit 102, a normal map deriving unit 103, a reference object region extracting unit 104, and a shadow information deriving unit 105. Note that these functional units are realized by, for example, a CPU (Central Processing Unit) executing a program. The shadow information deriving device 100 includes devices such as a storage unit 106, a display unit 107, and an operation unit 108.

The scene image input unit 101 inputs a scene image 201 having an image content as a scene from the outside, and stores it in the storage unit 106.
The scene image 201 in the present embodiment may be an image obtained by imaging a certain real environment, for example.
FIG. 2 shows an example of the image content of the scene image 201 obtained by imaging a certain real environment as described above.

  The depth map input unit 102 inputs a depth map 202 corresponding to the scene image 201 from the outside, and stores it in the storage unit 106. The depth map 202 is a pixel value (depth value) indicating a distance from a viewpoint to a subject such as a subject or a background included in the scene space in the scene image 201, and a pixel value (pixel value) for each pixel arranged on a two-dimensional plane ( Luminance). The pixels that form the depth map 202 correspond to the pixels that form the scene image 201.

  FIG. 3 shows an example of the depth map 202 corresponding to the scene image 201 of FIG. In the depth map 202 shown in this figure, the subject at the position corresponding to each pixel indicates that the distance from the shooting position (viewpoint) increases as the luminance value increases.

  It should be noted that the scene image 201 input by the scene image input unit 101 and the depth map 202 input by the depth map input unit 102 are obtained by shooting with an RGB camera and a depth sensor for consumers (end users). Can do. As one of such imaging apparatuses, for example, Microsoft Corporation's Kinect (registered trademark) is known.

  The normal map deriving unit 103 derives a normal map of the scene indicated by the scene image 201 based on the depth map 202 stored in the storage unit 106. The normal map here is data in which normals of the object surface regularized for each pixel are stored.

For this purpose, the normal map deriving unit 103 reads the depth map 202 from the storage unit 106. Next, the normal map deriving unit 103 uses the coordinates (pixel coordinates) and depth values (pixel values) of the pixels forming the depth map 202 to determine the position (pixel coordinates) of the three-dimensional vertex in the scene. calculate. Next, the normal map deriving unit 103 pays attention to one of the calculated three-dimensional vertices, and horizontally adjoins a vector from the noted three-dimensional vertex to the vertically adjacent three-dimensional vertex in the depth map 202. Find a vector going to the 3D vertex. Then, the normal map deriving unit 103 derives the normal at the target vertex by obtaining the outer product of these two vectors.
The normal map deriving unit 103 obtains a normal for each pixel by the above procedure, and generates a normal map 203 using these normals. As described above, the normal map deriving unit 103 derives the normal map 203. Then, the normal map deriving unit 103 stores the derived normal map 203 in the storage unit 106.

  As a specific example, the normal map deriving unit 103 obtains a normal n (u, v) at pixel coordinates (u, v) by the following equation (1).

  Depending on equation (1), a normal n (u, v) as a normal vector is obtained. In Expression (1), d (u, v) is a depth value (depth) in (u, v), and K is an internal parameter matrix of an imaging device (depth sensor) that captures the depth map 202. This internal parameter matrix K is a matrix for converting real world coordinates on the imaging plane imaged by the imaging device into pixel coordinates. The origin of the real world coordinates on the imaging plane is the imaging position, and the depth of the real world coordinates on the imaging plane corresponds to the depth value for each pixel indicated by the depth map 202.

  FIG. 4 shows an example in which the normal map 203 derived from the depth map 202 shown in FIG. 3 is visualized by an image. The normal map 203 shown in this figure is actually a color image expressed in RGB. In addition, the normal map 203 shown in this figure has, for example, the values of the three-dimensional normal vector components obtained for each pixel coordinate as described above, and the values of the RGB components of the pixels of the corresponding pixel coordinates, respectively. Has been converted. That is, the image of the normal map 203 in FIG. 4 is visualized so that the value of the normal vector component for each pixel is expressed by color.

  The reference object region extraction unit 104 extracts a reference object region that is a region portion of the reference object based on at least one of the scene image 201, the depth map 202, and the normal map 203. The reference object is an object that the shadow information deriving unit 105 refers to when acquiring light source information at a position where the CG object is arranged in the scene.

The reference object area extraction unit 104 extracts the reference object area as follows, for example. That is, the reference object region extraction unit 104 reads any one of the scene image 201, the depth map 202, and the normal map 203 from the storage unit 106 as extraction target data. The reference object region extraction unit 104 displays the image of the extraction target data read in this way on the display unit 107, for example.
The user determines an object to be a reference object from among the subjects of the displayed image. At this time, the user may determine, as a reference object, an object that is as close as possible to the CG object to be combined with the scene image and that has a clear shadow as much as possible. Then, the user performs a predetermined operation on the operation unit 108 for selecting a region where the reference object is displayed from the displayed image.
The reference object region extraction unit 104 extracts the region selected by the above operation as the reference object region 204 from the extracted extraction target data. The reference object area extraction unit 104 stores the extracted reference object area 204 in the storage unit 106.

  Alternatively, the reference object region extraction unit 104 may extract the reference object region 204 as follows. That is, the reference object region extraction unit 104 reads the scene image 201, the depth map 202, and the normal map 203 from the storage unit 106 as extraction target data. Then, the reference object region extraction unit 104 causes the display unit 107 to display any one of the read scene image 201, depth map 202, and normal map 203, for example. As an example, since the scene image 201 is most easily seen by the user, the reference object region extraction unit 104 may display the scene image 201.

In this case, the user performs an operation on the operation unit 108 for designating a partial area in the area determined as the reference object from the displayed image.
The reference object region extraction unit 104 individually sets a threshold value for each of the scene image 201, the depth map 202, and the normal map 203 that is extraction target data. In addition, the reference object region extraction unit 104 searches for adjacent pixels for each of the scene image 201, the depth map 202, and the normal map 203 with the representative point determined in the partial region designated by the operation as a starting point. To go.

  Then, the reference object region extraction unit 104 determines that, in the search process, in any one of the scene image 201, the depth map 202, and the normal map 203, the difference in pixel value between adjacent pixels is equal to or greater than a threshold value. The boundary between the adjacent pixels is determined as a boundary portion between the reference object region and the other region. Then, the reference object region extraction unit 104 synthesizes the boundary portion determined as described above among the scene image 201, the depth map 202, and the normal map 203, thereby obtaining pixel coordinates of the contour as the reference object region. Is identified. Even in this way, the reference object region extraction unit 104 can extract the reference object region 204.

Note that the reference object region extraction unit 104 performs processing for setting the threshold value in the data and extracting the reference object region 204 as described above, for example, two or less of the scene image 201, the depth map 202, and the normal map 203. A combination of the above may be performed. However, better to use all of the scene image 201 and depth map 202 and normal map 203, it is possible to identify the boundary between the reference object region 204 and the background by the combination of all the information of the luminance and depth value and the normal direction, The extraction accuracy of the reference object area 204 is increased.

  FIG. 5 is a diagram illustrating an extraction result of the reference object region 204 by the reference object region extraction unit 104 as an image. In this figure, a portion corresponding to the reference object region 204 is a portion that is white in the black background image. This white portion is, for example, the left upper arm portion of the person in the scene image of FIG.

The shadow information deriving unit 105 derives shadow information based on the scene image 201, the normal map 203, and the reference object region 204.
As a specific example, the shadow information deriving unit 105 derives shadow information as follows.

  First, the shadow information deriving unit 105 reads the scene image 201, the normal map 203, and the reference object region 204 from the storage unit 106.

  The shadow information deriving unit 105 assigns a number n (n = 1... N) to each pixel included in the read reference object region 204. Note that the correspondence between the arrangement of the pixels included in the reference object region 204 and the number n does not affect the calculation result, and therefore the rule for assigning the number n to the pixels included in the reference object region 204 is not particularly limited.

  The shadow information deriving unit 105 uses a spherical harmonic function to derive the shadow information. By using the spherical harmonic function, there is an advantage that, for example, the problem does not become complicated even when the number of light sources is increased, and the shadow of diffuse reflection can be expressed with high accuracy by a small number of bases.

  The spherical harmonic function is a function with respect to spherical coordinates (θ, φ), and is defined by the following equation (2).

  In the formula (2), l and m are integers, for example, l ≧ 0 and −1 ≦ m ≦ 1. l represents the order. That is, in this example, nine spherical harmonic functions of the third order of “0 ≦ l ≦ 2” are used. Klm is a regularization coefficient, and Plm is a Legendre power function.

  In the present embodiment, the robustness in deriving a diffuse reflection shadow from an object having an unknown reflection characteristic is improved by limiting the spherical harmonic functions to a certain number or less, for example, nine as described above. be able to.

  FIG. 6 shows an example of nine spherical harmonic functions corresponding to the reference object region 204 shown in FIG.

  The shadow information deriving unit 105 sets a coefficient (weight) for each spherical harmonic function when integrating the nine spherical harmonic functions expressed as shown in FIG. For this purpose, for example, the shadow information deriving unit 105 obtains a weight w that minimizes an error function E (w) expressed by the following equation (3). In Expression (3), In is a luminance value of the nth pixel in the reference object region 204, and (θn, φn) is a normal vector at the position of the pixel.

  The shadow information deriving unit 105 can strictly minimize E (w) by an operation using a technique such as QR decomposition.

Then, the shadow information deriving unit 105 uses the weight w lm for each combination of the integers l and m calculated by the equation (3) and the spherical harmonic function, and the shadow information I (θ, Find φ).

The shadow information deriving unit 105 causes the storage unit 106 to store the shadow information I (θ, φ) obtained as described above as the shadow information 205.
FIG. 7 shows the shadow information 205 obtained in accordance with the above description as an image visualized.

  The shadow information 205 derived as described above can be used by, for example, a rendering engine corresponding to augmented reality to synthesize a CG object with the scene image 201. At this time, for example, the rendering engine applies I (θ, φ) as a shadow according to the normal direction (θ, φ) of the CG object.

FIG. 8 shows an example of a CG object to which a shadow is given by applying the shadow information 205 shown in FIG.
FIG. 9 shows a synthesized image 201A obtained by synthesizing the CG object with a shadow as shown in FIG. 8 with the original scene image 201.
In this way, by applying the shadow information 205 to apply the shadow, the shadow of the CG object substantially matches the state of the light source in the scene, as shown in FIG. 9, for example. Thereby, the realism of the CG object existing in the scene of the composite image 201A is improved.

In FIG. 1, the storage unit 106 derives shadow information such as a scene image 201, a depth map 202, a normal map 203, a reference object region 204, and shadow information 205, as can be understood from the above description. Stores data to be used. As a device corresponding to the storage unit 106, for example, an HDD (Hard Disc Drive), a flash memory, or the like can be adopted.
In addition, the display unit 107 displays an image such as extraction target data in accordance with the control of the reference object region extraction unit 104.
The operation unit 108 collectively indicates operators and operation devices used for an operation for the user to select a reference object region from an image displayed on the display unit 107, for example. Such operators and operation devices of the operation unit 108 include, for example, a mouse and a keyboard.

[Example of processing procedure]
The flowchart in FIG. 10 illustrates an example of a processing procedure executed by the shadow information deriving device 100 according to the first embodiment.
First, the scene image input unit 101 inputs the scene image 201 and stores it in the storage unit 106 (step S101).
Further, the depth map input unit 102 inputs the depth map 202 and stores it in the storage unit 106 (step S102).

Next, the normal map deriving unit 103 reads the depth map 202 from the storage unit 106 (step S103).
The normal map deriving unit 103 uses the read depth map 202 to derive the normal map 203 as described above (step S104). The normal map deriving unit 103 stores the derived normal map 203 in the storage unit 106 (step S105).

Next, the reference object region extraction unit 104 inputs extraction target data (step S106). As described above, the extraction target data is at least one of the scene image 201, the depth map 202, and the normal map 203.
As described above, the reference object region extraction unit 104 extracts the reference object region 204 from the extraction target data in accordance with, for example, a user operation (step S107). The reference object area extraction unit 104 stores the extracted reference object area 204 in the storage unit 106 (step S108).

Next, the shadow information deriving unit 105 inputs the scene image 201, the normal map 203, and the reference object region 204 (step S109).
The shadow information deriving unit 105 derives the shadow information 205 as described above using the input scene image 201, normal map 203, and reference object region 204 (step S110).
The shadow information deriving unit 105 stores the shadow information 205 derived in this way in the storage unit 106 (step S111).

<Second Embodiment>
[Configuration example of shadow information deriving device]
Next, a second embodiment will be described.
FIG. 11 shows a configuration example of the shadow information deriving device 100A in the second embodiment. In this figure, the same parts as those in FIG. 1 are denoted by the same reference numerals, and description thereof is omitted here.

A shadow information deriving device 100A shown in FIG. 11 further includes a shadow information regularization unit 109 in addition to the configuration of the shadow information deriving device 100 in FIG.
The shadow information regularization unit 109 corrects a predetermined parameter used for deriving the shadow information 205 so that the regularized shadow information 205 is derived. Regularization here refers to applying a constraint so that the derived shadow matches the diffuse reflection shadow. Further, the parameter to be corrected by the shadow information regularization unit 109 is a coefficient (weight w) of a spherical harmonic function.
In addition, this shadow information regularization part 109 is realizable when CPU runs a program, for example.

  For example, because the reference object is specularly reflected or an error has occurred in the depth value indicated by the depth map 202, the weight between the orders of the weight w obtained based on the equation (3) The ratio can lead to undesirable results for diffuse reflection shading. When such a result occurs, the shadow information 205 itself does not conform to the diffuse reflection shadow.

As a specific example of the malfunction caused by the shadow information 205 that is not adapted to the diffuse reflection shadow, for example, a ringing artifact may occur.
FIG. 14 shows a shadow derived when the scene image 201 shown in FIG. 12 and the depth map 202 shown in FIG. 13 are input and the sphere region indicated by the white arrow in FIG. 12 is extracted as the reference object region 204. Information 205 is visualized by an image.
In the shadow information 205 shown in FIG. 14, a bright line is generated along the lower left edge of the circle as indicated by a white arrow. This bright line is a ringing artifact. This bright line as a ringing artifact is not observed in the real world diffuse reflection. If the shadow information 205 that does not cause such ringing artifacts can be derived, the degree of coincidence with the actual scene regarding the shadow given to the CG object is further increased.

The shadow information regularization unit 109 in the second embodiment regularizes the shadow information as follows in order to reduce the phenomenon such as the ringing artifact as shown in FIG.
That is, the shading information regularization unit 109 obtains the weight w that minimizes the error function E ′ (w) obtained by adding the regularization term to the error function E (w) represented by the above equation (3). Is. The error function E ′ (w) is shown in the following equation (5).

Note that λ ₁ and λ ₂ in the two regularization terms in Equation (5) are regularization coefficients for adjusting the strength of the constraint, respectively.

The relationship between the orders of the weights w of the spherical harmonic function when expressing the shadow of the diffuse reflection is “2” which is the absolute value of the first-order (l = 1) weight and the zero-order (l = 0) weight. It is known that the absolute value of the secondary (l = 2) weight is about [1/4] of the absolute value of the zeroth weight.
Based on this, the relation of the sum of squares between the orders of the spherical harmonic functions can be obtained as follows.
In other words, the square sum of the first-order weights for the zeroth order is “3”, which is the number of spherical harmonics corresponding to the first order, and the zeroth-order (l = 0) weight for the absolute value of the first-order weights. By the absolute value ratio “2/3”, 3 × (2/3) ² = 4/3 times.
The square sum of the second-order weights for the zeroth order is “5” which is the number of spherical harmonics corresponding to the second order, and the zeroth order (l = 1) for the absolute value of the second-order (l = 1) weight. 0 ×), which is a ratio of the absolute value of the weight, is 5 × (1/4) ² = 5/16 times.
The error function E ′ (w) in Equation (5) is given a constraint so as to maintain the coefficient ratio indicated by the multiple of the square sum obtained as described above. Giving the constraint in this way means that the constraint is given by the ratio of the weight w of the spherical harmonic function when the shadow of diffuse reflection is expressed.

The shadow information regularization unit 109 uses, for example, the minimization result of the error function E (w) of Equation (3) obtained by the shadow information deriving unit 105 as an initial value, and uses the error function E of Equation (5). The weight w that minimizes' (w) can be obtained. The weight w of the spherical harmonic function obtained in this way has a ratio suitable for expressing the shadow of diffuse reflection.
Note that the shadow information regularization unit 109 uses, for example, the minimization result of the error function E (w) of Equation (3) as an initial value, and the error function E ′ (w) of Equation (5) by the Newton method. The weight w for minimizing may be obtained.

  In the second embodiment, the shadow information deriving unit 105 uses the weight w corresponding to the error function E ′ (w) obtained by the shadow information regularization unit 109 as described above, and uses Equation (4). Regularized shadow information 205A is obtained by calculation. The regularized shadow information 205A derived in this way is, for example, regularized shadow information 205 in the first embodiment, and is adapted to the expression of diffuse reflection shadows.

FIG. 15 shows the regularized shadow information 205A derived as described above visualized by an image. As in FIG. 14, the regularized shadow information 205A shown in FIG. 15 is input with the scene image 201 shown in FIG. 12 and the depth map 202 shown in FIG. Is derived as a reference object region 204.
As can be seen by comparing FIG. 15 and FIG. 14, in the regularized shadow information 205 </ b> A shown in FIG. 15, the ringing artifacts at the locations indicated by the white arrows are greatly reduced.
Then, by adding a shadow to the diffusely reflected CG object using the regularized shadow information 205A, the presence of the CG object in the scene is enhanced.

[Example of processing procedure]
The flowchart in FIG. 16 illustrates an example of a processing procedure executed by the shadow information deriving device 100A according to the second embodiment. Note that, in the steps shown in this figure, the same processes as those in FIG. 10 are denoted by the same reference numerals, and description thereof is omitted here.
The shadow information deriving unit 105 inputs the scene image 201, the normal map 203, and the reference object region 204 (step S109), and then calculates a weight w that minimizes the error function E (w) of Equation (3). (Step S110A).
Next, the shading information regularization unit 109 corrects the weight w obtained in step S110A by obtaining a weight w that minimizes the error function E ′ (w) in Expression (5) (step S110B).
Then, the shadow information deriving unit 105 calculates the regularized shadow information 205A using the weight w corrected in step S110B (step S110C). Thereby, the shadow information deriving unit 105 derives the regularized shadow information. The shadow information deriving unit 105 stores the calculated regularized shadow information 205A in the storage unit 106 (step S111 A ). By such processing, the shadow information deriving device 100A according to the second embodiment can obtain regularized shadow information corresponding to diffuse reflection.

  Note that a program for realizing the functions of the functional units in FIGS. 1 and 11 is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system and executed. The shadow information of this embodiment may be derived. Here, the “computer system” includes an OS and hardware such as peripheral devices.

Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used.
The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Further, the “computer-readable recording medium” refers to a volatile memory (RAM) in a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In addition, those holding programs for a certain period of time are also included. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

  DESCRIPTION OF SYMBOLS 100, 100A ... Shadow information derivation | leading-out apparatus, 101 ... Scene image input part, 102 ... Depth map input part, 103 ... Normal map derivation part, 104 ... Reference object area extraction part, 105 ... Shadow information derivation part, 106 ... Memory | storage part 107: Display unit, 108: Operation unit, 109 ... Shadow information regularization unit, 201 ... Scene image, 201A ... Composite image, 202 ... Depth map, 203 ... Normal map, 204 ... Reference object area, 205 ... Shadow information , 205A ... Regularized shadow information

Claims (4)

A normal map derivation unit for deriving a normal map of the scene indicated by the scene image based on a depth map corresponding to a scene image having image content as a scene;
A reference object region extraction unit that extracts a reference object region that is a region portion of a reference object based on at least one of the scene image, the depth map, and the normal map;
A shadow information deriving unit for deriving shadow information based on the scene image, the normal map, and the reference object region;
When correcting the coefficient of the spherical harmonic function used by the shadow information deriving unit so that the normalized shadow information is derived, the coefficient ratio between the orders in the spherical harmonic function is maintained with respect to the error function including the coefficient. A shadow information deriving device comprising: a shadow information regularization unit that adds a regularization term for giving a constraint so as to obtain the coefficient that minimizes an error function to which the regularization term is added . The error function (E ′ (w)) obtained by adding the regularization term to the error function (E (w)) has a ratio of the coefficient of the spherical harmonic function of the 0th order and the 1st order to 2/3, the 0th order. The ratio of the coefficients of the spherical harmonics of the second order and the second order is 1/4, the coefficient is w, and the regularization coefficient for adjusting the constraint strength is λ ₁ , Λ ₂ As represented by the following formula 1.
  The shadow information deriving device according to claim 1.
A normal map derivation step for deriving a normal map of the scene indicated by the scene image based on a depth map corresponding to the scene image having image content as a scene;
A reference object region extraction step of extracting a reference object region that is a region portion of a reference object based on at least one of the scene image, the depth map, and the normal map;
Shadow information deriving step for deriving shadow information based on the scene image, the normal map, and the reference object region;
When correcting the coefficient of the spherical harmonic function used in the shadow information derivation step so that the regularized shadow information is derived, the coefficient ratio between the orders in the spherical harmonic function is maintained with respect to the error function including the coefficient. A shadow information deriving method comprising: adding a regularization term for giving a constraint so as to obtain a coefficient that minimizes an error function to which the regularization term is added .   A program for causing a computer to function as the shadow information deriving device according to claim 1.