JP2014164615A - Shading information derivation device, shading information derivation method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To derive shading information by a method which has been unavailable conventionally.SOLUTION: A shading information derivation device includes: an area extraction part which extracts a reference object region from a photographed image; a normal line derivation part which derives a normal line of a reference object based on the extracted reference object region; and a shading information derivation part which derives shading information based on the extracted reference object region and the normal line of the derived reference object.

Description

  The present invention relates to a technique for deriving shadow information.

  In augmented reality, for example, when a CG (computer graphics) object such as a virtual object is combined with a scene image, a shadow is added to the CG object. In this way, by adding a shadow corresponding to the CG object, it is possible to make it appear as if the CG object actually exists in the scene image.

  As described above, shadow information is used to add a shadow. The shadow information is information indicating how the shadow generated when the entity of the CG object is located in the actual scene in correspondence with the synthesis position of the CG object in the scene image is expressed in the scene image. is there.

The derivation of the shadow information is realized, for example, by estimating information (light source information) about the light source in the scene and rendering the CG object based on the light source information.
In estimating the light source information, for example, an omnidirectional camera that captures an omnidirectional image is used (for example, see Non-Patent Document 1). This method is based on the premise that a light source is imaged in any part of an image obtained by imaging all directions. In addition, this method estimates the position, color, position, intensity, and the like of the light source corresponding to the scene image based on the image captured by the omnidirectional camera.

  Further, a technique using a metal sphere is known (see, for example, Non-Patent Document 2). This method is based on the idea according to the above-mentioned omnidirectional camera. That is, this method is based on the premise that a light source is also included in an image of a surrounding scene reflected on a metal sphere. In addition, this method estimates the position, color, position, intensity, and the like of the light source based on the image of the metal sphere in which the surrounding scene is reflected.

In addition, a method using a white sphere having a diffuse reflection surface is known (see, for example, Non-Patent Document 3). In this method, the color, intensity, direction, and the like of the light source around the white sphere are estimated based on the shadow appearing in the white sphere image.
Further, there is also known a method for allowing a user to input light source position and room three-dimensional information in an image and estimating the color, intensity, position, etc. of the light source using the input three-dimensional information as a clue (for example, Non-patent document 4).
There is also known a method that does not require a pre-procedure such as measurement of reflection characteristics of an apparatus to be used or an object in a scene (for example, see Non-Patent Document 5). In this method, a depth map is acquired by acquiring depth information by distance measurement together with a color image using an RGB + D sensor such as Kinect (registered trademark). This method derives shadow information by obtaining only a small number of spherical harmonic function weights using the acquired color image and depth map as inputs.

Imari Sato, Morihiro Hayashida, Kaiyo, Yoichi Sato, Katsushi Ikeuchi, Image Generation under Real Light Environment: High-speed rendering method based on linear sum of basic images, IEICE Transactions. D-II, Information / System, II-Pattern processing J84-D-II (8), pages 1864-1872, 2001. Kusuma Agusanto, Li Li, Zhu Chuangui, and Ng Wan Sing, Photorealistic rendering for augmented reality using environment illumination, Proceedings of the Second IEEE and ACM International Symposium on Mixed and AugmentedReality (ISMAR), pages 208-216, 2003. Miika Aittala, Inverse lighting and photorealistic rendering or augmented reality, The Visual Computer, Volume 26, Issue 6, pages 669-678, 2010. Kevin Karsch, Varsha Hedau, David Forsyth, and Derek Hoiem, Rendering synthetic objects into legacy photographs, ACM Transactions on Graphics, Volume 30, Issue 6, December 2011, Article No. 157. Yasuhiro Yao, Kazuyuki Tsuji, Harumi Kawamura, Akira Kojima, Acquisition from general objects with shadows recognized as diffuse reflection, VR Gakken, Volume 17, Number CS-1, 2012.

When using the method according to Non-Patent Document 1, an omnidirectional camera is required. This omnidirectional camera is a special device including, for example, a fisheye lens, and is not easily available to a general end user called a consumer.
Also in the case of the methods according to Non-Patent Documents 2 and 3, it is necessary to prepare a metal sphere or a white sphere having a diffuse reflection surface. Such metal spheres and white spheres are also special as instruments and are not readily available to end users.

Moreover, when utilizing the method by nonpatent literature 4, it is necessary for a user to input the clue for estimating light source information. In this case, the user must estimate the position of the light source from the image and grasp the three-dimensional configuration of the scene of the image, and requires a certain level of expertise.
Further, when using the method according to Non-Patent Document 5, it is necessary to obtain a color image and a depth map using an RGB + D sensor such as Kinect (registered trademark). Since such Kinect (registered trademark) is generally not widely used as an apparatus, the usable scenes are limited.

  As described above, any method according to Non-Patent Documents 1 to 5 requires, for example, an uncommon apparatus or instrument, or requires specialized knowledge. Therefore, it is difficult to provide an end user with an environment in which, for example, a CG object having a shadow in augmented reality can be combined with a scene image by any of the methods of Non-Patent Documents 1 to 5.

  In view of the above circumstances, an object of the present invention is to provide a technique for deriving shadow information by an unprecedented method.

  One aspect of the present invention is an area extracting unit that extracts a reference object region from a captured image, a normal deriving unit that derives a normal of a reference object based on the extracted reference object region, A shadow information deriving device comprising: a shadow information deriving unit that derives shadow information based on the reference object region and the derived normal line of the reference object.

  One aspect of the present invention is the above-described shadow information deriving device, wherein the shadow information deriving unit derives the shadow information by calculating a coefficient that optimizes a quadratic polynomial for a coefficient of a spherical harmonic function. .

  One aspect of the present invention is an area extraction step of extracting a reference object region from a captured image, a normal derivation step of deriving a normal of a reference object based on the extracted reference object region, and an extracted A shadow information deriving method comprising: a shadow information deriving step for deriving shadow information based on the reference object region and the derived normal line of the reference object.

  One embodiment of the present invention is a program for causing a computer to function as the above-described shadow information deriving device.

  According to the present invention, it is possible to derive shadow information by an unprecedented method.

The structural example of the shadow information derivation | leading-out apparatus 100 in this embodiment is shown. It is a figure which shows an example of the scene image 201 and the reference object image 202. FIG. It is a figure which shows the derivation | leading-out result of a three-dimensional shape. It is a figure which shows the spherical coordinate system expressing the normal line 203 of a reference object. It is a figure which visualizes and shows the spherical harmonic function utilized in this embodiment. It is a figure which visualizes and shows the shadow information in this embodiment. The example of the process sequence which the shadow information derivation device 100 in this embodiment performs is shown.

[Configuration example of shadow information deriving device]
Hereinafter, the shadow information deriving device of the present invention will be described. The shadow information deriving device of this embodiment derives shadow information by inputting a single scene image.

  FIG. 1 shows a configuration example of a shadow information deriving device 100 in the present embodiment. The shadow information deriving device 100 shown in this figure includes functional units of an image input unit 101, a region extracting unit 102, a normal deriving unit 103, and a shadow information deriving unit 104. Note that these functional units are realized by, for example, a CPU (Central Processing Unit) executing a program. The shadow information deriving device 100 includes a device such as the storage unit 105.

  The image input unit 101 receives image data input to the shadow information deriving device 100. The image input unit 101 may read image data recorded on a recording medium such as a CD-ROM or a USB memory (Universal Serial Bus Memory). The image input unit 101 may receive an image captured by a still camera or a video camera from the camera. When the shadow information deriving device 100 is built in a still camera or a video camera, the image input unit 101 may receive a captured image or an image before imaging from the bus. Further, the image input unit 101 may receive image data from another information processing apparatus via a network. The image input unit 101 may be configured in a different manner as long as it can receive input of image data. An image whose input is accepted by the image input unit 101 is referred to as a “scene image”.

  The image input unit 101 inputs a scene image 201 from the outside and stores it in the storage unit 105. The scene image 201 in the present embodiment may be an image obtained by imaging a certain real environment, for example.

  The region extraction unit 102 generates a reference object image 202 by extracting a reference object region that is a region portion of the reference object from a single scene image 201 stored in the storage unit 105. The reference object is an object that the shadow information deriving unit 104 refers to when acquiring light source information at a position where the CG object is arranged in the scene. The region extraction unit 102 derives the reference object image 202 as follows, for example.

First, the area extraction unit 102 reads a single scene image 201 from the storage unit 105. Next, the region extraction unit 102 performs region division on the scene image 201. For example, the region extraction unit 102 performs region division by grab cut. In Grab cut, first, the user manually designates a pixel in the reference object region and a pixel outside the reference object region. Thereafter, the region extraction unit 102 performs region division by graph cut using each designated pixel as a seed. The region extraction unit 102 generates a reference object image 202 by extracting a reference object region from among the divided regions.
The region extraction unit 102 stores the generated reference object image 202 in the storage unit 105.

  Also, the reference object region extraction method need not be limited to the method described above. For example, the region extraction unit 102 may extract the reference object region by decomposing the scene image 201 by HSV (Hue Saturation Value). In this method, the user first designates a pixel in the reference object region by manual input. Thereafter, the region extraction unit 102 refers to the adjacent pixel of the designated pixel, and adds the adjacent pixel to the region of the reference object when the difference in HUE (hue) between the pixels is equal to or less than the threshold value. When the color of the reference object is known, the region extraction unit 102 may extract the reference object region by extracting the color region of the reference object. Specifically, when the calculation is performed based on the shadow of a person, such as a virtual clothes try-on simulation, the reference object area may be extracted by extracting the area of the person's skin color.

FIG. 2 is a diagram illustrating an example of the scene image 201 and the reference object image 202.
FIG. 2A shows an example of the scene image 201 input to the image input unit 101. Shadow information is derived based on the single scene image 201 shown in FIG.
FIG. 2B is a diagram illustrating an example of the reference object image 202. A reference object image 202 as shown in FIG. 2B is generated from a single scene image 201 by the processing of the region extraction unit 102.

  The normal derivation unit 103 derives a reference object normal 203 based on the reference object image 202 stored in the storage unit 105. The normal deriving unit 103 derives the normal 203 of the reference object as follows, for example. First, the normal derivation unit 103 reads the reference object image 202 from the storage unit 105. Next, the normal derivation unit 103 restores the three-dimensional shape of the reference object from the read reference object image 202. Then, the normal derivation unit 103 derives the reference object normal 203 based on the restored three-dimensional shape.

  The three-dimensional shape restoration process will be described in detail. It is difficult to accurately restore a three-dimensional shape from only a two-dimensional image. However, in the shadow derivation of the present embodiment, a rough three-dimensional shape only needs to be obtained because the expression of the shadow by a small number of spherical harmonic functions is used. This is to impose regularization so as to derive a solution that preserves the nature of the shadow. Therefore, the normal derivation unit 103 restores a rough three-dimensional shape of the reference object region from the reference object image 202 using the technique of Reference 1 (Reference 1: Nathaniel R. Twarog, Marshall F. Tappen, Edward H. Adelson, Playing with Puffball: see simple scale-invariant inflation for use in vision and graphics, Proceedings of the ACM Symposium on Applied Perception, pages 47-54, 2012.).

In Reference Document 1, a three-dimensional region I (S) is generated according to Equation (1) at each point in the two-dimensional region of the image. In equation (1), S represents the reference object region, B ³ (p, r) represents the inner region of the sphere with radius r from pεS, and B ² (p, r) has radius from pεS. Represents the area inside the circle of r.

Next, the normal object derivation process of the reference object will be described in detail. The normal derivation unit 103 assigns a number k (k = 1... K) to the reference object region where the pixels excluding the pixels at the boundary between the reference object region and other than the reference object region are K pixels. If the coordinates of the kth pixel are expressed as (x _k , y _k ), the distance z (x _k , y _k ) from the center to the surface at the coordinates (x _k , y _k ) can be expressed by equation (2). Calculated. In this _{case, (x} k, _{y k)} normal _{n k} in is calculated by outer product of formula (3).

Note that, at the boundary of the reference object region, the pixel adjacent to the pixel of the reference object region is outside the reference object region, and thus the normal derivation unit 103 does not calculate the normal. Note that the correspondence between the array of pixels (K pixels) included in the reference object region and the number k does not affect the calculation result, and therefore, the rule for assigning the number k to the pixels included in the reference object region is not particularly limited. .
The normal derivation unit 103 records the reference object normal 203 derived by the above processing in the storage unit 105 in association with the pixel value by the number k.

  FIG. 3 is a diagram illustrating a derivation result of the three-dimensional shape. A three-dimensional region as shown in FIG. 3 is restored from the single scene image 201 by the processing of the normal derivation unit 103. The normal derivation unit 103 derives the reference object normal 203 by performing the above-described processing based on such a three-dimensional region.

FIG. 4 is a diagram showing a spherical coordinate system expressing the normal 203 of the reference object. As shown in FIG. 4, the spherical coordinate system is a coordinate system that represents three-dimensional coordinates on a unit sphere with the origin at the center by two declination angles θ and φ. The normal derivation unit 103 calculates the normal direction (θ _k , φ _k ) expressed by the spherical coordinate system using the equations (4) and (5). The ranges in the normal direction are 0 ≦ θ _k ≦ π / 2 and −π / 2 ≦ φ _k ≦ π / 2.

The shadow information deriving unit 104 derives shadow information based on the reference object image 202 and the normal line 203 of the reference object. The shadow information deriving unit 104 derives the shadow information as follows.
The shadow information deriving unit 104 uses a spherical harmonic function to derive the shadow information. When the spherical harmonic function is used, there is an advantage that the problem does not become complicated even if the number of light sources increases. Further, when the spherical harmonic function is used, there is an advantage that the shadow of diffuse reflection can be expressed with high accuracy by a small number of bases. The spherical harmonic function is a function with respect to spherical coordinates (θ, φ), and is defined by the following equation (6).

In Expression (6), j and m are integers, and j ≧ 0 and −j ≦ m ≦ j. j represents the order. Further, K_j_m is a normalization coefficient represented by Expression (7), and P_j_m (x) is a Legendre power function represented by Expression (8). The notation K_j_m indicates that j is a subscript of K and m is a superscript of K. The notation P_j_m indicates that j is a subscript of P and m is a superscript of P.

The shadow of the reference object can be expressed using the weight w_j_m as L (θ, φ) in the following equation (9) by the spherical harmonic function represented by equation (6). That is, deriving the weight w_j_m of the spherical harmonic function is the shadow information derivation, and in this embodiment, the weight of the spherical harmonic function is handled as the shadow information. The notation w_j_m indicates that j is a subscript of w and m is a superscript of w. The shadow is a luminance value according to the normal direction of the object surface. FIG. 5 shows a schematic diagram of the shading expression of Equation (9). FIG. 5 is a diagram showing a spherical harmonic function used in the present embodiment in a visualized manner.

In the present invention, nine spherical harmonic functions of 0 ≦ j ≦ 2 are used. Nine spherical harmonic functions of 0 ≦ j ≦ 2 can be expressed as, for example, Equation (10) by numerically calculating Equation (6).

As described above, the reference object image 202 is numbered k = 1... K for each pixel in the K pixel area. The shadow information deriving unit 104 derives the weight w_j_m according to the equation (11) using the nine spherical harmonic functions shown in the equation (10), the reference object image 202, and the normal line 203 of the reference object, thereby performing the shadow. Get information. In Equation (11), w is a vector representation of the spherical harmonic function weights. e (w) is a term relating to an error between the weighted sum of the spherical harmonic functions shown in Equation (12) and the shadow of the reference object, and r (w) is a regularization term shown in Equation (13). In Expression (12), Lk is a luminance value of the kth pixel in the reference object region, and (θk, φk) is a normal vector at the position of the pixel.

Note that a _0, a _{1, and} a ₂ in Equation (13) are coefficients, and for example, the relationship indicated by the weight of the spherical harmonic function when the diffuse reflection shadow shown in Reference 2 is expressed is maintained. Such values may be a ₀ = 1, a ₁ = 9/4, and a ₂ = 16 (Reference 2: Ravi Ramamoorthi, Pat Harahan, On the relationship between radiance and irradiance: determining the illumination from images of a convex Lambertian object, J. Opt. Soc. Am. A, Vol. 18, No. 10, pages 2448-2459, October 2001.). In addition, g (K) in Expression (13) is a coefficient for scaling the regularization term r (w), and is expressed as g (K) = K, for example.

An example of a solution to the optimization problem of equation (11) is shown below. The optimization problem of equation (11) can be solved exactly by the matrix operation shown in equation (14). In equation (14), ^ w (^ is on w, the same applies hereinafter) is a solution to the optimization problem. Note that Φ in Expression (14) is represented by a K × 9 matrix in Expression (15), A in Expression (14) is represented by a 9 × 9 diagonal matrix in Expression (16), and t is represented by a K-dimensional vector of Expression (17). The method for deriving the above optimization problem is an example, and may be derived by other methods. As long as there is no calculation error, the result does not change with any derivation method.

The derived weight ^ w is expressed as the following equation (18).
The shadow information deriving unit 104 causes the storage unit 105 to record the derived weight ^ w as the shadow information 204.
FIG. 6 is a diagram visualizing and showing the shadow information in the present embodiment.

Based on the shadow information 204 derived as described above, a CG object can be rendered by the following method and synthesized into an actual image.
The virtual object data includes information on I (I is a natural number) three-dimensional vertices, and the normal direction of the object surface at each point is expressed by spherical coordinates (θ _i , φ _i ), (i = 1... I ). At this time, the color information c _i of the i-th vertex of the virtual object to be combined with the real image is calculated by Expression (19). Ρ in equation (19) is calculated by equation (20). Α in Expression (20) is a luminance ratio indicating how bright the virtual object is compared to the reference object, and the color {r, g of the virtual object normalized by the luminance value l (r, g, b) , B} is the product of ^T. Here, the calculation of the luminance value is performed, for example, as l (r, g, b) = 0.299r + 0.587 g + 0.114b. Each value of α and {r, g, b} ^T is an arbitrary value of 0 or more, and may be given by input, for example.

By expressing the CG object using the color information c _i obtained by Expression (19) and arranging the CG object in the actual image, it is possible to synthesize the CG object that matches the scene illumination.

[Example of processing procedure]
The flowchart of FIG. 7 shows an example of a processing procedure executed by the shadow information deriving device 100 in the present embodiment.
First, the image input unit 101 inputs the captured scene image 201 and stores it in the storage unit 105 (step S101).

Next, the area extraction unit 102 reads the scene image 201 from the storage unit 105. Then, the area extraction unit 102 generates the reference object image 202 by performing area division as described above using the read scene image 201 (step S102).
The region extraction unit 102 stores the generated reference object image 202 in the storage unit 105 (step S103).

Next, the normal derivation unit 103 inputs the reference object image 202. As described above, the normal derivation unit 103 restores the three-dimensional region from the reference object image 202 (step S104).
As described above, the normal derivation unit 103 derives the reference object normal 203 from the restored three-dimensional region (step S105). The normal derivation unit 103 stores the derived normal 203 of the reference object in the storage unit 105 (step S106).

The shadow information deriving unit 104 derives the shadow information 204 using the reference object image 202 and the normal line 203 of the reference object (step S107).
The shadow information deriving unit 104 stores the derived shadow information 204 in the storage unit 105 (step S108).

  The shadow information deriving device 100 configured as described above can derive the shadow information by an unconventional method. Specifically, the shadow information deriving device 100 derives shadow information using only a single scene image. Therefore, it is not necessary for the shadow information deriving device 100 to prepare a special device such as the omnidirectional camera described in Non-Patent Document 1. Moreover, it is not necessary to prepare special instruments such as the metal spheres described in Non-Patent Documents 2 and 3 or white spheres having a diffuse reflection surface. Further, it does not require a certain level of expertise such as estimating the position of the light source from the image described in Non-Patent Document 4 and grasping the three-dimensional configuration. Further, there is no need to use an apparatus such as Kinect (registered trademark) described in Non-Patent Document 5. As a result, application scenes are not limited, and shadow information can be easily derived. In addition, since the shadow information deriving device 100 can derive the shadow information using only the captured single scene image, the versatility can be improved.

<Modification>
The scene image may be stored in the storage unit 105 in advance.
The function units of the image input unit 101, the region extraction unit 102, the normal derivation unit 103, and the shadow information derivation unit 104 may be implemented in different devices, or a part of the functional units may be installed in different devices. May be implemented.
Information stored in the storage unit 105 may be stored in a plurality of storage devices.
The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.
In the present embodiment, a program for realizing the function of each functional unit in FIG. 1 is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system and executed. The shadow information may be derived. Here, the “computer system” includes an OS and hardware such as peripheral devices.
Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used.
The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Further, the “computer-readable recording medium” refers to a volatile memory (RAM) in a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In addition, those holding programs for a certain period of time are also included. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

DESCRIPTION OF SYMBOLS 100 ... Shadow information derivation | leading-out apparatus, 101 ... Image input part, 102 ... Area extraction part, 103 ... Normal-line derivation part, 104 ... Shadow information derivation part, 105 ... Memory | storage part, 201 ... Scene image, 202 ... Reference object image, 203 ... Normal of reference object, 204 ... Shadow information

Claims (4)

An area extraction unit for extracting a reference object area from the captured image;
A normal derivation unit for deriving a normal of the reference object based on the extracted reference object region;
A shadow information deriving unit for deriving shadow information based on the extracted reference object region and a normal line of the derived reference object;
A shadow information deriving device comprising:   The shadow information deriving device according to claim 1, wherein the shadow information deriving unit derives the shadow information by calculating a coefficient that optimizes a quadratic polynomial for a coefficient of a spherical harmonic function. A region extraction step of extracting a reference object region from the captured image;
A normal derivation step of deriving a normal of the reference object based on the extracted reference object region;
A shadow information deriving step for deriving shadow information based on the extracted reference object region and a normal of the derived reference object;
A method for deriving shadow information.   A program for causing a computer to function as the shadow information deriving device according to claim 1.