JP2013235537A - Image creation device, image creation program and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To simply create a virtual image that performs a light rendering to a target part by use of a photographing image.SOLUTION: The image creation device comprises: an actual image acquisition part 11 that acquires an actual image; a three-dimensional information acquisition part 12 that acquires three-dimensional information including the actual image and at the same time depth information; a contour extraction part 13a that extracts contour images from the actual image and the three-dimensional information respectively; a correction part 13b that matches a contour of a three-dimensional (3D) contour image to a contour of a two-dimensional (2D) contour image; a three-dimensional model estimation part 14 that uses the 3D contour image to estimate a shape estimation of an occlusion part, converts the 3D contour image into a coordinate system for performing a shading process and sets reflectance of an object contained in the 3D contour image; a light source estimation part 21 that estimates light source information for performing a bright rendering to the target part on the actual image to be designated by an emphasis designation part 20; a shading process part 15 that defines a virtual light source in a virtual space of the 3D contour image on the basis of estimated light source information to generate a shading image on the basis of reflectance information of the object and outputs a two-dimensional shading image; and a blending part 16 that synthesizes the shading image with the actual image to create a two-dimensional virtual image.

Description

  The present invention relates to an image creation device, an image creation program, and a recording medium that create a virtual image using a depth map and a real image.

  As a technology for taking impressive photographs, there is a technology that installs an external strobe or a reflex board and applies lighting to the subject to produce a sense of depth and emphasize the subject. Although such a technique enables photographing under an arbitrary light source, it requires expensive equipment and user skills and is difficult for a general user to easily realize. Further, since such lighting conditions are determined at the time of shooting, it is difficult to correct an image after shooting.

  On the other hand, a technique called Image-Based Rendering (IBR) is known as a technique for arranging a real model in a computer graphic with a natural appearance (Patent Document 1). In IBR, an image image under an arbitrary light source is created by linearly combining a plurality of real model images photographed in a plurality of light source states. That is, it is possible to create an image image under an arbitrary light source by using a plurality of captured images with different light source irradiation directions.

JP 2001-52208 A (paragraph [0002])

  Although it is possible to create an image of an actual model under an arbitrary light source using a general IBR technique, it is extremely unrealistic to capture a plurality of images with different light sources during normal imaging.

  In view of the above problems, the present invention provides an image creation device, an image creation program, and a recording medium that create a virtual image that achieves a desired shadow and effect using an image obtained from a single shooting. Objective.

  An aspect of the image creating apparatus exemplifying the present invention includes: a real image acquiring unit that acquires a real image; a three-dimensional information acquiring unit that acquires three-dimensional information including depth information corresponding to the real image; A three-dimensional image generating means for generating a three-dimensional image based on the image and the three-dimensional information; a specifying means for specifying a target portion to effect the light on the real image; and the specified target portion Estimation means for estimating information on a virtual light source for performing the effect based on the shading, and shading for generating shadow information by defining the virtual light source in the virtual space of the three-dimensional image based on the estimated information And blending means for synthesizing the shading information with the actual image. The present invention may be an image creation program. Further, a computer-readable recording medium in which an image creation program is recorded may be used.

  According to the present invention, it is possible to easily create a virtual image in which a light effect is applied to a desired target portion.

It is a block diagram which shows the outline of the image production apparatus of this invention. It is a flowchart which shows the operation | movement procedure of an image preparation apparatus. It is an image which shows an example of the real image which the real image acquisition part acquired. It is an image which shows an example of the depth map which the depth map acquisition part acquired. It is an image which shows an example of the outline image extracted from the real image. It is an image which shows an example of the outline image extracted from the depth map. It is explanatory drawing explaining the process which samples each point of an outline by carrying out a raster scan of the outline image extracted from the real image. It is explanatory drawing which shows the process which matches the point of the outline image of a depth map with each point sampled from the outline of the real image. It is explanatory drawing which shows the process which correct | amends the outline of a depth map based on the outline of a real image. It is a flowchart which shows the operation | movement procedure of a light source estimation part. It is explanatory drawing which shows a mode that a target part is designated with respect to a real image. It is explanatory drawing which shows the state which divided the range of the three-dimensional outline image corresponding to a target part into meshes. It is explanatory drawing which shows the surface normal vector calculated from each mesh. It is explanatory drawing which shows the virtual light source defined on the direction line of the synthetic | combination vector calculated | required from the sum of a surface normal vector. It is explanatory drawing which shows the shadow information which the shading part produced | generated. It is explanatory drawing which shows the shadow image which synthesize | combined the shadow information and the real image. It is a flowchart which shows the operation | movement procedure of 2nd Embodiment which designates a to-be-photographed object as a target part and performs a light effect. It is explanatory drawing which shows virtually the outline image which performed the clustering process which extracts the outline image of a to-be-photographed object according to distance by clustering a to-be-photographed object. It is explanatory drawing which shows the screen which selects the to-be-photographed object which gives a light effect with respect to a real image. It is explanatory drawing which shows virtually the state which defines a virtual light source behind the outline image of the designated subject, and produces | generates shadow information. It is explanatory drawing which shows the virtual image produced | generated by synthesize | combining a shadow image and a real image.

[First Embodiment]
As shown in FIG. 1, the image creating apparatus 10 includes a real image acquisition unit 11, a depth map acquisition unit 12, a three-dimensional model estimation unit 14, a shading processing unit 15, a blending unit 16, an object estimation unit 17, an object / reflectance factor. A correspondence unit 18, a reflectance setting unit 19, an emphasis designation unit 20, a light source estimation unit 21, a light source information change operation unit 22, and a blend image output unit 23 are provided. The real image acquisition unit 11 is a known imaging device for acquiring real image information (hereinafter referred to as “real image”), and includes a memory 11 a that stores the acquired real image.

  The depth map acquisition unit 12 acquires a depth map that is depth information or three-dimensional information including the depth map (hereinafter referred to as “depth map”) so as to be in the same imaging range simultaneously with the actual image. And a memory 12a for storing the acquired depth map. The depth map acquisition unit 12 includes the well-known “depth map generation method using a stereo camera”, “TOF (Time of Flight) method using infrared light”, and “method using a reflection pattern of infrared light”. , Or “a method for generating a depth map from a motion vector” or the like, and a detailed description thereof is omitted here. The depth map is a grayscale image in which the luminance corresponding to the depth of each pixel is obtained, for example, in the case of a pixel having a deep depth based on a certain reference position, the luminance of the pixel is increased.

  The depth map correction unit 13 includes a contour extraction unit 13a and a correction unit 13b. The contour extraction unit 13a receives the depth map and the actual image, executes high-pass filter processing, etc., and extracts the contour by emphasizing the contours in the depth map and the actual image. The correction unit 13b corrects the contour (3D contour) of the depth map in accordance with the contour (2D contour) of the actual image. In the 3D contour image and the 2D contour image, there is a mismatch particularly in the contour portion, and the shading at the time of shading deviates from the contour of the actual image. In order to prevent this, the inconsistency in the shading processing unit 15 is reduced by correcting the contour (3D contour) extracted from the depth map based on the contour (2D contour) extracted from the actual image.

  The 3D model estimation unit 14 receives a 3D contour image subjected to contour correction, and performs coordinate conversion and shape estimation. The 3D model estimation unit 14 is an area that is hidden by an object in the 3D contour image and for which depth information is not obtained, that is, an occlusion processing unit 14a that performs shape estimation of an occlusion portion, and a shading processing unit 15 A coordinate conversion unit 14b that converts to a coordinate system, and a reflectance setting processing unit 14c that sets reflectance for a three-dimensional model based on reflectance information input from the reflectance setting unit 19 are provided.

  In the depth map, it is difficult to acquire three-dimensional information of an area where an object in the back is blocked by an object in front, and this is generally called occlusion. Therefore, in the 3D contour image input to the occlusion processing unit 14a, a part of the three-dimensional information (occlusion part) is missing. In the occlusion processing unit 14a, the shape of the occlusion unit is estimated using the depth map, and the three-dimensional information for performing the shading process is complemented. The coordinate conversion unit 14b converts the coordinate system of the depth map acquired by the depth map acquisition unit 12 into a coordinate system for performing a shading process. The reflectance setting processing unit 14c sets the reflectance of the object included in the 3D contour image.

  The shading processing unit 15 receives the interpolated 3D contour image, the reflectance information of the object, and the information on the virtual light source input from the light source estimation unit 21 as input to define a virtual light source in the virtual space of the 3D contour image. A three-dimensional shadow image is generated, and a two-dimensional shadow image is output from the three-dimensional shadow image based on the camera position of the real image. As the shading processing unit 15, it is preferable to generate a shadow image expressed in, for example, a gray scale by using a known technique such as a ray tracing method or a radiosity method.

  The blending unit 16 receives a two-dimensional shadow image and a real image as input, and blends these two images. As the blending process, a well-known α blending technique may be used. The blend image output unit 23 outputs the two-dimensional virtual image generated by the blending unit 16 to, for example, a display unit or a printer that is externally connected to the apparatus 10.

  The object estimation unit 17 estimates an object included in the actual image based on the actual image and the color and shape obtained from the depth map. For the object estimation, a method using a geometric shape such as human face recognition or a template matching method using a difference from a template may be used for the object estimation. Such an object estimation method is widely known in the field of recognition / learning algorithms.

The object / reflectance correspondence unit 18 obtains the reflectance for the object estimated by the object estimation unit 17 using a DB (database) 18a in which the object and the reflectance are associated in advance.
The reflectance setting unit 19 outputs the reflectance corresponding to the estimated object to the three-dimensional model estimation unit 14. Note that a setting operation unit that can arbitrarily set the reflectance of the object selected by the user may be provided, and the reflectance input from the setting operation unit may be output to the three-dimensional model estimation unit 14. .

  The emphasis designating unit 20 designates a target portion to be rendered bright. For example, although not shown, a display unit with a touch panel is provided, an actual image is displayed on the display unit, and a frame or a selection tool is selected in a range including the operation position in response to a tap operation or a touch operation. In response to the operation of the OK button displayed on the screen, the range is specified by expanding or contracting the frame or lasso by dragging or pinching and displaying the wrinkles over the actual image. It is preferable to input the coordinates of the frame and lasso set at that time. Here, the emphasis designation unit 20 is an example of a designation unit.

  The light source estimation unit 21 estimates information related to a virtual light source (hereinafter referred to as “light source information”) for providing a brightening effect on the target portion specified by the enhancement specification unit 20. Here, the light source information includes a position, a direction (irradiation direction), a light source distribution, and a light source wavelength. In this embodiment, position information is estimated among these pieces of information, and other information uses predetermined values. All information may be estimated, or a plurality of information may be estimated, and a predetermined value may be used for the remaining information.

  The light source information change operation unit 22 is an operation unit for changing the estimated light source information to desired information. The shading processing unit 15 redefines the virtual light source based on the information changed by the operation from the light source information changing operation unit 22 to generate a shadow image. When operating the light source information changing operation unit 22, it is desirable to display the shadow image on the display unit so that the changed light source information is reflected immediately, because the changing operation can be easily performed. The light source information change operation unit 22 is an example of a light source information change unit. The light source information change operation unit 22 may be omitted.

  Next, the operation of the present embodiment will be described with reference to FIG. First, the real image acquisition part 11 acquires the real image shown, for example in FIG. 3 (S-1). The acquired actual image is stored in the memory 11a. Thereafter, the actual image is read from the memory 11 a and sent to the depth map correction unit 13. The depth map correction unit 13 performs contour extraction by performing high-pass filter processing on the actual image (S-2). As shown in FIG. 4, the extracted 2D contour image is an image in which the contour is highlighted.

  Next, the depth map acquisition unit 12 acquires a depth map corresponding to the actual image (S-3). For example, as shown in FIG. 5, the depth map is obtained by highlighting the brightness of a pixel having a shallower depth from a certain reference position, while the image has a blurred outline. The acquired depth map is stored in the memory 12a. Note that the depth map may not be acquired simultaneously with the actual image as long as the imaging range and the subject image are substantially the same.

  Next, the obtained depth map is read from the memory 12 a and sent to the depth map correction unit 13. The depth map correction unit 13 performs contour extraction by performing a high-pass filter process on the depth map (S-4). A three-dimensional image obtained by extracting a contour from the depth map (hereinafter referred to as “3D contour image”) is an image in which the contour is highlighted as shown in FIG. 6, but the 2D contour described in FIG. Compared with the image, the outline is jagged in steps. After that, the depth map correction unit 13 performs alignment between the 2D contour image and the 3D contour image, and corrects the contour of the 3D contour image to match the contour of the 2D contour image (S-5).

  The contour correction of the 3D contour image will be described in detail. As shown in FIG. 7, raster scanning is performed from the upper left of the 2D contour image 30 extracted from the actual image, and the contour points of the 2D contour image are sampled. The 2D contour image is normally a grayscale image, and points that exceed a threshold during raster scanning are sampled as contour points. In this way, the contour point of the 2D contour image can be acquired at the sampling rate of the raster scan. Note that the 2D contour image 30 may be extracted by area extraction by labeling processing or edge detection processing instead of the threshold processing described above. Thereafter, the 3D contour image 31 is extracted from the depth map by similar processing as shown in FIG.

  Each point of the contour of the 2D contour image is associated with the contour of the 3D contour image. This associates each contour point of the 2D contour image with the contour point closest to the 3D contour image as a similar contour point. For example, for a dice contour among the objects included in the 2D contour image, a contour point A (see FIG. 7) at the position (x, y) of the 2D contour image is included in the contour of the 3D contour image. Is associated with a point A ′ (see FIG. 8) at a position (x1, y1) that is in the vicinity of the position (x, y).

  With the above method, each point of the contour of the 3D contour image is associated with each point of the contour of the 2D contour image, and each point of the contour 32 of the 3D contour image shown in FIG. Moving to the position of the contour point of the contour image, the contour of the 3D contour image is corrected as shown in FIG. For this correction, known techniques such as lens distortion correction and coordinate conversion used in calibration can be used.

  The corrected 3D contour image is sent to the three-dimensional model estimation unit 14. The occlusion processing unit 14a of the three-dimensional model estimation unit 14 estimates the shape of the occlusion part included in the 3D contour image and interpolates the 3D contour image (S-6). The interpolated 3D contour image is converted from the coordinate system of the 3D contour image to the world coordinate system for shading processing by the coordinate conversion unit 14b (S-7). The coordinate-converted 3D contour image is sent to the reflectance setting processing unit 14c.

  Reflectivity information for each object (subject) included in the 3D contour image is input from the reflectivity setting unit 19 to the reflectivity setting processing unit 14c (S-8). Information on the object estimated by the object estimation unit 17 based on the actual image is input to the reflectance setting unit 19 (S-9). The reflectance setting unit 19 identifies the reflectance with reference to the DB 18a based on the identified object. The reflectance setting processing unit 14c sets the reflectance on the surface of the object included in the 3D contour image based on the reflectance information for each object (S-10). Information on the reflectance set on the surface of each object is sent to the shading processing unit 15.

  The target portion for providing a brightening effect is designated in advance for the actual image using the emphasis designating unit 20 (S-11). Note that the order in which the emphasizing parts are specified is not limited to the order described above, and may be the first of the procedures described in FIG. In this case, the coordinates of the designated target portion may be temporarily stored in the memory. The light source estimation unit 21 estimates light source information for providing a brightening effect on the specified range of the 3D contour image corresponding to the target portion based on the target portion specified by the enhancement specifying unit 20 (S). -11).

  An example of a procedure for estimating the light source information will be described with reference to FIG. The real image 33 is displayed on a display unit, for example. By operating the emphasis designating unit 20, as shown in FIG. 11, a range 36 ′ in which a light effect is applied to the displayed real image 33 is designated. As illustrated in FIG. 12, the light source estimation unit 21 divides the designated range 36 in the 3D contour image 35 corresponding to the range 36 ′ designated in the actual image 33 into a plurality of triangular meshes 37 (S-13). ). Thereafter, as shown in FIG. 13, the surface normal vectors 38 of the divided meshes 37 are respectively calculated (S-14), and the sum of the plurality of surface normal vectors 38 is calculated to calculate the composite vector 39. (S-15). Then, as shown in FIG. 14, the light source estimation unit 21 estimates the virtual light source 40 so as to be defined on the direction line indicated by the composite vector 39 (S-16).

  The light source information estimated by the light source estimation unit 21 is sent to the shading processing unit 15. The shading processing unit 15 defines the virtual light source 40 in the virtual space of the 3D contour image based on the estimated light source information, and further, based on the reflectance information set on the surface of each object, as shown in FIG. In addition, a shadow image is generated with a shadow 41 on the opposite side of the object from the virtual light source 40 (S-17). The generated shadow image is converted into a two-dimensional shadow image viewed from the same direction as the camera position of the actual image and output.

  The blending unit 16 receives the shadow image and the real image, blends these two images, and generates a virtual image in which a virtual shadow is added to the real image (S-18). Then, the image is output to the display unit or printer by the blend image output unit 23 (S-19). Thereby, as shown in FIG. 16, the virtual image 43 which gave the bright effect with respect to the target range desired among real images can be obtained.

[Second Embodiment]
In the second embodiment, the emphasis designating unit 20 is used to designate a subject for which a light effect is to be applied to an actual image as a target part, thereby illuminating the boundary of the subject designated by the light source estimating unit 21 from behind. Estimate the light source, eg, Back Light.

  Specifically, as illustrated in FIG. 17, the light source estimation unit 21 performs clustering processing for each subject based on the actual image and the depth map (S-17). For example, as shown in FIG. 18, the clustering process is performed as a set of clusters having the same distance based on feature (color and luminance information) information extracted from a real image and distance information extracted from a depth map. A 3D contour image 45 is generated by extracting contour images of a plurality of subjects A to D. Thereafter, the depth map correction unit 13 may correct the 3D contour image 45, but may omit it.

  After that, as shown in FIG. 19, for example, a desired subject D ′ is designated using the emphasis designating unit 20 for the real image 48 displayed on the display unit (S-18). As the designation, the subject D ′ may be touch-operated or may be designated to trace the outline 49 of the subject D ′. Thereafter, based on the range including the designated subject D ′, a contour image of the corresponding subject is selected from the clustered subject contour images from the 3D contour image 45 (S-19).

  After that, as shown in FIG. 20, the virtual light source 46 defined behind the subject D is estimated so that the boundary portion of the selected subject D is brightly emphasized (S-20). Thereafter, the shading processing unit 15 defines the virtual light source 46 behind the clustered subject D based on the estimated light source information, and the subject surface based on the reflectance information set on the surface for each object. By calculating the upper luminance, the subject is rendered to generate a three-dimensional shadow image. Then, the shading processing unit 15 outputs a two-dimensional shadow image from the three-dimensional shadow image. The blending unit 16 generates a virtual image by synthesizing a two-dimensional shadow image with a real image. As a result, as shown in FIG. 21, a virtual image 50 can be obtained in which a light effect in which the boundary 49 of the desired subject D is illuminated with natural brightness from behind is provided.

  By the way, conventionally, in the contour emphasis processing in the two-dimensional image processing application software, for example, the contour of the designated subject is entirely emphasized, and thus unnaturalness remains. In the above embodiment, not only a solid object such as a solid object but also a solid object such as a hair, a kimono, or a cloth is not a solid object that can not be obtained by two-dimensional image processing. A feeling can be created. Further, according to the present invention, it is possible to give a natural shadow in consideration of a sense of distance to the target portion or the boundary of the subject, that is, an effect such as shading not only the subject but also surrounding subjects. Further, while the boundary between subjects wearing clothes of the same color cannot be distinguished, the present invention can be distinguished because a depth map (distance information) is used.

  In the above embodiment, the light source estimation unit 21 estimates position information out of information related to the virtual light source. However, in the present invention, not only the position information but also information such as distribution, wavelength, and direction is estimated. Also good. Further, in order to easily change information such as distribution, wavelength, and orientation, for example, a light changing unit that changes the type of light such as a point light source or a surface light source may be provided in the light source information change operation unit 22.

  In the light source estimation unit 21 of each of the embodiments described above, the information on the virtual light source is estimated so as to give a brightening effect to the target part. However, the present invention is not limited to this, and for example, a shadowing effect is given. The virtual light source information may be estimated. In the present invention, contour correction in the depth map correction unit 13 may be omitted.

  Although the present invention has been specifically described above based on the preferred embodiments, it is needless to say that the present invention is not limited to the above-described ones, and various modifications can be made without departing from the scope of the present invention.

  In each of the above embodiments, the image creating apparatus is described. However, the present invention may be an image creating method. Furthermore, the present invention may be an invention of a program for causing a computer to execute the image creation method and a computer-readable recording medium on which the program is recorded.

DESCRIPTION OF SYMBOLS 10 Image production apparatus 14 3D model estimation part 15 Shading process part 20 Enhancement designation | designated part 21 Light source estimation part

Claims (8)

Real image acquisition means for acquiring a real image;
3D information acquisition means for acquiring 3D information including depth information corresponding to the real image;
3D image generation means for generating a 3D image based on the real image and the 3D information;
Designating means for designating a target portion to effect the real image with light;
Estimating means for estimating information on a virtual light source for performing the effect based on the target portion specified by the specifying means;
Shading means for generating shadow information by defining the virtual light source in the virtual space of the three-dimensional image based on the information estimated by the estimating means;
Blending means for combining the shadow information with the real image;
An image creating apparatus comprising: The image creating apparatus according to claim 1,
An image creating apparatus comprising light source information changing means for changing information on the virtual light source. In the image creation device according to claim 1 or 2,
The designation means is means for designating a range to be emphasized,
Means for dividing the range in the three-dimensional image corresponding to the range specified by the specifying means;
Means for calculating a surface normal vector of each of the divided meshes;
Means for calculating a composite vector by taking the sum of the plurality of surface normal vectors;
Means for estimating the virtual light source for definition on a direction line indicated by the combined vector;
An image creating apparatus comprising: In the image creation device according to claim 1 or 2,
The designation means is a means for designating a subject to be emphasized or a boundary of the subject,
The estimation means includes
Means for performing clustering processing on a region including the subject specified by the specifying unit or a region in the three-dimensional image corresponding to the region including the boundary to extract the region;
Means for estimating the virtual light source for definition behind the extracted region;
An image creating apparatus comprising: In the image creation device according to any one of claims 1 to 4,
The three-dimensional image generation means includes
Contour extracting means for extracting contour images from the real image and the three-dimensional information;
Correction means for correcting the three-dimensional information so as to match the contour of the three-dimensional information with the contour of the real image to generate a three-dimensional image;
An image creating apparatus comprising: The image creating apparatus according to claim 5.
A reflectance setting means for setting the reflectance specified by specifying the reflectance of the object included in the real image for each object included in the three-dimensional image after correction by the correction means;
An image creating apparatus comprising: A real image acquisition step of acquiring a real image;
A three-dimensional information acquisition step for acquiring three-dimensional information including depth information corresponding to the real image;
A three-dimensional image generation step of generating a three-dimensional image based on the real image and the three-dimensional information;
A designation step for designating a target portion to be rendered with light on the real image;
An estimation step for estimating information on a virtual light source for performing the effect based on the designated target portion;
A shading step of generating shadow information by defining the virtual light source in the virtual space of the three-dimensional image based on the estimated information;
A blending step for combining the shadow information with the real image;
An image creation program for causing a computer to execute.   A computer-readable recording medium on which the image creating program according to claim 7 is recorded.