WO2011096136A1

WO2011096136A1 - Simulated image generating device and simulated image generating method

Info

Publication number: WO2011096136A1
Application number: PCT/JP2010/072529
Authority: WO
Inventors: 修遠山; 卓也川野
Original assignee: コニカミノルタホールディングス株式会社
Priority date: 2010-02-02
Filing date: 2010-12-15
Publication date: 2011-08-11

Abstract

Disclosed is a technology capable of generating simulated images, with reduced uncanny valleys, from images of photographic subjects that correspond to photography from virtual viewpoints. A simulated image generating device comprises a first acquisition means for acquiring reference images wherein photographic subjects are photographed from a first viewpoint; a second acquisition means for acquiring information of all distances at least on the basis of actual measurements of subjects that are being focused upon; a distinguishing means for distinguishing between the subjects that are being focused upon and each background subject; a correspondence acquisition means for acquiring correspondences between the reference images and simulated images; a first generating means for generating the simulated images on the basis of the correspondences and the reference images, the simulated images further comprising second foreground images for the subjects that are being focused upon and second background images for each background subject; a first identifying means for identifying occlusion regions within the simulated images; and a second generating means for generating images of the occlusion regions on the basis of the respective information of the subjects that are being focused upon and of each background subject.

Description

Pseudo image generation apparatus and pseudo image generation method

The present invention relates to a pseudo image generation apparatus and method for generating a pseudo image of an image of a subject taken from a virtual viewpoint different from the viewpoint using an image of the subject taken from one viewpoint.

In recent years, a pseudo image has been generated in which a pseudo image of an image obtained when a subject is photographed from a virtual viewpoint different from the viewpoint where the subject is actually photographed is simulated without performing actual photographing from the virtual viewpoint. Generation devices are beginning to be used for purposes such as generating a group of images that can be viewed stereoscopically.

In the pseudo image generation device of Patent Document 1, the depth of the subject (depth estimation model) is estimated from the screen configuration of one captured image (reference image), and on the image of the reference image based on the obtained depth information. The pseudo image is generated from the reference image by obtaining the correspondence between each of the coordinates and each coordinate on the image of the pseudo image.

Here, since it is not possible to obtain the correspondence between the reference image and the pseudo image for the subject area that is visible from the virtual viewpoint but not from the viewpoint where the reference image is taken, the pseudo image corresponding to this area cannot be obtained. An appropriate pixel value cannot be obtained for the region (occlusion region) depending on the correspondence.

Therefore, in the pseudo image generation apparatus of Patent Document 1, after dividing the screen area by the area integration method, the pixel value of the occlusion area is set using the statistical amount of the texture in each area.

JP 2005-151534 A

However, the pseudo image generation apparatus of Patent Document 1 cannot obtain an appropriate correspondence because the correspondence between the reference image and the pseudo image is obtained based on the estimated depth. Therefore, since the range of the occlusion area is not accurate, there is a problem that an observer who sees the generated pseudo image feels uncomfortable.

In addition, the occlusion area normally includes information on both the subject and its background. However, in the pseudo image generation device of Patent Document 1, the occlusion area contains information on the subject and its background. In particular, since the improvement in image quality is not aimed at, there is a problem that the frequency of generation of an image of an occlusion area in which an observer is likely to feel discomfort increases.

The present invention has been made to solve these problems, and an object of the present invention is to provide a technique for more accurately specifying the range of an occlusion area and generating a pseudo image with less discomfort.

In order to solve the above-described problem, the pseudo image generation device according to the first aspect includes a first acquisition unit that acquires a reference image in which each subject is captured from a first viewpoint, and at least attention among the subjects. A second acquisition unit configured to acquire distance information based on actual measurement for each point of the subject; the target subject; and a first portion that is a background portion of a first foreground image that is an image of the target subject among the reference images. The identification means for identifying each background subject photographed in the background image of the image, the correspondence relationship between the reference image and the pseudo image of each subject corresponding to photographing from a virtual viewpoint different from the first viewpoint From the virtual viewpoint based on the correspondence relationship acquisition means for acquiring information based on each distance information, the correspondence relationship for at least the first foreground image of the reference image, and the reference image. A first pseudo-image is generated that includes a second foreground image that is a corresponding image of the subject of interest and a second background image that is an image of each background subject corresponding to shooting from the virtual viewpoint. Generating means; first specifying means for specifying an occlusion area that does not include the second foreground image and the second background image of the pseudo image; and an image of the occlusion area as the subject of interest. And second generation means for generating based on the respective information on each background subject.

The pseudo image generation device according to the second aspect is the pseudo image generation device according to the first aspect, and corresponds to the first region corresponding to the subject of interest in the occlusion region and the background subjects. Second specifying means for specifying the second area is further provided. The second generation means generates an image of the first area based on information on the subject of interest, and generates an image of the second area based on information on each background subject.

The pseudo image generation device according to the third aspect is the pseudo image generation device according to the second aspect, and is third acquisition means for acquiring shape information representing the entire three-dimensional shape of the object of interest. Is further provided. The second specifying means specifies the first region based on the shape information.

The pseudo image generation device according to a fourth aspect is the pseudo image generation device according to the second aspect, wherein the second generation means is a boundary region with the first region in the second foreground image. The image of the first area is generated based on the second area image, and the image of the second area is generated based on a boundary area between the second background image and the second area.

The pseudo image generation device according to a fifth aspect is the pseudo image generation device according to the fourth aspect, wherein the second generation means is a boundary region on the second region side of the first region. From the second region, the images of the first region and the second region are generated so that the pixel values of the region over the boundary region on the first region side in the second region gradually change.

The pseudo image generation device according to a sixth aspect is the pseudo image generation device according to the first aspect, wherein the second generation means includes: (a) the second foreground image in the occlusion area; An image of the first boundary region is generated based on a boundary region with the occlusion region in the second foreground image, and a second boundary region with the second background image in the occlusion region is generated. An image is generated based on a boundary area with the occlusion area in the second background image, and (b) a pixel value of the occlusion area gradually increases from the first boundary area to the second boundary area. An image of the occlusion area is generated so as to change to

A pseudo image generation device according to a seventh aspect includes a first acquisition unit configured to acquire a plurality of time-series images in which each subject is photographed in time sequence, and one image among the plurality of time-series images as a reference image. Second acquisition means for acquiring distance information based on actual measurement for at least each point of the subject of interest in the state in which the reference image has been acquired, the subject of interest, and the reference image of the subject Identification means for identifying each background subject photographed in the first background image that is the background portion of the first foreground image that is the image of the subject of interest, the reference image, and the first image in which the reference image is photographed A correspondence relationship acquisition means for acquiring a correspondence relationship with the pseudo image of each subject corresponding to shooting from a virtual viewpoint different from the viewpoint based on each distance information, and at least the first of the reference images Based on the correspondence relationship for the foreground image and the reference image, the second foreground image, which is the image of the subject of interest corresponding to the shooting from the virtual viewpoint, and the shooting from the virtual viewpoint First generation means for generating the pseudo image containing a second background image that is an image of each background subject; the second foreground image of the pseudo image; and the second background image; First occlusion area that does not contain the image, and second generation means for generating an image of the occlusion area based on the plurality of time-series images.

The pseudo image generation device according to the eighth aspect is the pseudo image generation device according to the first aspect, and the second generation means performs a smoothing process on the generated image of the occlusion area.

The pseudo image generation device according to the ninth aspect is based on actual measurement for at least each point of the subject of interest in the first acquisition unit that acquires a reference image in which each subject is captured from a first viewpoint. A second acquisition unit that acquires each distance information, a third acquisition unit that acquires shape information that represents the entire three-dimensional shape of the subject of interest, and a virtual viewpoint different from the first viewpoint A first region corresponding to the subject, and a target region of interest. The occlusion region is a region where the corresponding portion of the reference image is not photographed among the pseudo images of each subject corresponding to photographing from A second region corresponding to each background subject photographed in the background portion of the first image based on the reference image, each distance image, and the shape information, and Picture The generated based on the information of the focused object, and a generation means for generating the pseudo-image by generating on the basis of an image of the second region on the information of the background object.

A pseudo image generation device according to a tenth aspect is the pseudo image generation device according to the ninth aspect, wherein the generation means includes (a) an image of the target subject and the target subject among the reference subject. Identification means for identifying each background subject photographed in the first background image that is the background portion of a certain first foreground image, and (b) the correspondence between the reference image and the pseudo image, Correspondence acquisition means for acquiring based on distance information; and (c) shooting from the virtual viewpoint based on the correspondence between at least the first foreground image of the reference images and the reference image. A first pseudo-image is generated that includes a second foreground image that is a corresponding image of the subject of interest and a second background image that is an image of each background subject corresponding to shooting from the virtual viewpoint. And (d) the generating means 1 region is specified based on the distance information and the shape information, and the second region is defined as the second foreground image, the first region, and the second region of the pseudo image. (E) generating an image of the first area based on the information of the subject of interest, and specifying the image of the second area as the respective areas. Second generation means for generating based on information on the background subject.

The pseudo image generation method according to the eleventh aspect includes a step of obtaining a reference image in which each subject is photographed from a first viewpoint, and each distance information based on actual measurement for at least each point of the subject of interest among the subjects. And obtaining the subject of interest and identifying each background subject photographed in a first background image that is a background portion of a first foreground image that is an image of the subject of interest in the reference image. Obtaining a correspondence relationship between the reference image and a pseudo image of each subject corresponding to photographing from a virtual viewpoint different from the first viewpoint based on each distance information; and A second foreground image that is an image of the subject of interest corresponding to shooting from the virtual viewpoint based on at least the correspondence relationship with respect to the first foreground image and the reference image, and the temporary image Generating the pseudo image containing a second background image that is an image of each background subject corresponding to shooting from a viewpoint; the second foreground image of the pseudo image; and the second image A step of specifying an occlusion area not including a background image, and a step of generating an image of the occlusion area based on information on the subject of interest and each of the background subjects.

The pseudo image generation method according to the twelfth aspect includes a step of acquiring a plurality of time-series images in which each subject is photographed in time sequence, and using one of the plurality of time-series images as a reference image, the reference image A step of acquiring distance information based on actual measurement for at least each point of the subject of interest in the state in which the image has been obtained, and a first image that is the image of the subject of interest among the subject of interest and the reference image A step of identifying each background subject photographed in the first background image that is the background portion of the foreground image, and a virtual viewpoint different from the reference image and the first viewpoint from which the reference image was photographed Obtaining a correspondence relationship with the pseudo image of each subject corresponding to the shooting of the subject based on the distance information, and the correspondence relationship with respect to at least the first foreground image among the reference images. Based on the reference image, a second foreground image that is an image of the subject of interest corresponding to shooting from the virtual viewpoint and an image of each background subject corresponding to shooting from the virtual viewpoint. Generating the pseudo image including two background images, identifying an occlusion area that does not include the second foreground image and the second background image among the pseudo images, and the plurality Generating an image of the occlusion area based on the time-series images.

The pseudo image generation method according to the thirteenth aspect includes a step of acquiring a reference image in which each subject is photographed from a first viewpoint, and each distance information based on actual measurement for at least each point of the subject of interest among the subjects. A step of acquiring, a step of acquiring shape information representing a three-dimensional shape of the entire circumference of the subject of interest, and a pseudo image of each subject corresponding to photographing from a virtual viewpoint different from the first viewpoint Each occlusion area, which is an area in which the corresponding portion of the reference image is not photographed, and each background subject photographed in the first area corresponding to the subject and the background portion of the image of the subject of interest Is determined based on the reference image, each distance image, and the shape information, and the image of the first area is determined based on the information on the subject of interest. There generated, and a step of generating the pseudo-image by generating on the basis of an image of the second region on the information of the background object.

Occlusion on the pseudo image based on the distance information of the subject based on the actual measurement by the pseudo image generation device according to any of the first to tenth aspects or the pseudo image generation method according to any of the eleventh to thirteenth aspects. The range of the region can be specified more accurately, and the image of the specified occlusion region is generated based on the subject of interest and each background subject, so that a pseudo image with less discomfort can be generated.

It is a block diagram which shows an example of the main structures of the pseudo image generation system which concerns on embodiment. It is a block diagram which shows an example of the main function structures of the pseudo image generation apparatus which concerns on embodiment. It is a figure which shows an example of a reference | standard image. It is a figure which shows an example of a pseudo image. It is a figure which shows an example of distance information. It is a figure which shows an example of the pseudo image based on the subject's perimeter shape information. It is a figure which shows an example of the pseudo image by which the 1st area | region and the 2nd area | region were set to the occlusion area | region. It is a figure which shows an example of the method of producing | generating the image of an occlusion area | region. It is a figure which shows an example of the method of producing | generating the image of an occlusion area | region. It is a figure which shows an example of the method of producing | generating the image of an occlusion area | region. It is a figure which shows an example of the method of producing | generating the image of an occlusion area | region. It is a figure which shows an example of the method of producing | generating the image of an occlusion area | region. It is a figure which shows an example of the method of producing | generating the image of an occlusion area | region. It is a figure which shows an example of the method of producing | generating the image of an occlusion area | region. It is a figure which shows an example of the method of producing | generating the image of an occlusion area | region. It is a figure which shows an example of the method of producing | generating the image of an occlusion area | region. It is a figure which shows an example of the method of producing | generating the image of an occlusion area | region. It is a figure which shows an example of the method of producing | generating the image of an occlusion area | region based on time series image information. It is a figure which shows an example of the operation | movement flow of the pseudo image generation apparatus which concerns on embodiment.

<About embodiment:>
<About the pseudo image generation system 100A:>
FIG. 1 is a block diagram illustrating an example of a main configuration of a pseudo image generation system 100A according to the embodiment.

As shown in FIG. 1, the pseudo image generation system 100A mainly includes a stereo camera 300 and a pseudo image generation device 200A.

◎ About stereo camera 300:
As shown in FIG. 1, the stereo camera 300 mainly includes a base camera 31 and a reference camera 32. In addition, the reference camera 31 and the reference camera 32 are mainly configured by an imaging optical system and a control processing circuit (not shown), respectively.

Further, the reference camera 31 and the reference camera 32 are provided with a predetermined baseline length apart, and the subject information is processed by synchronizing the light ray information from the subject incident on the photographing optical system with a control processing circuit or the like. For example, a standard image 1A and a reference image 1R that are digital images of a predetermined size such as VGA are generated.

The generated standard image 1A and reference image 1R are supplied to the input / output unit 41 of the pseudo image generating apparatus 200A via the data line DL. Various operations of the stereo camera 300 are controlled based on control signals supplied from the pseudo image generation device 200A via the input / output unit 41 and the data line DL.

Note that the stereo camera 300 can also generate a plurality of reference images 1A and a plurality of reference images 1R by continuously photographing the subject in time sequence while synchronizing the reference camera 31 and the reference camera 32. . Further, the standard image 1A and the reference image 1R may be color images or monochrome images.

In the pseudo image generation system 100A, the stereo camera 300 is employed. For example, instead of the reference camera 32 of the stereo camera 300, light projection that projects various detection lights for shape measurement, such as laser light, onto a subject. By adopting the device, the reference camera 31 and the light projecting device may constitute an active distance measuring type three-dimensional measuring machine, and the three-dimensional measuring machine may be used instead of the stereo camera 300.

According to the stereo camera 300 and the coordinate measuring machine, the image about the subject and the image used for the measurement of the distance information can be shared, and therefore the correspondence 56 (see FIG. 2) performed by the correspondence acquisition unit 15 described later. ), The processing cost for associating the image with the distance information can be reduced.

Even if the coordinate measuring machine adopts a configuration that measures the distance information 52 (FIG. 2) about the subject based on an image taken from a predetermined viewpoint different from the reference image 1A, Since the reference image 1A and the distance information 52 can be associated with each other through matching between the image and the reference image 1A, the usefulness of the present invention is not impaired.

About the configuration of the pseudo image generation device 200A:
As shown in FIG. 1, the pseudo image generation device 200A mainly includes a CPU 11A, an input / output unit 41, an operation unit 42, a display unit 43, a ROM 44, a RAM 45, and a storage device 46. This is realized by a computer or a dedicated hardware device.

The input / output unit 41 is configured by an input / output interface such as a USB interface, for example, and inputs image information and the like supplied from the stereo camera 300 to the pseudo image generation device 200A, and from the pseudo image generation device 200A to the stereo camera 300. Output various control signals to

The operation unit 42 includes, for example, a keyboard or a mouse. When the operator operates the operation unit 42, various control parameters are set in the pseudo image generation device 200A, and various operations of the pseudo image generation device 200A. The mode is set.

The display unit 43 includes, for example, a liquid crystal display, and displays various image information such as a reference image 1A supplied from the stereo camera 300 and a pseudo image 4A (FIG. 2) generated by the pseudo image generation device 200A. In addition, various information related to the device and control GUI (Graphical User Interface) are displayed.

ROM (Read Only Memory) 44 is a read-only memory and stores a program for operating the CPU 11A. A readable / writable nonvolatile memory (for example, a flash memory) may be used instead of the ROM 44.

A RAM (Random Access Memory) 45 is a readable and writable volatile memory that stores various images acquired by the first acquisition unit 12, pseudo images generated by the generation unit 21A, and processing information of the CPU 11A. Functions as a temporary work memory.

The storage device 46 is composed of, for example, a readable / writable nonvolatile memory such as a flash memory, a hard disk device, or the like, and permanently records various information such as setting information for the pseudo image generation device 200A.

The storage device 46 is provided with a parameter storage unit 47 and a shape data storage unit 48. The parameter storage unit 47 includes a three-dimensional parameter 51 (FIG. 2), an imaging parameter 54 (FIG. 2), and Various parameters such as coordinate system information 55 (FIG. 2) are stored.

In addition, the shape data storage unit 48 stores model group shape data 61 (FIG. 2) that represents the overall three-dimensional shape of each of various subjects, as will be described later. It is referred to by the third acquisition unit 14 and used for the acquisition process of the shape information 62 (FIG. 2) about the subject of interest.

The CPU (Central Processing Unit) 11A is a control processing device that controls each functional unit of the pseudo image generation device 200A, and executes control and processing according to a program stored in the ROM 44.

The CPU 11A also functions as the first acquisition unit 12, the second acquisition unit 13, the third acquisition unit 14, the correspondence acquisition unit 15, and the generation unit 21A, as will be described later.

With these functional units, the CPU 11A uses the reference image 1A for the subject photographed from the first viewpoint to the pseudo image 4A for the subject corresponding to photographing from a virtual viewpoint different from the first viewpoint (FIG. 2). Is generated.

Furthermore, the generation unit 21A is configured by functional units such as a first specification unit 22, a second specification unit 23, a first generation unit 24, a second generation unit 25, and an identification unit 26.

Further, each of the CPU 11A, the input / output unit 41, the operation unit 42, the display unit 43, the ROM 44, the RAM 45, the storage device 46, and the like are electrically connected via a signal line 49. Therefore, for example, the CPU 11A can execute control of the stereo camera 300 via the input / output unit 41, acquisition of image information from the stereo camera 300, display on the display unit 43, and the like at a predetermined timing.

In the configuration example shown in FIG. 1, the first acquisition unit 12, the second acquisition unit 13, the third acquisition unit 14, the correspondence relationship acquisition unit 15, and the functional units of the generation unit 21A and the generation unit 21A are configured. The function units of the first specifying unit 22, the second specifying unit 23, the first generating unit 24, the second generating unit 25, and the identifying unit 26 are realized by the CPU 11A executing predetermined programs. Each of these functional units may be realized by a dedicated hardware circuit, for example.

As described above, in the pseudo image generation system 100A, the pseudo image generation device 200A acquires the standard image 1A and the reference image 1R captured by the stereo camera 300, and the pseudo image generation device 200A processes the standard image 1A and the reference image 1R. By doing so, a pseudo image corresponding to shooting from a virtual viewpoint different from the first viewpoint from which the reference image 1A was shot based on the reference image 1A, that is, shot from a virtual viewpoint different from the first viewpoint A pseudo image corresponding to the image of the subject is generated.

Here, in the pseudo image, an occlusion area in which an appropriate pixel value corresponding to shooting from the virtual viewpoint is set in the generation process is generated, but the pseudo image generation apparatus 200A has the occlusion area generated in the pseudo image generation process. Generate

pseudo images

4A and 4B (FIG. 2).

In the present application, an area (pseudo-image occlusion area) on the pseudo image corresponding to the area of the subject (subject occlusion area) that can be taken from the virtual viewpoint but cannot be taken from the viewpoint where the reference image is taken. This is referred to as an “occlusion area”.

<About operation of pseudo image generation apparatus 200A:>
FIG. 2 is a block diagram illustrating an example of a main functional configuration of the pseudo image generation apparatus 200A according to the embodiment.

FIG. 19 is a diagram illustrating an example of an operation flow of the pseudo image generation apparatus 200A according to the embodiment.

Hereinafter, the operation of each functional unit of the pseudo image generation apparatus 200A illustrated in FIG. 2 will be described in detail with reference to the operation flow of FIG. 19 as appropriate.

First, the operator can position and position the stereo camera 300 so that the subject of interest who wants to create a pseudo image corresponding to shooting from a virtual viewpoint can be shot from both the reference camera 31 and the reference camera 32 of the stereo camera 300. Adjust. The position of the reference camera 31 of the stereo camera 300 in this state is the first viewpoint. More specifically, for example, the principal point position of the photographing optical system of the reference camera 31 is the first viewpoint.

When the installation of the stereo camera 300 is completed and the operator clicks the control GUI button displayed on the display unit 43 using the mouse of the operation unit 42 in order to instruct the start of the pseudo image generation operation, A control signal corresponding to the button operation is supplied to the CPU 11A.

When this control signal is supplied to the CPU 11A, the CPU 11A supplies a control signal for causing the stereo camera 300 to perform a shooting operation.

The stereo camera 300 to which the control signal is supplied performs a photographing operation using the standard camera 31 and the reference camera 32 to generate the standard image 1A and the reference image 1R for each subject in the photographing field of view, and generates a pseudo image. Supply to apparatus 200A.

○ Operation of the first acquisition unit 12:
Next, the first acquisition unit 12 acquires the reference image 1A and the reference image 1R obtained by shooting each subject from the first viewpoint via the input / output unit 41 (step S10 in FIG. 19).

FIG. 3 is a diagram illustrating an example of the reference image 1A. As shown in FIG. 3, the first foreground image 1a, which is an image of a person facing the front, is captured in the reference image 1A. The background portion of the first foreground image 1a is the first background image 2a obtained by photographing the back wall of the person.

2, the acquired reference image 1A is supplied to the second acquisition unit 13, the correspondence relationship acquisition unit 15, the first generation unit 24, and the identification unit 26. Further, the acquired reference image 1R is supplied to the second acquisition unit 13.

Note that the first acquisition unit 12 may acquire the reference image 1A and the reference image 1R that have been captured in advance and stored in the recording medium via the input / output unit 41.

○ Operation of the second acquisition unit 13:
When the reference image 1 </ b> A and the reference image 1 </ b> R are supplied to the second acquisition unit 13, the second acquisition unit 13 acquires a three-dimensional parameter 51 such as a baseline length and focal length information from the parameter storage unit 47.

The second acquisition unit 13 that has acquired the three-dimensional parameter 51 performs a matching process between the standard image 1A and the reference image 1R to obtain a parallax with the reference image 1R for each pixel of the standard image 1A.

Next, the second acquisition unit 13 converts the parallax for each pixel of the reference image 1A based on the principle of triangulation using the three-dimensional parameter 51, thereby each subject corresponding to each pixel of the reference image 1A. Distance information 52 that is a set of three-dimensional coordinate values for each of the above points is generated.

As the coordinate system of the distance information 52, for example, a camera coordinate system depending on the position and orientation of the stereo camera 300 is employed. As the camera coordinate system of the stereo camera, for example, an XYZ orthogonal coordinate system in which the principal point of the reference camera is the origin and the Z axis is along the optical axis direction of the reference camera is used.

Here, since at least the subject of interest among the subjects is captured by both the base camera 31 and the reference camera 32, the second acquisition unit 13 actually measures at least each point of the subject of interest among the subjects. Is obtained (step S20 in FIG. 19).

FIG. 5 is a diagram illustrating an example of the distance information 52 displayed as the distance image 5A. The distance image 5A shown in FIG. 5 is an image in which the Z-axis coordinates in the distance information 52 corresponding to each pixel of the reference image 1A are used as the pixel value of each pixel. In the distance image 5A in FIG. 5, the unit of the pixel value is meter.

In addition, the dotted line in the distance image 5A displays the outline of the first foreground image 1a on the distance image 5A in order to display the relationship between the pixel value of the distance image 5A and the first foreground image 1a in the reference image 1A in an easy-to-understand manner. It is a supplementary indication.

Here, with respect to each background subject image captured in the background portion of the subject subject image, a part or all of it is limited by the measurement range of a distance measuring device such as a stereo camera, and the reflectance of the background subject. In some cases, the distance information 52 cannot be acquired.

Further, for example, when the shooting angle of view of the base camera 31 and the reference camera 32 is the same, the image of the end region of the base image 1A due to the parallax between the base camera 31 and the reference camera 32 is the reference image 1R. Is not photographed, distance information 52 is not generated for the end region.

However, even in this case, as described later, it is possible to generate a pseudo image with which the observer feels uncomfortable, so that the usefulness of the present invention is not impaired.

For example, by adopting a reference camera 32 having a shooting field angle larger than the shooting field angle of the standard camera 31, all the standard images 1A are included in any region of the reference image 1R. It is also possible.

2, the distance information 52 acquired by the second acquisition unit 13 is supplied to the third acquisition unit 14, the correspondence relationship acquisition unit 15, the second specification unit 23, and the identification unit 26.

In addition, when the reference image is constituted by, for example, each image of a short-distance person, a medium-distance partition, and a long-distance building, not only the occlusion area related to the person and the partition, Although an occlusion area related to the partition and the building also occurs, even in this case, if the method of the present invention is applied to the partition as a subject of interest, the range of the occlusion area related to the partition and the building behind the partition is specified, Image generation can be performed.

○ Operation of the third acquisition unit 14:
When the third acquisition unit 14 receives the distance information 52 from the second acquisition unit 13, the third acquisition unit 14 stores a model group that expresses the entire three-dimensional shape of various subjects stored in advance in the shape data storage unit 48. From the shape data 61, shape data closest to the shape information represented by the distance information 52 is identified, and the identified shape data is acquired as shape information 62 representing the entire three-dimensional shape of the subject of interest (FIG. 19). Step S30).

Note that various methods can be employed for identifying the shape data closest to the distance information 52 for the subject of interest from various shape data. For example, the distance image 5A for the distance information 52 and the model group shape The method disclosed in Japanese Patent Laid-Open No. 2001-143072 that performs the identification by comparing each distance image with respect to each of the data 61 can be employed.

The model group shape data 61 stored in the shape data storage unit 48 is preferably as close as possible to the actual entire circumference shape data of the subject of interest. For example, the standard shape data of “boy”, for example, Even if standard (average) shape data for the subject of interest is stored as the model group shape data 61, the range of the occlusion area corresponding to the subject of interest is usually specified to such an extent that the observer does not feel discomfort. Therefore, the usefulness of the present invention is not impaired.

In addition, when it is known in advance what the subject is, the shape information 62 for the entire circumference of the subject is set in the pseudo image generation device 200A, so that the corresponding shape is obtained from the model group shape data 61. The object shape information 62 may be acquired without searching for the information 62.

2, the shape information 62 acquired by the third acquisition unit 14 is supplied to the second specifying unit 23.

○ Operation of the identification unit 26:
When receiving the reference image 1A and the distance information 52 from the first acquisition unit 12 and the second acquisition unit 13, respectively, the identification unit 26 receives the subject of interest and the first foreground image that is an image of the subject of interest in the reference image 1A. Each background subject photographed in the first background image 2a that is the background portion of 1a is identified, and identification information 53 that is the identification result is generated (step S40 in FIG. 19).

As a method for identifying the subject of interest and each background subject, there are a method of identifying based on image information, a method of identifying based on distance information, and the like.

When identifying based on the distance information, for example, in the reference image 1A, assuming that a portion where the difference in distance information corresponding to each pixel exceeds a predetermined range is a boundary between the subject of interest and the background subject, May be identified, or a portion having irregularities exceeding a predetermined reference may be a subject of interest, and a portion having irregularities that are equal to or smaller than a predetermined reference value may be identified as a background subject.

If the identification is performed based on the distance information, the boundary between the subject image and the background image is unclear because the pattern and color of the subject and the background subject are similar. Even when the subject cannot be identified based only on the image information, the subject and the background subject can be appropriately identified.

Note that even if the target subject and the background subject are identified only by image processing based on the image information, in many cases, the target partial image and the background partial image can be accurately identified, which impairs the usefulness of the present invention. There is nothing.

2, the identification information 53 generated by the identification unit 26 is supplied to the first generation unit 24 and the second generation unit 25.

Note that, as shown in FIG. 2, a pseudo image 2 </ b> A generated by a first generation unit 24 described later is also supplied to the identification unit 26.

Here, as will be described later, the first generation unit 24 extracts only the first foreground image 1a corresponding to the subject of interest from the reference image 1A using the identification information 53, and the virtual viewpoint based on the first foreground image 1a. Is provided with an operation mode for generating a second foreground image 3a (FIG. 4) which is a pseudo image of the subject of interest corresponding to the shooting from

The first generation unit 24 can generate the pseudo image 2A at a lower processing cost than when acquiring the pseudo image 2A for the entire reference image 1A by operating in the operation mode.

When the first generation unit 24 does not execute the operation mode, only the second generation unit 25 among the first generation unit 24 and the second generation unit 25 uses the identification information 53. In this case, The identification unit 26 can also identify the subject of interest and the background subject based on the pseudo image 2A generated by the first generation unit 24.

○ Operation of the correspondence acquisition unit 15:
As shown in FIG. 2, the correspondence acquisition unit 15 supplied with the reference image 1A, the distance information 52, the imaging parameter 54, and the coordinate system information 55 is a virtual image different from the reference image 1A and the first viewpoint. A correspondence 56 with the pseudo image 2A of each subject corresponding to shooting from the viewpoint is acquired based on the distance information 52 (step S50 in FIG. 19).

More specifically, the correspondence relationship 56 is a correspondence relationship between each coordinate on the image of the reference image 1A and each coordinate on the image of the pseudo image 2A.

The shooting parameter 54 and the coordinate system information 55 are stored in the parameter storage unit 47. The shooting parameter 54 includes the reference camera 31, a virtual camera at a virtual viewpoint different from the reference camera 31, and distance information 52. These are imaging parameters such as the focal length, the number of pixels, and the pixel size for each of the distance measuring devices to be measured (stereo camera 300 of the present embodiment).

Also, the coordinate system information 55 is information representing the relationship between the position and orientation of the reference camera 31, the virtual camera, and the distance measuring device.

If the imaging parameter 54 and the coordinate system information 55 are known, distance information 52 corresponding to each pixel of the reference image 1A is obtained even if the position and orientation of the reference camera 31 and the distance measuring device are different. A correspondence relationship when the three-dimensional shape represented by the distance information 52 is perspective-projected on the pseudo image 2A is also obtained.

Accordingly, the correspondence 56 between each pixel of the reference image 1A, that is, the coordinates on the image of the reference image 1A, and each pixel of the pseudo image 2A, that is, the coordinates on the image of the pseudo image 2A is obtained via the distance information 52. be able to.

If one camera is a camera that captures the reference image 1A, such as a stereo camera, and is also a camera that captures an image used for ranging, the reference among the shooting parameters 54 and the coordinate system information 55 is used. Even if the relationship between the position and orientation of the camera 31 and the distance measuring device is unknown, each pixel of the reference image 1A can be associated with the distance information 52, so the correspondence 56 can be obtained.

As shown in FIG. 2, the correspondence 56 acquired by the correspondence acquisition unit 15 is supplied to the first generation unit 24 and used to generate the pseudo image 2A.

Here, in the case where the subject of interest is not planar, the occlusion of the subject due to the parallax of the distance measuring device that measures the distance information 52 based on the principle of triangulation, and the measurement optical system of the distance measuring device For some areas such as the peripheral portion of the first foreground image 1a corresponding to the subject of interest due to a decrease in the amount of light incident on the measurement optical system from the subject due to the tilt angle of the subject surface with respect to the optical axis of Information 52 may not be measurable.

However, even in this case, the ratio of the number of pixels for which distance information 52 is not obtained with respect to the total number of pixels of the first foreground image 1a is usually quite low.

Therefore, based on the distance information about the pixels for which accurate distance information 52 has been obtained by actual measurement, distance information for the pixels for which distance information 52 has not been obtained is estimated, and the distance information 52 is based on the estimated distance information. For the pixels for which the distance information 52 is not obtained, for example, the correspondence relationship 56 can be obtained with higher accuracy than when the distance information 52 is not used by actual measurement. Therefore, the distance image 5A for all the pixels of the first foreground image 1a in the quasi-image 1A. Even if the distance information 52 is not obtained, the usefulness of the present invention is not impaired.

In addition, when the correspondence relationship 56 is not obtained for the pixels of the reference image 1A for which the distance information 52 is not obtained, the second foreground image 3a cannot be formed from the pixels. Can be easily distinguished from the pixels of the first foreground image 1a from which the distance information 52 has been acquired.

Accordingly, as a result of the pixels of the first foreground image 1a that have not acquired the correspondence relationship 56 being erroneously treated as the second background image 4a, the pixel values corresponding to the subject of interest are displayed in the second region 7a corresponding to the background subject. The situation where is set can be easily avoided.

Thus, even if the correspondence relationship 56 is not acquired for the pixels of the first foreground image 1a for which the distance information 52 has not been acquired, the usefulness of the present invention is not impaired.

Needless to say, even if the reference image 1A and the pseudo image 2A have different numbers of pixels, the correspondence relationship 56 is appropriately acquired.

○ Operation of the first generator 24:
FIG. 4 is a diagram illustrating an example of the pseudo image 2A. When the first generation unit 24 is supplied with the reference image 1A, the correspondence relationship 56, and the identification information 53 from the first acquisition unit 12, the correspondence relationship acquisition unit 15, and the identification unit 26, respectively, as shown in FIG. The second foreground image, which is the image of the subject of interest corresponding to the shooting from the virtual viewpoint, based on the correspondence 56 of the first foreground image 1a corresponding to at least the subject of interest of the reference image 1A and the reference image 1A. A pseudo image 2A including 3a and a second background image 4a that is an image of each background subject corresponding to shooting from a virtual viewpoint is generated (step S60 in FIG. 19).

More specifically, even when the correspondence relationship 56 for the first background image 2a is acquired, the first generation unit 24 sets the identification information 53 according to the operation mode set from the operation unit 42 or the like. Referring to FIG. 4, only the second foreground image 3a (FIG. 4) of the pseudo image 2A is generated from the first foreground image 1a (FIG. 3) of the reference image 1A according to the correspondence 56, and the first background image 2a of the reference image 1A is Without using the correspondence relationship 56, the first background image 2a can be used as it is as the second background image 4a in the pseudo image 2A.

Further, even when the correspondence 56 is not obtained, as a result, a pseudo image 2A similar to that when the operation mode is applied is generated.

Note that the pseudo image 2A shown in FIG. 4 is a pseudo image when the operation mode in which the first background image 2a is used as it is as the second background image 4a is selected. Further, the occlusion area 5a in FIG. 4 is an area where neither the second foreground image 3a nor the second background image 4a exists in the pseudo image 2A.

When the first background image 2a for the background subject is not changed and is used as the second background image 4a in the pseudo image 2A, the image of the background subject does not strictly correspond to the positional relationship corresponding to the virtual viewpoint. .

However, since the parallax between the reference image 1A and the pseudo image 2A for the background subject in the distance is smaller than the parallax for the subject of interest, the parallax for the subject of interest focused by the observer corresponds to the virtual viewpoint. As long as it is a value, the viewer feels less uncomfortable with the pseudo image 2A.

In this way, in this operation mode, a pseudo image with less discomfort can be generated at a lower processing cost than when using the correspondence between both the subject portion of interest and the background subject portion, so that the identification information 53 is used. Thus, even if the second foreground image 3a in the pseudo image 2A is obtained only for the subject of interest, the usefulness of the present invention is not impaired.

The generated pseudo image 2A is supplied to the first specifying unit 22 and the second generating unit 25, and is also supplied to the identifying unit 26 as described in the explanation section of the identifying unit 26.

The first generation unit 24 may adopt the shape information 62 instead of the “depth estimation model” and acquire the pseudo image 2A by applying the method of Patent Document 1.

○ Operation of the first specifying unit 22:
When the first specifying unit 22 is supplied with the pseudo image 2A from the first generating unit 24, the second foreground image 3a corresponding to the subject of interest in the pseudo image 2A and the second background image 4a corresponding to each background subject. A region that does not include the is specified as the occlusion region 5a (step S70 in FIG. 19).

Further, as shown in FIG. 2, information on the specified occlusion area 5 a is supplied to the second specifying unit 23 and the second generating unit 25.

The information related to the specified occlusion area 5a may be generated as coordinate information of each pixel included in the area of the occlusion area 5a or each pixel on the boundary of the occlusion area 5a, for example, as shown in FIG. It may be generated as an image such as the pseudo image 2A shown.

○ Operation of the second specifying unit 23:
As illustrated in FIG. 2, the second specifying unit 23 includes the pseudo image 2A from each of the first generating unit 24, the first specifying unit 22, the parameter storage unit 47, the second acquiring unit 13, and the third acquiring unit 14. The occlusion area 5a, the imaging parameter 54, the coordinate system information 55, the distance information 52, and the shape information 62 are supplied.

Based on these pieces of information, the second specifying unit 23 specifies the first region 6a related to the subject of interest in the occlusion region 5a and the second region 7a related to the background subject (step S80 in FIG. 19).

FIG. 6 is a diagram showing an example of the pseudo image 6A related to the shape information 62 of the entire circumference of the subject.

The shape region 6aA shown in FIG. 6 is a region of the three-dimensional shape on the pseudo image related to the virtual viewpoint when the three-dimensional shape represented by the shape information 62 is installed at the same position and posture as the subject of interest.

When the shape information 62 is acquired by the third acquisition unit 14, the posture information of the three-dimensional shape represented by the distance information 52 can also be acquired.

Here, since the three-dimensional shape represented by the distance information 52 and the three-dimensional shape represented by the shape information 62 are three-dimensional shapes for the same subject of interest, the three-dimensional shape represented by the shape information 62 is The same position and posture as the position and posture in the three-dimensional space of the three-dimensional shape represented by the distance information 52 can be given.

Therefore, the correspondence between the three-dimensional shape represented by the shape information 62 and the three-dimensional shape image represented by the shape information 62 formed on the pseudo image related to the virtual viewpoint by perspective projection is expressed by the shooting parameter 54 and the coordinate system information 55. Can be obtained using

Based on the correspondence, the second specifying unit 23 specifies the shape area 6aA on the pseudo image related to the virtual viewpoint, and assigns a predetermined pixel value only to the shape area 6aA, for example. A pseudo image 6A expressing 6aA is generated.

As the method for the second specifying unit 23 to generate the pseudo image 6A from the shape information 62, the method disclosed in Japanese Patent Laid-Open No. 10-293862 may be employed.

FIG. 7 is a diagram showing an example of the pseudo image 3A in which the first area 6a and the second area 7a are set in the occlusion area 5a.

The second specifying unit 23 is a first region that is an occlusion region corresponding to the subject of interest on the occlusion region 5a based on the generated shape region 6aA and information on the occlusion region 5a supplied from the first specifying unit 22. For example, an area that does not include the first area 6a in the occlusion area 5a is specified as the second area 7a that is an occlusion area related to the background subject.

That is, the second specifying unit 23 specifies the first region 6a corresponding to the subject of interest in the occlusion region 5a based on the shape information 62, and further specifies the second region 7a corresponding to each background subject.

Information relating to the identified first region 6a and second region 7a is supplied to the second generation unit 25.

Note that the occlusion region 5a is defined based on, for example, a predetermined ratio based on statistical data such as an area ratio between the first region 6a and the second region 7a or a pixel number ratio in the horizontal or vertical direction of 1: 3. Even if the first area 6a and the second area 7a are set so as to have a ratio, usually, the first area 6a that is the range of the occlusion area corresponding to the subject of interest and the occlusion area corresponding to the background object Since the second region 7a as the range can be specified to such an extent that the observer does not feel uncomfortable, the usefulness of the present invention is not impaired.

For example, even if the first area 6a and the second area 7a in the occlusion area 5a are set based on a predetermined ratio not based on statistical data or the like, the first area 6a corresponding to the subject of interest in the occlusion area 5a. Compared to the case where no is set, the viewer can specify the level so as not to feel uncomfortable, so that the usefulness of the present invention is not impaired.

Note that the information related to the identified first region 6a and second region 7a may be generated as coordinate information of each pixel included in these regions or each pixel on the boundary between these regions, for example. The image may be generated as an image such as the pseudo image 3A shown in FIG.

Further, in the above-described identification method of the first area 6a and the second area 7a, the information of the occlusion area 5a is used. However, the second identification section 23 uses the occlusion area according to the operation mode set from the operation section 42. The first region 6a and the second region 7a can be specified without using the information of 5a.

Specifically, the second specifying unit 23 first specifies the shape region 6aA by, for example, the above-described method, and the second foreground of the pseudo image 2A supplied from the first generation unit 24 in the shape region 6aA. An area not including the image 3a is specified as the first area 6a.

Next, the second specifying unit 23 specifies a region that does not include any of the second foreground image 3a, the first region 6a, and the second background image 4a in the pseudo image 2A as the second region 7a. Thus, the first region 6a and the second region 7a are specified without using the information of the occlusion region 5a.

That is, the second specifying unit 23 specifies the first area 6a based on the distance information 52 and the shape information 62, and specifies the second area 7a as the second foreground image 3a and the first area in the pseudo image 2A. 6a and the second background image 4a also function as a specifying means for specifying as an area that does not include both.

○ Operation of the second generation unit 25:
As illustrated in FIG. 2, the second generation unit 25 includes the pseudo image 2A, the occlusion area 5a, and the first area from the first generation unit 24, the first specification unit 22, the second specification unit 23, and the identification unit 26. 6a, the second area 7a, and the identification information 53 are supplied.

The second generation unit 25 generates the pseudo image 4A (FIG. 2) by generating images of the first region 6a and the second region 7a from these pieces of information according to the operation mode input from the operation unit 42. (Step S100 in FIG. 19).

In addition, the second generation unit 25 generates an image of the occlusion region 5a without using information for specifying the first region 6a and the second region 7a according to the operation mode input from the operation unit 42. The pseudo image 4B (FIG. 2) can also be generated.

In the occlusion area 5a of the pseudo image 2A supplied to the second generation unit 25, an appropriate pixel value has not yet been set. In order to make it difficult for the observer to feel uncomfortable with the pseudo image, it is necessary to generate an appropriate image for the occlusion region 5a.

Here, the occlusion area normally includes information on both the subject of interest and each background subject.

Therefore, the second generation unit 25 uses the occlusion area 5a or the images of the first area 6a corresponding to the target subject in the occlusion area 5a and the images of the second area 7a corresponding to the background subjects as the target subject and each background subject. It generates based on each information.

As a method for generating the occlusion area 5a or the first area 6a and the second area 7a based on the information of the subject of interest and each of the background subjects, for example, the subject of interest in an image such as the reference image 1A or the pseudo image 2A A method of generating based on image information relating to each background subject is employed.

Also, in a situation where it is known in advance what the target subject and the background subject are, for example, the colors and patterns of the target subject and the background subject are displayed from the operation unit 42 without being based on the image information. When the operation signal is input from the operation unit 42 to instruct an image generation process for each occlusion area using the color, pattern, or the like according to the setting, the second generation unit 25 causes the color according to the setting. Based on the pattern or the like, an image of the occlusion area 5a or the first area 6a and the second area 7a is generated.

As described above, the second generation unit 25 generates an image of the occlusion region 5a or the first region 6a and the second region 7a based on the image information and characteristics of the subject of interest and the background subject. To do.

That is, the second generation unit 25 generates an image of the occlusion area 5a or the areas of the first area 6a and the second area 7a based on the respective information of the subject of interest and the background object.

Note that when the images of the occlusion area 5a, the first area 6a, and the second area 7a are generated based on the image information of the subject of interest and each of the background objects, each of the images based on either the reference image 1A or the pseudo image 2A. An image of the occlusion area may be generated.

Specifically, for example, a method of determining an area used for generating an image of each occlusion area and generating an image of each occlusion area based on the image of the area is adopted.

FIG. 14 is a diagram illustrating an example of a technique for generating an image of the occlusion area 5b based on the partial area 8g provided in the second foreground image 3b.

FIG. 15 is a diagram showing an example of a technique for generating an image of the occlusion area 5b based on the partial area 8h provided in the second background image 4b.

In FIG. 14, the image of the occlusion area 5b is generated by copying the texture of the partial area 9a of, for example, 3 × 3 pixels provided in the partial area 8g to the partial area 9b.

The image of the partial area 9a may be copied not only to the partial area 9b but also to other partial areas in the occlusion area 5b.

Further, in FIG. 15, the image of the occlusion area 5b is generated by copying the texture of the partial area 9c to the partial area 9b as in FIG.

In addition to the examples shown in FIG. 14 and FIG. 15, for example, a pixel value (mode) indicating a mode value of frequency when a histogram of pixel values in a predetermined area in a non-occlusion area is taken, or A method of generating an image of an occlusion area using an average value of pixel values of a predetermined area may be employed.

In addition, the second generation unit 25 switches the operation mode using the operation unit 42, so that the occlusion area 5 a or the first occlusion area 5 a is generated based on the boundary area with the occlusion area 5 a in the image related to the subject of interest and the background subject. Images of the region 6a and the second region 7a can be generated.

・ About border areas:
The term “boundary region” in the present invention will be described below.

The occlusion areas such as the occlusion area 5a, the first area 6a, and the second area 7a according to the present embodiment are such that the boundary portion between the image of the subject of interest and the image of the background subject is displayed on the image when the pseudo image is generated. This is a region generated by separation.

Therefore, in order to generate a pseudo image that makes it difficult for the observer to feel uncomfortable, it is possible to generate an image of each occlusion area based on the vicinity of the boundary with each occlusion area in a non-occlusion area that is in contact with each occlusion area. desirable.

For example, the boundary area for a three-dimensional subject such as a person is obtained by obtaining a normal defined based on the distance information 52 for each pixel in the target area, for example, and the normal angle for the pixel at the boundary of the target area A method is adopted in which the pixel is determined as a partial region of the region to which each pixel to which the difference is equal to or less than a predetermined angle range.

For example, 45 degrees is adopted as the predetermined angle range. It is desirable that the smaller the predetermined angle range is, the closer the maximum range of the boundary region set is to the boundary of the region of interest.

Since the subject of interest is often three-dimensional and often close to the camera, it is desirable to set the boundary region range based on the angular range such as the normal line described above.

The normal line for the target pixel is obtained based on the distance information 52, for example, based on the distance information 52 for the target pixel and each of the two pixels adjacent to the target pixel in the horizontal and vertical directions. A plane is defined from the three-dimensional coordinate values, and a normal of the plane is acquired as a normal for the pixel of interest.

The setting of the boundary region based on the normal angle described above may also be adopted for the boundary region of the background part.

In addition, since the background part is usually far from the subject of interest, the distance information 52 about the background part is often not obtained, and the background part is often flat. For example, the boundary area of the second background image 4a is directed from the boundary with the occlusion area 5a of the second background image 4a toward the inside of the second background image 4a at a predetermined ratio with respect to the number of horizontal pixels or the number of vertical pixels of the second background image 4a. For example, a method of determining as a partial region of regions set based on the number of pixels may be employed. For example, 1/5 is adopted as the predetermined ratio.

Also, for the subject of interest, the maximum range of the boundary area may be determined by the number of pixels described above.

That is, the “boundary region” in the present application is a predetermined condition that defines a range of a predetermined geometric characteristic related to the subject, such as the predetermined normal angle range described above, or a region range such as the number of pixels of the region, the size, etc. Based on a predetermined mathematical condition to be determined, a partial region of a region whose maximum range is a range set from the boundary between two regions on the image to the inside of one of the two regions .

Therefore, the “boundary area” is not limited to a partial area in contact with the boundary.

The second generation unit 25 is configured to be able to implement several types of occlusion region image generation methods using the boundary region, and these functions are switched by an input from the operation unit 42.

8 to 13, 16, and 17 are diagrams illustrating an example of a technique for generating an image of an occlusion area using a boundary area.

In FIG. 8, the image of the first region 6a is generated based on the boundary region set as the boundary 8a between the second foreground image 3a and the first region 6a. In FIG. 9, the second background image 4a in the second background image 4a is generated. An image of the second region 7a is generated based on the boundary region set as the boundary 8b with the two regions 7a.

FIG. 10 shows an example in which an image of the occlusion area 5b is generated based on the partial area 8c that is a boundary area in the second foreground image 3b.

FIG. 11 shows an example in which an image of the occlusion area 5b is generated based on the partial area 8d that is a boundary area in the second foreground image 3b that is in contact with the boundary between the second foreground image 3b and the occlusion area 5b. Is shown.

12 and 13 show an example in which an image of the occlusion area 5b is generated based on the

partial areas

8e and 8f that are boundary areas provided around the occlusion area 5b in contact with the second foreground image 3b. Yes.

In these examples, the partial area 8e is a boundary area that is not in contact with the boundary between the occlusion area 5b and the second background image 4b, and the partial area 8f is in contact with the boundary between the occlusion area 5b and the second background image 4b. This is the boundary area.

In the example shown in FIGS. 8 to 13, when an image of the occlusion area is generated based on each boundary area, the mode value or average value of the pixel values for each boundary area as described above. For the occlusion area 5b, or by copying the image of the partial area in the boundary area into the occlusion area 5b as described with reference to FIGS. Is generated.

In the example shown in FIG. 16, the second generation unit 25 first selects an image of the first boundary region (not shown) set near the boundary with the second foreground image 3b in the occlusion region 5b. 2 generated based on the partial region group 9d which is a boundary region with the occlusion region 5b in the foreground image 3b, and is set near the boundary with the second background image 4b in the occlusion region 5b. An image of the boundary region is generated based on the partial region group 9e that is a boundary region with the occlusion region 5b in the second background image 4b.

Next, the second generation unit 25 generates an image of the occlusion area 5b so that the pixel value of the occlusion area 5b gradually changes from the first boundary area to the second boundary area.

Here, the arrow 12a indicates the shift direction when the second foreground image 3b is generated based on the correspondence 56, and an image of the occlusion area 5b is generated along the shift direction.

According to the method shown in FIG. 16, the image of the occlusion area 5b is generated so that the pixel value of the occlusion area 5b gradually changes from the first area 6b to the boundary area of the second area 7b. A small number of pseudo images can be generated.

In the example shown in FIG. 17, the second generation unit 25 starts from the partial region group 9f that is the boundary region on the second region 7b side in the first region 6b, and on the first region 6b side in the second region 7b. The image of the occlusion region 5a is generated so that the pixel value of the partial region 10a over the partial region group 9g that is the boundary region of the region gradually changes.

Here, the arrow 12b indicates the shift direction when the second foreground image 3b is generated based on the correspondence 56, and the first region 6b and the first region 6b gradually change along the shift direction. An image of two areas 7b is generated. The shift direction can also be set by the operator from the operation unit 42.

According to the method according to FIG. 17, since the change in the pixel value at the boundary portion between the first region 6b and the second region 7b becomes smooth, a pseudo image with less discomfort can be generated.

In order to easily explain the correspondence relationship with respect to the shift direction, each of the partial region groups 9d to 9g in FIGS. 16 and 17 includes each boundary region including each of the partial region groups 9d to 9g. It is shown as an example of partial areas set discretely.

The second generation unit 25 generates an image of each occlusion area such as the occlusion area 5a or the first area 6a and the second area 7a by the method described above.

Even if the image at this stage is adopted as the final image of each occlusion area, it does not impair the usefulness of the present invention. However, the second generation unit 25 determines whether each occlusion area is in accordance with the set operation mode. For example, a smoothing process such as by using a smoothing filter such as a 3 × 3 pixel Gaussian filter may be performed on the above image.

Since the change of the pixel value is smoothed by performing the smoothing process on the image of each occlusion area, it is possible to further reduce the uncomfortable feeling about the pseudo image that the observer remembers.

The second generation unit 25 displays the pseudo image in which the image of each occlusion area is generated on the display unit 43 (step S100 in FIG. 19), and ends the pseudo image generation process.

As described above, according to the pseudo image generation device 200A, the occlusion area 5a on the pseudo image can be more accurately specified based on the distance information 52 of each subject based on the actual measurement, and the specified Since the image of the occlusion area 5a is generated based on the subject of interest and each background subject, a pseudo image with little discomfort can be generated.

Further, according to the pseudo image generation device 200A, the first area 6a corresponding to the subject of interest in the occlusion area 5a specified on the pseudo image and the second area 7a corresponding to each background subject are specified, and the first Since the image of the area 6a is generated based on the information of the subject of interest, and the image of the second area 7a is generated based on the information of each background object, the images of the first area 6a and the second area 7a are pseudo images. Can be made more similar to an actual image corresponding to, so that a pseudo image with less discomfort can be generated.

Further, according to the pseudo image generation device 200A, the shape information 62 expressing the entire three-dimensional shape of the subject of interest is acquired, and the first region 6a is specified based on the shape information 62, so that the subject of interest is identified. Since the first region 6a, which is a corresponding occlusion region, can be specified more accurately, a pseudo image with less discomfort can be generated.

Further, according to the pseudo image generation device 200A, the image of the first area 6a is generated based on the boundary area with the first area 6a in the second foreground image 3a, and the second area 7a in the second background image 4a By generating the image of the second region 7a based on the boundary region, the images of the first region 6a and the second region 7a can more closely resemble the actual image corresponding to the pseudo image, so that the user feels more uncomfortable. It is possible to generate a pseudo image with less.

<About modification:>
Although the embodiments of the present invention have been described above, the present invention is not limited to the above embodiments, and various modifications can be made.

For example, an image of the occlusion area may be generated based on a reference image that is taken in time sequence. Hereinafter, a method for generating an image of an occlusion area based on a time-series image will be described.

FIG. 18 is a diagram illustrating an example of a method for generating an image of an occlusion area based on a time-series image.

The reference images 1B to 1E shown in FIG. 18 are a series of time-series images of the subject of interest photographed in time sequence. The reference images 1B to 1E are displayed in the shooting order along the time axis t1.

The first foreground images 1b to 1e are images of the subject of interest in the reference images 1B to 1E, respectively, and the subject of interest moves relative to the camera.

The partial areas 11b to 11e are provided at the same position and range with respect to the reference images 1B to 1E. Note that the positions are not completely the same, and may be slightly shifted.

In this case, as shown in FIG. 18, the entire area of the partial area 11b and almost the entire area of the partial area 11c are respectively present in the background portions of the

first foreground images

1b and 1c, and the

partial areas

11d and 11e. Are all present within the

partial areas

1d and 1e, respectively.

Therefore, by obtaining the mode value or average value of the pixel values for each pixel group having the same coordinates in the partial areas 11b to 11e, a new image in which the pixel values of the image of the subject of interest and the background subject are mixed is obtained. Can be generated.

As described above, the occlusion region in the pseudo image generated from one reference image and taken from a different viewpoint from the reference image contains information on the subject and its background. By generating an image of the occlusion area based on a simple image, a pseudo image with little uncomfortable feeling can be generated.

Note that the partial areas 11b to 11e are not limited to a part of the reference images 1B to 1E, respectively, and may be all of the reference images 1B to 1E, for example.

The pseudo image generation system according to the modification will be described below.

The pseudo image generation system according to the modification includes the stereo camera 300 of the pseudo image generation system 100A according to the embodiment, and the pseudo image generation apparatus having the same configuration as the pseudo image generation apparatus 200A according to the embodiment. Yes.

As described above, the stereo camera 300 has a continuous shooting function for continuously shooting a subject in time sequence.

The stereo camera 300 according to the modified example uses the continuous shooting function to generate a plurality of standard images and a plurality of reference images for each subject, and supplies the generated images to the pseudo image generation device according to the modified example. To do.

The pseudo image generation device according to the modification is an embodiment except for the first acquisition unit and the second generation unit corresponding to the first acquisition unit 12 and the second generation unit 25 of the pseudo image generation device 200A according to the embodiment, respectively. Each of the functional units is the same as that of the pseudo image generation apparatus 200A according to FIG.

Here, the second acquisition unit and the second generation unit among the functional units of the pseudo image generation device according to the modification will be described, and description of the other functional units will be omitted.

○ About the 1st acquisition part concerning a modification:
The first acquisition unit according to the modification acquires a plurality of reference images and a plurality of reference images, which are a plurality of time-series images, for each subject photographed in time sequence by the stereo camera 300.

Next, the first acquisition unit supplies one reference image among the plurality of acquired reference images to the second acquisition unit 13, the correspondence relationship acquisition unit 15, the first generation unit 24, and the identification unit 26. At the same time, the acquired plurality of reference images are supplied to the second generation unit 25.

Furthermore, the first acquisition unit supplies one reference image taken at the same time as the first reference image to the second acquisition unit 13 among the plurality of acquired reference images.

Note that the reference image supplied to the second acquisition unit 13 may be a reference image in which each subject in the one standard image and each subject in the same state are captured. It is not limited to the reference image taken at the time.

That is, the second acquisition unit 13 supplied with the reference image acquires distance information based on actual measurement for at least each point of the subject of interest in the state in which the first reference image is acquired.

○ About the 2nd generation part concerning a modification:
For example, the second generation unit according to the modification may have a predetermined number of reference images supplied from the first generation unit, such as a region corresponding to the occlusion region of the pseudo image supplied from the first generation unit 24. An area is set, and an image used for generating an image of the occlusion area is generated by applying the method described with reference to FIG. 18 to the area, and the generated image is used. Generate an image of the occlusion area.

That is, the second generation unit according to the modification generates an image of the occlusion area in the pseudo image based on a plurality of reference images that are a plurality of time-series images.

When a plurality of time-series images are captured in a state where the angle of the subject with respect to the first viewpoint is changed, and a pseudo image is generated based on one of the plurality of time-series images, the pseudo image In some cases, an image corresponding to the occlusion area is already captured in another time-series image.

In that case, if an image corresponding to the occlusion area is searched by an image recognition process for a plurality of time-series images and used to generate an image of the occlusion area, it is based on the actually photographed subject image. Thus, an image of the occlusion area can be generated, so that an image of the occlusion area closer to the real object can be generated.

Also, due to the influence of noise or the like, one of a plurality of time series images may be significantly different from other time series images taken before and after that.

In such a case, if a pseudo image is generated based on another image other than the one image and an image of an occlusion area in the pseudo image is generated based on each other image, the occlusion is more accurately performed. It is possible to generate an image of an occlusion region that can specify the region and lessen feel uncomfortable with the pseudo image.

For example, when the subject is known in advance, an abnormal image affected by noise or the like may be extracted based on the continuity of the motion of the subject over a plurality of time-series images. It becomes easy.

In addition, the movement of the subject varies depending on, for example, whether the subject is a person or a car, but if the subject is known in advance, the subject characteristics are also expressed in a plurality of time series. By using it for the prediction of the movement of the subject over the image, it becomes easier to extract an abnormal image affected by noise or the like.

According to the pseudo image generation device according to the modified example described above, the range of the occlusion area in the pseudo image that is generated from one reference image out of a plurality of time-series images and is taken from a different viewpoint from the reference image. Since it is possible to specify more accurately based on the distance information of each subject photographed in one reference image, and to generate an image of the specified occlusion area based on a plurality of time-series images. The ratio of the occlusion area image to be an image including information on each of the subject of interest and the background subject is high, and the ratio at which a pseudo image with little discomfort can be generated can be increased.

Further, in the pseudo image generation system 100A that captures a set of the reference image 1A and the reference image 1R, and the above-described modification that captures a plurality of the reference images 1A and the reference image 1R that are a series of time-series images, the reference image By using not only the correspondence relationship based on the distance information 52 between 1A and the pseudo image, but also the correspondence relationship based on the distance information 52 between the reference image 1R and the pseudo image photographed in synchronization with the standard image 1A, each occlusion Even if the range of the region is specified and the image of each occlusion region using the standard image 1A and the reference image 1R is generated, the usefulness of the present invention is not impaired.

As described above, if the reference image 1R is also used, the range of each occlusion region can be specified more accurately and narrowly. For example, more appropriate information about each subject in the reference image 1A and the reference image 1R is used. Since an image of each occlusion area can be generated using, a pseudo image with less discomfort can be generated.

It should be noted that the same method can be adopted when the stereo camera is constituted by three or more cameras.

Also, by photographing the subject while moving the stereo camera relative to the subject, or by providing a plurality of sets of stereo cameras, a set of subjects from each position in a direction substantially perpendicular to the baseline length direction is provided. A stereo image or a series of time-series images may be taken, and a pseudo image may be generated using the various methods described above.

In other words, by shooting the subject image from each position in a direction substantially orthogonal to the baseline length direction, for example, when shooting the subject image from each position in a direction substantially parallel to the baseline length direction In general, since the occlusion area on the subject can be reduced, the occlusion area on the pseudo image can be specified as a narrower and more accurate range, and each subject photographed from multiple directions can be specified. The information on the subject can be acquired more accurately based on the image, and a pseudo image with less uncomfortable feeling can be generated.

100A, 100B pseudo image generation system 200A, 200B pseudo image generation apparatus 300

stereo camera

1A, 1B, 1C, 1D, 1E base image

1R reference image

2A, 2B, 3A, 3B, 4A, 4B pseudo image 5A distance image 6A pseudo image 31 Reference Camera 32 Reference Camera 49 Signal Line 51 Three-Dimensional Parameter 52 Distance Information 53 Identification Information 54 Imaging Parameter 55 Coordinate System Information 56 Correspondence 61 Model Group Shape Data 62

Shape Information

1a, 1b, 1c, 1d, 1e First Foreground Image 2a

First background image

3a, 3b

Second foreground image

4a, 4b

Second background image

5a,

5b Occlusion region

6a, 6b First region

6aA Shape region

7a, 7b Second region 8a,

8b Boundary

8c, 8d, 8e, 8f, 8g, 8h

Partial regions

9a, 9b, 9c, 9d, 9 e, 9f, 9g Partial area 10a

Partial area

11b, 11c, 11d, 11e Partial area t1 Time axis DL Data line

Claims

A pseudo image generation device,
First acquisition means for acquiring a reference image obtained by shooting each subject from a first viewpoint;
Second acquisition means for acquiring distance information based on actual measurement for at least each point of the subject of interest among the subjects;
Identifying means for identifying the subject of interest and each background subject photographed in a first background image that is a background portion of a first foreground image that is an image of the subject of interest of the reference image;
Correspondence relationship acquisition means for acquiring a correspondence relationship between the reference image and the pseudo image of each subject corresponding to shooting from a virtual viewpoint different from the first viewpoint based on each distance information;
A second foreground image that is an image of the subject of interest corresponding to shooting from the virtual viewpoint, based on the correspondence between at least the first foreground image of the reference image and the reference image; First generation means for generating the pseudo image containing a second background image that is an image of each background subject corresponding to shooting from the virtual viewpoint;
First identifying means for identifying an occlusion area that does not include the second foreground image and the second background image of the pseudo image;
Second generation means for generating an image of the occlusion area based on information of the subject of interest and each of the background subjects;
A pseudo image generation apparatus comprising:
The pseudo image generation device according to claim 1,
A second specifying means for specifying a first area corresponding to the subject of interest in the occlusion area and a second area corresponding to each of the background objects;
The second generation unit generates an image of the first area based on the information on the subject of interest, and generates a pseudo image that generates an image of the second area based on the information on each background subject. apparatus.
The pseudo image generation device according to claim 2,
Further comprising third acquisition means for acquiring shape information expressing the three-dimensional shape of the entire circumference of the subject of interest;
The pseudo image generating device, wherein the second specifying means specifies the first region based on the shape information.
The pseudo image generation device according to claim 2,
The second generation means generates an image of the first area based on a boundary area with the first area in the second foreground image, and also generates the second area in the second background image. A pseudo image generation device that generates an image of the second area based on a boundary area with the area.
The pseudo image generation device according to claim 4,
The second generation means includes
The first region is such that a pixel value of a region extending from a boundary region on the second region side in the first region to a boundary region on the first region side in the second region is gradually changed. A pseudo image generation device for generating images of the second region and the second region.
The pseudo image generation device according to claim 1,
The second generation means includes
(a) generating an image of a first boundary area with the second foreground image in the occlusion area based on a boundary area with the occlusion area in the second foreground image; and Generating a second boundary region image with the second background image based on a boundary region with the occlusion region in the second background image,
(b) A pseudo image generation device that generates an image of the occlusion area so that a pixel value of the occlusion area gradually changes from the first boundary area to the second boundary area.
A pseudo image generation device,
First acquisition means for acquiring a plurality of time-series images in which each subject is photographed in time sequence;
Second acquisition for acquiring each distance information based on actual measurement for at least each point of the subject of interest in the state where the reference image is acquired, using one image of the plurality of time-series images as a reference image Means,
Identifying means for identifying the subject of interest and each background subject photographed in a first background image that is a background portion of a first foreground image that is an image of the subject of interest of the reference image;
Correspondence to acquire a correspondence relationship between the reference image and the pseudo image of each subject corresponding to shooting from a virtual viewpoint different from the first viewpoint from which the reference image was shot based on each distance information Relationship acquisition means;
A second foreground image that is an image of the subject of interest corresponding to shooting from the virtual viewpoint, based on the correspondence between at least the first foreground image of the reference image and the reference image; First generation means for generating the pseudo image containing a second background image that is an image of each background subject corresponding to shooting from the virtual viewpoint;
First identifying means for identifying an occlusion area that does not include the second foreground image and the second background image of the pseudo image;
Second generation means for generating an image of the occlusion region based on the plurality of time-series images;
A pseudo image generation apparatus comprising:
The pseudo image generation device according to claim 1,
The pseudo image generation device, wherein the second generation means performs a smoothing process on the generated image of the occlusion area.
A pseudo image generation device,
First acquisition means for acquiring a reference image obtained by shooting each subject from a first viewpoint;
Second acquisition means for acquiring distance information based on actual measurement for at least each point of the subject of interest among the subjects;
Third acquisition means for acquiring shape information representing the three-dimensional shape of the entire circumference of the subject of interest;
Of the pseudo images of each subject corresponding to shooting from a virtual viewpoint different from the first viewpoint, an occlusion area in which a corresponding portion of the reference image is not shot, Based on the reference image, each distance image, and the shape information, the corresponding first region and the second region corresponding to each background subject photographed in the background portion of the image of the subject of interest. And generating the pseudo image by generating the image of the first area based on the information of the subject of interest and generating the image of the second area based on the information of each background subject. Generating means for
A pseudo image generation apparatus comprising:
The pseudo image generation device according to claim 9,
The generating means includes
(a) identifying means for identifying the subject of interest and each background subject photographed in a first background image that is a background portion of a first foreground image that is an image of the subject of interest of the reference image;
(b) correspondence relationship acquisition means for acquiring a correspondence relationship between the reference image and the pseudo image based on the distance information;
(c) a second foreground that is an image of the subject of interest corresponding to shooting from the virtual viewpoint based on at least the correspondence relationship of the first foreground image in the reference image and the reference image. First generation means for generating the pseudo image containing an image and a second background image that is an image of each background subject corresponding to shooting from the virtual viewpoint;
(d) identifying the first region based on the distance information and the shape information, and identifying the second region as the second foreground image and the first region of the pseudo image. Specifying means for specifying as an area that does not include any of the second background area;
(e) second generation means for generating an image of the first area based on the information of the subject of interest and generating an image of the second area based on the information of each background object;
A pseudo image generation apparatus comprising:
A pseudo image generation method,
Obtaining a reference image in which each subject is photographed from a first viewpoint;
Obtaining each distance information based on actual measurement for at least each point of the subject of interest among the subjects;
Identifying the subject of interest and each background subject photographed in a first background image that is a background portion of a first foreground image that is an image of the subject of interest of the reference image;
Obtaining a correspondence relationship between the reference image and the pseudo image of each subject corresponding to photographing from a virtual viewpoint different from the first viewpoint based on each distance information;
A second foreground image that is an image of the subject of interest corresponding to shooting from the virtual viewpoint, based on the correspondence between at least the first foreground image of the reference image and the reference image; Generating the pseudo image containing a second background image that is an image of each background subject corresponding to shooting from the virtual viewpoint;
Identifying an occlusion area that does not include the second foreground image and the second background image of the pseudo image;
Generating an image of the occlusion area based on respective information of the subject of interest and each of the background subjects;
A pseudo image generation method comprising:
A pseudo image generation method,
Acquiring a plurality of time-series images in which each subject is photographed in time sequence;
Obtaining each distance information based on actual measurement for at least each point of the subject of interest among the subjects in a state where the reference image is acquired, using one image of the plurality of time-series images as a reference image;
Identifying the subject of interest and each background subject photographed in a first background image that is a background portion of a first foreground image that is an image of the subject of interest of the reference image;
Obtaining a correspondence relationship between the reference image and the pseudo image of each subject corresponding to shooting from a virtual viewpoint different from the first viewpoint from which the reference image was shot based on the distance information. When,
A second foreground image that is an image of the subject of interest corresponding to shooting from the virtual viewpoint, based on the correspondence between at least the first foreground image of the reference image and the reference image; Generating the pseudo image containing a second background image that is an image of each background subject corresponding to shooting from the virtual viewpoint;
Identifying an occlusion area that does not include the second foreground image and the second background image of the pseudo image;
Generating an image of the occlusion region based on the plurality of time-series images;
A pseudo image generation method comprising:
A pseudo image generation method,
Obtaining a reference image in which each subject is photographed from a first viewpoint;
Obtaining each distance information based on actual measurement for at least each point of the subject of interest among the subjects;
Obtaining shape information representing the three-dimensional shape of the entire circumference of the subject of interest;
Of the pseudo images of each subject corresponding to shooting from a virtual viewpoint different from the first viewpoint, an occlusion area in which a corresponding portion of the reference image is not shot, Based on the reference image, each distance image, and the shape information, the corresponding first region and the second region corresponding to each background subject photographed in the background portion of the image of the subject of interest. And generating the pseudo image by generating the image of the first area based on the information of the subject of interest and generating the image of the second area based on the information of each background subject. And a process of
A pseudo image generation method comprising: