JP2000215311A - Method and device for generating virtual viewpoint image - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make generable a smooth virtual viewpoint image by a relatively simple processing by relieving the positional limitation for a virtual viewpoint capable of being generated in a virtual viewpoint image generating method generating an image seen from a viewpoint position where a camera is not actually placed. SOLUTION: Two or more depth maps are generated from plural images of different viewpoints (101 to 106), and a virtual viewpoint depth map seen from a virtual viewpoint is generated on the basis of the depth maps (107). The depth value of an area being invisible from a certain viewpoint is interpolated with information from another viewpoint, a pixel whose depth value is not decided is subjected to linear interpolation by using the depth values of ambient pixels and is further subjected to smoothing processing (108 and 109). Then, the pixels of a multi viewpoint image (real image) are rearranged on the basis of the virtual viewpoint depth map to reconfigure a virtual viewpoint image.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention receives a plurality of images having different viewpoint positions and depth information of a subject viewed from the viewpoint positions as inputs and views the images from an arbitrary viewpoint position where no camera is actually placed. The present invention relates to a virtual viewpoint image generation method and an apparatus for outputting an image.

[0002]

2. Description of the Related Art Conventionally, as a method of reconstructing an image at a viewpoint different from a position captured based on a real image, for example, "generation of arbitrary viewpoint video from multi-view video" (IEICE Technical Report,
IE96-121: 91-98, 1997.).
In this method, a depth map of an object is estimated from a multi-view image, the map is converted into a depth map of a virtual viewpoint, and a virtual viewpoint image is generated using the given multi-view image.

FIG. 17 is a diagram for explaining the concept of camera arrangement and virtual viewpoint image generation in a multi-lens camera system used in the conventional method. In FIG. 17, reference numeral 81 denotes a reference camera, reference numerals 82 to 85 denote reference cameras, and reference numeral 86 denotes a virtual viewpoint position (a viewpoint position and a line-of-sight direction of a generated virtual viewpoint image). In this method, a reference camera 82, 83,
While moving the matching window one pixel at a time along the epipolar line of each reference image captured at 84 and 85, the matching scale SSD (sum of squared-diff) is used.
erence). When the matching window is moved by d, SSD values are calculated from four directions. Of these, the smaller of the two values is added. Such processing is performed over the search range, and d at the minimum value is obtained as parallax. The parallax d and the depth z have the following relationship with the focal length f of the camera and the distance b between the cameras.

Z = bf / d Using this relationship, a depth map viewed from the camera position of the reference camera 81 is generated. Next, this depth map is converted into a depth map viewed from the virtual viewpoint position 86. For the area that can be observed from the reference camera 81, texture mapping is performed at the same time. For a region newly generated due to the movement of the viewpoint, a virtual viewpoint image is generated by linearly interpolating the depth value and mapping the texture of the reference image.

However, in this conventional method, since a corresponding point must be estimated for each pixel of the multi-view image to be used, the distance between the reference camera 81 and the reference cameras 82 to 85,
That is, the base line length is limited. Since the virtual viewpoint image is generated by texture mapping from the multi-view image,
The virtual viewpoint position at which a natural virtual viewpoint image is obtained is shown in FIG.
Within the range shown by the dotted line. Therefore, there is a problem that the range in which the virtual viewpoint can be placed is limited.

As another conventional technique, for example, "View
Generation for Three-Dimentional Scenes from Vide
o Sequence '' (IEEE Trans.Image Processing, vol.6 p
p.584-598, Apr. 1997). This involves acquiring information on the position and brightness of an object in a three-dimensional space based on a series of video images captured by a video camera, and geometrically mapping this information to the viewpoint of the image to be reconstructed. It is a method of transforming and projecting on a two-dimensional plane.

FIG. 18 is a view geometrically showing the photographing method of the conventional method. In FIG. 18, reference numeral 87 denotes a subject, 88 denotes a video camera, and 89 denotes a horizontal trajectory when shooting with a video camera. In this method, a video camera 88 is held in a hand, and the video camera 88 is moved along an orbit 89.
The information of the position and the brightness of the object in the three-dimensional space is acquired using the video sequence in which the subject 87 is imaged while moving.

FIG. 19 shows a positional relationship between individual video frames included in a video sequence shot by the method shown in FIG. In FIG. 19, reference numerals 91 to 95 denote video frames photographed by the video camera 88. FIG.
As shown in (1), each frame becomes a parallax image. By extracting corresponding points between these images, information on the position and luminance of the subject in the three-dimensional space is obtained.

In this method, a parallax image is obtained from a video sequence taken by moving a video camera.
There is a problem that it can be applied to stationary objects but not to moving objects. In addition, since this method is a method of generating a three-dimensional shape model based on the position information of the subject in the three-dimensional space and performing texture mapping on the model, the error of the position information causes the generated virtual viewpoint image to be generated. There is a problem that unnaturalness is conspicuous. further,
Since this method generates a three-dimensional shape model, there is a problem that the amount of data to be processed is large and the method is not suitable for real-time processing.

[0010]

SUMMARY OF THE INVENTION The present invention is to solve the above problems. SUMMARY OF THE INVENTION It is an object of the present invention to provide a virtual viewpoint image generation method capable of relaxing a restriction that a range of a viewpoint position of a virtual viewpoint image that can be generated is narrow and generating a natural and smooth virtual viewpoint image including a moving image by relatively simple processing. And to provide a device.

[0011]

An outline of typical means for achieving the above object of the present invention will be briefly described below.

(1) In a virtual viewpoint image generation method for generating an image viewed from an arbitrary viewpoint position, a plurality of image data each having a viewpoint at a different position are input, and based on the multi-view image data. And generating a depth map for each pixel of the multi-view image data from the viewpoint position of the multi-view image data to the subject, and based on any one depth map of the multi-view depth map, Generates a virtual viewpoint depth map viewed from the viewpoint where the camera is not placed, and the depth of the occlusion area hidden as a shadow of an object when viewed from any one viewpoint position of the multi-view depth map. The data can be interpolated with the data of another multi-view depth map viewed from another viewpoint position so that the occlusion area is not hidden, and interpolation can be performed using the above method. The depth data of the occlusion area of the virtual viewpoint depth map is interpolated by using the surrounding depth data, and the interpolated portion of the virtual viewpoint depth map in which the depth value distribution is not smooth is smoothed. By rearranging the pixels of the multi-viewpoint image data based on the depth data of the subject in the smoothed virtual viewpoint depth map and drawing the grayscale value,
In actuality, an image viewed from a viewpoint where a camera is not placed is generated.

(2) As a virtual viewpoint image generating apparatus for achieving the object of (1), a multi-view image data input means for inputting a plurality of image data each having a different position as a viewpoint, A multi-view depth map generating means for generating a depth map for retaining a depth value from a camera to a subject for each pixel of the multi-view image data based on the data; A virtual viewpoint depth map generating means for generating a depth map viewed from a viewpoint where the camera is not placed, a virtual viewpoint depth map correction processing means for performing interpolation processing on the virtual viewpoint depth map, and the virtual viewpoint depth map By actually rearranging the pixels of the multi-viewpoint image data based on the depth data of the subject and drawing the grayscale values, Virtual viewpoint image generation means for generating an image viewed from a viewpoint on which the object is not placed, wherein the virtual viewpoint depth map correction processing means includes an object when viewed from any one viewpoint position of the multi-viewpoint depth map. The depth data of the occlusion area hidden by the shadow of the image is interpolated by data of another multi-view depth map viewed from another viewpoint position so that the occlusion area is not hidden, and the data is viewed from the other viewpoint position. Virtual viewpoint depth map interpolation processing means for interpolating depth data of an occlusion area, which cannot be interpolated by means of multi-view depth map data, using surrounding depth data; Virtual viewpoint depth map smoothing processing means for performing smoothing processing on a portion where the value distribution is not smooth.

(3) In the virtual viewpoint image generating apparatus of (2), the multi-viewpoint depth map generating means extracts a corresponding point between the multi-viewpoint image data and estimates the depth by a stereo method. Features.

(4) In the virtual viewpoint image generating apparatus of (2), the multi-viewpoint depth map generating means estimates the depth by irradiating a laser beam or an image pattern.

(5) In the virtual viewpoint image generation device of (2), the multi-view image data input means combines multi-view image data at a plurality of viewpoint positions with one camera by combining a camera and a mirror. It is characterized in that it can be acquired.

That is, the present invention is characterized in that depth maps viewed from a plurality of viewpoints are simultaneously obtained, and these are integrated to generate a depth map viewed from a virtual viewpoint position.

A method of extracting a corresponding point between multi-view images and interpolating between the multi-view images as in the prior art is disclosed in the present invention. The difference is that a smooth virtual viewpoint image can be generated even if the position is set. The present invention is different from the method of capturing a parallax image while moving a video camera in that the present invention is applicable to a moving object and does not generate a clear three-dimensional shape model.

The operation of the present invention is as follows. In the present invention, since a plurality of depth maps are integrated to generate a single virtual viewpoint depth map, it is not necessary to estimate corresponding points between images that are not involved in the generation of the depth map. Therefore, a virtual viewpoint image can be generated even if corresponding points between all multi-viewpoint images are not extracted. Therefore, even when the camera interval of the multi-viewpoint image is far, a smooth virtual viewpoint image can be generated.

The present invention is a method of generating a virtual viewpoint depth map by interpolating depth data of an occlusion area that cannot be seen from one viewpoint with depth data viewed from another viewpoint. A dense virtual viewpoint depth map can be generated as compared with a method of generating a viewpoint depth map.

Also, the present invention is a method of generating a depth map once viewed from a virtual viewpoint and mapping multi-view image data to the virtual viewpoint depth map. In comparison, a smoother virtual viewpoint image can be generated from a lower-accuracy depth map.

Further, according to the present invention, since a three-dimensional shape model is not generated, the amount of data required for the virtual viewpoint image generation processing can be reduced.

[0023]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [First Embodiment] FIG. 1 is a diagram showing an outline of an overall configuration of a virtual viewpoint image generating apparatus according to a first embodiment of the present invention. In the figure, 10 is a subject, 21 to 24, 31 to 34 are cameras, 100 is a virtual viewpoint image generation unit, and 40 is an image display device.

As the depth map, a total of two depth maps are obtained from the multi-view images obtained by the multi-view cameras 21 to 24 and the multi-view images obtained by the multi-view cameras 31 to 34, respectively. . The depth map and the multi-view images acquired by the cameras 21 to 24 and 31 to 34 are input to the virtual viewpoint image generation unit 100, and the virtual viewpoint image generated by the virtual viewpoint image generation unit 100 is transmitted to the image display device 40. indicate.

Although FIG. 1 shows a case where one depth map is obtained from four cameras, one depth map may be obtained from any number of cameras.
Although FIG. 1 illustrates the case where a depth map viewed from two viewpoint positions is obtained, a depth map viewed from three or more viewpoint positions may be obtained.

The processing operation in the virtual viewpoint image generating apparatus configured as described above will be described with reference to FIG. FIG.
FIG. 3 is a block diagram for explaining a processing flow of the virtual viewpoint image generation device shown in FIG. 1. In the figure, camera device 2
Reference numerals 0 and 30 are multi-view camera devices arranged in a matrix of up, down, left, and right for capturing a multi-viewpoint image. The multi-view image input unit 101 inputs a multi-view image captured by the camera devices 20 and 30. Depth detector 102
Detects the depth data of the subject viewed from the respective viewpoint positions of the camera devices 20 and 30.

The viewpoint position designation unit 103 is a means for designating a viewpoint position of a virtual viewpoint image to be generated. The depth map input unit 104 inputs the depth data detected by the depth detection unit 102 as a depth map. The virtual viewpoint position input unit 105 includes the viewpoint position designation unit 10
The viewpoint position designated by 3 is input. The camera parameter data input unit 106 includes the multi-eye camera device 20,
Camera parameters such as the position of 30 and the focal length are input. The virtual viewpoint depth map generation unit 107 includes a depth map of the depth map input unit 104 and the virtual viewpoint position input unit 1.
A virtual viewpoint depth map is generated based on the virtual viewpoint position data 05 and the camera parameter data of the camera parameter data input unit 106.

The depth map interpolation processing unit 108 performs an interpolation process on a missing part in the virtual viewpoint depth map generated by the virtual viewpoint depth map generation unit 107. The depth map smoothing processing unit 109 includes a depth map interpolation processing unit 108
Performs a smoothing process on the depth map interpolated. The virtual viewpoint image reconstructing unit 110 includes a virtual viewpoint depth map processed by the depth map smoothing processing unit 109 based on the multi viewpoint image of the multi viewpoint image input unit 101 and virtual viewpoint position data of the virtual viewpoint position input unit 105. Then, the virtual viewpoint image is reconstructed based on the camera parameter data of the camera parameter data input unit 106. The image display unit 111 displays the virtual viewpoint image formed by the virtual viewpoint image reconstruction unit 110. The display screen 41 is a device for displaying a virtual viewpoint image.

As the depth detection unit 102, for example, a device that extracts corresponding points of a multi-view camera image and estimates the depth by a stereo method is used. The operation of the depth detection unit 102
This will be described in detail with reference to FIG. FIG. 3 is a diagram for explaining the concept of camera arrangement and projection of the multi-lens camera system.

The reference camera 51 is placed at the origin, and the
The four reference cameras 52 to 55 are placed at a fixed distance L. You
The optical axes of all cameras are parallel. Also, all turtles
The same specifications are used for the camera.
And a geometric configuration as shown in FIG.
You. In the arrangement of FIG. 3, a point P = (X, Y,
Z) is a reference image located at a distance of the focal length f from the XY plane.
Upper point p₀= (U₀, V ₀). Where u
₀= FX / Z, v₀= FY / Z. Point P is also
Reference camera C_iThe point p on the image of (i = 1, 2, 3, 4)
_i= (U_i, V_i) Is also projected. Where u_i= F (X-D_{i, x}) / Z v_i= F (Y-D_{i, y}) / Z where baseline length vector D_i= (D_{i, x}, D_{i, y})
Each D₁= (L, 0), D_Two= (-L, 0), D_Three=
(0, L), D_Four= (0, -L). All references
A configuration in which the baseline lengths of the cameras 52 to 55 and the reference camera 51 are equal.
Under the condition, the true parallax d of the point P_iFor all i
D_i= FL / Z = | p_i-P₀|, The depth is estimated by estimating the parallax.
Can be obtained. At least 2 units to obtain depth from parallax
It is possible if there is a camera of
The use of a la structure allows for true occlusion.
When it is difficult to estimate the parallax of the object and the depth cannot be determined
Can be avoided.

Next, the virtual viewpoint depth map generation unit 107
Will be described in more detail. The depth map holds the value of the distance from the camera to the subject for each pixel in an image captured from a certain viewpoint position. In other words, while a normal image has brightness and chromaticity corresponding to each pixel on the image plane, a depth map has a depth value corresponding to each pixel on the image plane. is there.

In the virtual viewpoint depth map generation unit 107,
On the basis of the depth map held in the depth map input unit 104, the virtual viewpoint position data of the virtual viewpoint position input unit 105, and the camera parameter data of the camera parameter data input unit 106, a depth map viewed from a virtual viewpoint is generated. It is assumed that the depth map held in the depth map input unit 104 has subject depth information on each pixel of the image viewed from each viewpoint position where the multi-viewpoint image is captured.

First, based on the virtual viewpoint position data of the virtual viewpoint position input unit 105 and the camera parameter data of the camera parameter data input unit 106, a depth map viewed from the viewpoint position closest to the virtual viewpoint position is obtained. 2 or more are selected from the depth maps stored in.

FIG. 4 shows the camera coordinate system of the viewpoint from which the real image was captured, the virtual viewpoint, and the coordinate system of the projected image plane. A camera coordinate system that captures an arbitrary one of the selected depth maps is (X ₁ , Y ₁ , Z ₁ ) ^T , and a camera coordinate system of the virtual viewpoint position is (X ₂ , Y ₂ , Z ₂ ) ^T. I do. When the depth arbitrary point _{p i = (u 1, v} 1) on the map point of the projected three-dimensional space in _{P = (X 1, Y 1} , Z 1) T Z 1 of is sought, X, Y of point P viewed from the coordinate system of the real viewpoint
The coordinates are given by X ₁ = Z ₁ u ₁ / f (Equation 1) and Y ₁ = Z′v ₁ / f (Equation 2). Here, f is the focal length of the camera.

Now, two coordinate systems (X ₁ , Y ₁ , Z ₁ ) ^T
And (X ₂ , Y ₂ , Z ₂ ) ^T form a rotation matrix R ₂₁ =
[R _ij ] ∈R ^3x3 and the translation sequence T ₂₁ = (Δx, Δy, Δ
z) Using ^T , it can be expressed by the relationship of (X ₂ , Y ₂ , Z ₂ ) ^T = R ₂₁ (X ₁ , Y ₁ , Z ₁ ) ^T + T ₂₁ (Equation 3). The depth value Z ₂ obtained from (Equation 3) is the depth value of the point P viewed in the virtual viewpoint coordinate system (X ₂ , Y ₂ , Z ₂ ) ^T. Point P = (X ₂ , Y ₂ , Z ₂ )
^T is a point on the virtual viewpoint depth map, p ₂ = (u ₂ ,
v ₂ ). (U ₂ , v ₂ ) is obtained by the following equation using X ₂ and Y ₂ obtained by (Equation 3).

U ₂ = fX ₂ / Z ₂ (Equation 4) v ₂ = fY ₂ / Z ₂ (Equation 5) Therefore, the point p ₂ = on the virtual viewpoint depth map
The depth value of (u ₂ , v ₂ ) can be determined as Z ₂ .

The above processing is repeated for all the points (u ₁ , v ₁ ) in the depth map, and the depth value held by the selected depth map is changed to the pixel value in the depth map viewed from the virtual viewpoint. Convert to depth value. By performing the same processing for all the selected depth maps, a dense virtual viewpoint depth map with a small depth value blank is created.

Next, the depth map interpolation processing unit 108 and the depth map smoothing processing unit 109 will be described with reference to FIG. 5B to 5E show an image obtained by photographing the sphere shown in FIG. 5A cut along a scanning line AB, and the depth value on the scanning line is represented on the vertical axis.

In the depth map interpolation processing unit 108, the pixels 6 having no depth value in the virtual viewpoint depth map shown in (B) generated by the virtual viewpoint depth map generation unit 107 because parallax could not be estimated due to occlusion.
Linear interpolation processing is performed on the assumption that the depth does not change abruptly in a local area, using the depth value of 1 as the depth value of the surrounding pixels 62 whose depth value is known. As a result, all pixels have depth values (C)
A virtual viewpoint depth map shown in FIG.

The depth map smoothing unit 109 performs a smoothing process on the depth value of the virtual viewpoint depth map shown in FIG.
First, the depth value of the pixel 63 whose depth is rapidly changing on the scanning line of the virtual viewpoint depth map is removed, and the depth does not change abruptly in the local region using the depth value of the surrounding pixels 64. Under this assumption, a linear interpolation process is performed to generate a virtual viewpoint depth map as shown in FIG. Further, in order to approximate the surface of the subject with a smooth curved surface, a smoothing process is performed on the entire virtual viewpoint depth map to obtain a virtual viewpoint depth map shown in FIG.

For the smoothing process, for example, a general second-order differential filter can be used. Such a smoothing process for approximating the surface of a subject with a smooth curved surface has an effect of suppressing a decrease in image quality when mapping a texture to a virtual viewpoint image.

Next, the virtual viewpoint image reconstruction unit 110 will be described with reference to FIG. Virtual viewpoint image reconstruction unit 11
0 maps the texture of the multi-viewpoint image input by the multi-viewpoint image input unit 101 based on the camera parameter data of the camera parameter data input unit 106 and the data of the virtual viewpoint depth map of the depth map smoothing processing unit 109. Then, a virtual viewpoint image viewed from the virtual viewpoint position of the virtual viewpoint position input unit 105 is generated.

The coordinate transformation used in the virtual viewpoint image reconstruction unit 110 corresponds to the inverse transformation of that used in the virtual viewpoint depth map generation unit 107. However, since the depth values held by the virtual viewpoint depth map have changed due to the processing performed by the depth map interpolation processing unit 108 and the depth map smoothing processing unit 109, it is necessary to perform the coordinate conversion again using the new depth values. is there.

Here, the coordinate system of the virtual viewpoint depth map is (X ₂ , Y ₂ , Z ₂ ) ^T , and any one coordinate system in the multi-view image is (X ₃ , Y ₃ , Z ₃ ). ^T. When the depth of a pixel at an arbitrary point p ₂ = (u ₂ , v ₂ ) in the virtual viewpoint depth map is Z ₂ , this pixel p ₂ = (u ₂ ,
v ₂ ), a point P = (X
₂ , Y ₂ , Z ₂ ) The coordinates of ^T are given by X ₂ = Z ₂ u ₂ / f (Equation 6) Y ₂ = Z ₂ V ₂ / f (Equation 7) Here, f is the focal length of the camera.

Now, two coordinate systems (X ₂ , Y ₂ , Z ₂ ) ^T
And (X ₃ , Y ₃ , Z ₃ ) ^T form a rotation matrix R ₃₂ =
[R _ij ] ∈R ^3x3 and the translation sequence T ₃₂ = (Δx, Δy, Δ
z) Using ^T , it can be expressed by the relationship of (X ₃ , Y ₃ , Z ₃ ) ^T = R ₃₂ (X ₂ , Y ₂ , Z ₂ ) ^T + T ₃₂ (Equation 8). (Equation 6) and (Equation 7) are replaced by (Equation 8)
, The point P = (X ₃ ) in the three-dimensional space of the subject projected on the point (u ₂ , v ₂ ) in the virtual viewpoint image viewed in the (X ₃ , Y ₃ , Z ₃ ) ^T system. , Y ₃ , Z ₃ ) ^T is calculated. The point P is projected on a point p ₃ = (u ₃ , v ₃ ) on the real image. (U ₃ , v ₃ ) is the X obtained by (Equation 8)
₃ and Y _3, and can be calculated by the following equation.

U ₃ = fX ₃ ′ / Z ₃ (Equation 9) v ₃ = fY ₃ ′ / Z ₃ (Equation 10) The points in the multi-viewpoint image calculated by these (Equation 9) and (Equation 10) The luminance value and the chromaticity value of the pixel of u ₃ , v ₃ ) are drawn at the point (u ₂ , v ₂ ) in the virtual viewpoint image. By repeating this process for all points in the multi-view image,
A virtual viewpoint image viewed from the viewpoint position of the virtual viewpoint position input unit 105 is generated.

[Second Embodiment] Next, a second embodiment of the present invention will be described.
The embodiment will be described with reference to FIG. FIG.
FIG. 7 is a diagram illustrating an outline of an overall configuration of a virtual viewpoint image generation device according to a second embodiment. In FIG.
31 is a depth detector, 220 and 230 are cameras, 10 is a subject, 200 is a virtual viewpoint image generation unit, and 240 is an image display device.

The depth detectors 221 and 231 are devices that irradiate a laser beam or an image pattern to actively acquire the depth of the subject. For example, a laser range finder can be used as such a depth detector. The depth map acquired by the depth detectors 221 and 231, and the multi-viewpoint image acquired by the cameras 220 and 230 are input to the virtual viewpoint image generation unit 200, and the virtual viewpoint image generated by the virtual viewpoint image generation unit 200 is input to the virtual viewpoint image generation unit 200. , Are displayed on the image display device 240.

In FIG. 7, the depth detectors 221 and 231 and the cameras 220 and 230 have been described as separate devices.
A device that can simultaneously acquire a depth map and an image with one device may be used. In addition, the depth detector 22
Any device capable of detecting depth information for each pixel of an image captured by the cameras 220 and 230 can be used as the devices 1 and 231. Although FIG. 7 illustrates the case where depth maps viewed from two viewpoint positions are obtained, depth maps viewed from three or more viewpoint positions may be obtained.

[Third Embodiment] Next, a third embodiment of the present invention will be described.
The embodiment will be described with reference to FIG. FIG.
FIG. 1 is a diagram showing a configuration of a camera device used in the present embodiment.

In the present embodiment, by combining a camera, a liquid crystal shutter, a deflecting beam splitter, and a mirror, one camera acquires multi-viewpoint image data viewed from a plurality of viewpoint positions. 8, 301 is a camera, 302 is a liquid crystal shutter (TN), 303 is a deflecting beam splitter (PBS), 304 is a mirror, 305 is a viewpoint position of multi-view image data, and a position where no camera is actually placed. Is shown. The liquid crystal shutter 302 passes only P-polarized light when a voltage is applied, and passes only S-polarized light when no voltage is applied.
The deflection beam splitter 303 transmits the P-polarized light,
It is an element that reflects polarized light.

First, using the camera 301, the camera 30
The case where image data viewed from the position 1 is acquired will be described. A voltage is applied to the liquid crystal shutter 302 so that only the P-polarized light passes therethrough. Then, only the P-polarized light, which is the transmitted component of the deflection beam splitter 303, is
And as a result, image data viewed from the position of the camera 301 can be obtained.

Next, using the camera 301, the viewpoint position 3
A case where the image data viewed from 05 is acquired will be described. The voltage of the liquid crystal shutter 302 is turned off so that only S-polarized light passes. Then, the deflection beam splitter 303
Only the S-polarized light, which is the reflection component of, enters the camera 301. Since the S-polarized light entering the camera 301 is reflected by the mirror 304, image data viewed from the viewpoint position 305 can be obtained.

[0054]

Next, a specific embodiment of the present invention will be described with reference to FIG. In this embodiment, as shown in FIG. 9, depth images A and B viewed from two viewpoints whose three-dimensional positions are known are used to convert an occlusion area invisible from one viewpoint into information from the other viewpoint. To generate a virtual viewpoint depth image viewed from the virtual viewpoint. Where 2
It is not necessary for the depth images A and B to have a correspondence between individual pixels.

This virtual viewpoint depth image has a point where the depth value is not determined in a portion where the influence of noise or the depth changes rapidly. These points are linearly interpolated with surrounding pixels. Further, the virtual viewpoint depth image is smoothed using a second derivative filter, and the real image is texture-mapped to the processed virtual viewpoint depth image to generate a virtual viewpoint image. At that time, the real image closer to the viewpoint is prioritized.

The feature of this method is that a depth image viewed from a virtual viewpoint is generated based on two depth images, and processing is performed on the depth image to texture-map an actual image, thereby obtaining the accuracy of the original depth image. However, even if the image quality is bad or there is a lot of noise, a smooth virtual viewpoint image can be generated.

FIGS. 10 to 16 are diagrams showing images used in this embodiment. 10 and 11 are real images.
The depth image viewed from the viewpoint of these two real images is shown in FIG.
And FIG. The depth image was calculated using a multi-baseline stereo method based on five multi-view images captured by a camera moving in the x-axis direction at 1 cm intervals. In these depth image diagrams, the depth value is actually represented by a change in the density value. However, in this figure, the outline of the portion where the depth value (density value) greatly changes is shown for convenience of notation in the drawing. I have. A virtual viewpoint depth image generated by using the method of the present invention from these two depth images is the image shown in FIG. FIG. 15 shows the virtual viewpoint depth image shown in FIG.
12 is a virtual viewpoint image obtained by texture mapping the real images of FIGS. 10 and 11.

FIG. 16 is a diagram showing a comparison between the method according to the present invention and the conventional method. As shown in FIG.
In the virtual viewpoint image according to the method of the present invention, noise at the face outline portion where the depth changes rapidly is reduced.
6 (B), a clearer image is obtained as compared with the virtual viewpoint image according to the conventional method. In addition, since the virtual viewpoint depth image is subjected to the smoothing process, a decrease in the resolution of the virtual viewpoint image during texture mapping is suppressed.

[0059]

As described above, according to the present invention,
By reconstructing the image viewed from the observer based on a given viewpoint position and multiple images and its depth information, the image corresponding to the viewpoint position is output smoothly when the viewpoint position is moved. can do. In this method, a virtual viewpoint image is generated by reconstructing an image based on depth information, so that a smooth virtual viewpoint image is generated even when the distance between the positions at which the original image is captured is long. There are advantages that can be done. The storage capacity required for the device is extremely small as compared with the case where all images corresponding to conceivable viewpoint movements are held. The processing speed can be increased as compared with a method of mapping a texture on a three-dimensional shape model. Since this is a method of generating a virtual viewpoint image based on a smoothly corrected virtual viewpoint depth map, it is possible to generate a smooth virtual viewpoint image.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating an outline of an overall configuration of a virtual viewpoint image generation device according to a first embodiment of the present invention.

FIG. 2 is a block diagram for explaining a processing flow of the virtual viewpoint image generation device according to the first embodiment.

FIG. 3 is a diagram for explaining the concept of camera arrangement and projection of the multi-lens camera system.

FIG. 4 is a diagram for explaining coordinate conversion used in a virtual viewpoint depth map generation unit.

FIG. 5 is a diagram for explaining processing of a depth map interpolation processing unit and a depth map smoothing processing unit;

FIG. 6 is a diagram for describing coordinate conversion used in a virtual viewpoint image reconstruction unit.

FIG. 7 is a diagram illustrating an outline of an overall configuration of a virtual viewpoint image generation device according to a second embodiment of the present invention.

FIG. 8 is a diagram illustrating a configuration of a camera device used in a third embodiment of the present invention.

FIG. 9 is a diagram for explaining an outline of an embodiment of the present invention.

FIG. 10 is a diagram illustrating an example of a real image.

FIG. 11 is a diagram illustrating an example of a real image.

FIG. 12 is a diagram illustrating an example of a depth image corresponding to the real image in FIG. 10;

FIG. 13 is a diagram illustrating an example of a depth image corresponding to the real image in FIG. 11;

FIG. 14 is a diagram illustrating an example of a virtual viewpoint depth image.

FIG. 15 is a diagram illustrating an example of a generated virtual viewpoint image.

FIG. 16 is a diagram showing a comparison between a virtual viewpoint image according to the method of the present invention and a virtual viewpoint image according to a conventional method.

FIG. 17 is a diagram for explaining a conventional technique.

FIG. 18 is a diagram for explaining a conventional technique.

FIG. 19 is a diagram for explaining a conventional technique.

[Explanation of symbols]

 Reference Signs List 10 subject 20, 30 camera device 21 to 24 camera 31 to 34 camera 40 image display device 41 display screen 100 virtual viewpoint image generation unit 101 multi viewpoint image input unit 102 depth detection unit 103 viewpoint position specification unit 104 depth map input unit 105 virtual Viewpoint position input unit 106 camera parameter data input unit 107 virtual viewpoint depth map generation unit 108 depth map interpolation processing unit 109 depth map smoothing processing unit 110 virtual viewpoint image reconstruction unit 111 image display unit

 ──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Kenji Nakazawa 3-19-2 Nishi-Shinjuku, Shinjuku-ku, Tokyo Inside Japan Telegraph and Telephone Co., Ltd. No. 2 Nippon Telegraph and Telephone Corporation F-term (reference) 5B050 BA04 BA09 EA15 EA27 5B057 BA02 BA11 CA01 CA08 CA13 CA16 CB01 CB08 CB13 CB16 CE05 DB03 DB06 DB09 5C061 AA06 AA20 AA21 AB04 AB08

Claims (6)

[Claims] 1. A virtual viewpoint image generating method for generating an image corresponding to a viewpoint position, comprising the steps of: inputting a plurality of image data each having a viewpoint at a different position; Based on a process of generating a depth map that holds a depth value from the viewpoint position of the multi-view image data to a subject for each pixel of the multi-view image data, and any one depth map of the multi-view depth map, A process of generating a virtual viewpoint depth map viewed from a viewpoint where a camera is not actually placed by performing coordinate transformation of a viewpoint position in a three-dimensional space, and the multi-viewpoint depth map for the virtual viewpoint depth map. Any one of
The depth value of an occlusion area that is hidden as an object shadow when viewed from one viewpoint is used, and data of another multi-view depth map that is viewed from another viewpoint that does not hide the occlusion area is used. A virtual viewpoint depth map obtained by interpolating using the processed virtual viewpoint depth map or a known depth value around the virtual viewpoint depth map when there is a part that cannot be interpolated in the virtual viewpoint depth map, or Furthermore, the pixels of the multi-viewpoint image data are rearranged by three-dimensional coordinate conversion of the viewpoint position based on the depth data of the subject in the virtual viewpoint depth map obtained by smoothing the depth value of the virtual viewpoint depth map, and the brightness thereof is changed. And the process of generating an image viewed from the viewpoint where the camera is not actually placed by drawing Virtual viewpoint image generation method characterized by comprising. 2. A virtual viewpoint image generating apparatus for generating an image corresponding to a viewpoint position, comprising: multi-view image data input means for inputting a plurality of image data having different positions as viewpoints; A multi-view depth map generating means for generating a depth map for holding a depth value from a camera to a subject for each pixel of the multi-view image data, and an arbitrary depth map of the multi-view depth map Virtual viewpoint depth map generating means for generating a virtual viewpoint depth map as viewed from a viewpoint where a camera is not actually placed by performing coordinate transformation of a viewpoint position in a three-dimensional space based on Regarding the viewpoint depth map, any one of the multi viewpoint depth maps
The depth value of an occlusion area that is hidden as a shadow of an object when viewed from one viewpoint is used, and data of another multi-view depth map that is viewed from another viewpoint that does not occlude the occlusion area is used. A virtual viewpoint depth map correction processing means for performing interpolation processing, and rearranging the pixels of the multi-viewpoint image data based on the depth value of the subject included in the virtual viewpoint depth map to draw the luminance and chromaticity thereof. A virtual viewpoint image generation device, comprising: virtual viewpoint image generation means for generating an image viewed from a viewpoint where a camera is not actually placed. 3. The virtual viewpoint image generating apparatus according to claim 2, wherein the virtual viewpoint depth map correction processing means interpolates a depth value of an occlusion area that cannot be interpolated by the interpolation processing using a depth value around the occlusion area. A virtual viewpoint image generating apparatus, comprising: means for processing; and / or means for performing smoothing processing on a portion where the distribution of depth values of the interpolated virtual viewpoint depth map is not smooth. 4. A virtual viewpoint image generating apparatus according to claim 2, wherein said multi-viewpoint depth map generating means extracts a corresponding point between said multi-viewpoint image data and performs triangulation by a stereo method. A virtual viewpoint image generation apparatus characterized in that a depth map is generated by passively estimating a depth by using a depth map. 5. The virtual viewpoint image generation device according to claim 2, wherein the multi-viewpoint depth map generation means actively estimates the depth by irradiating an image pattern with a laser beam to a subject. A virtual viewpoint image generation apparatus, wherein a depth map is generated by the method. 6. The virtual viewpoint image generating apparatus according to claim 2, wherein said multi-viewpoint image data inputting means includes a light-emitting device on a light path corresponding to a plurality of viewpoint positions. A plurality of optical means for partially or entirely reflecting or transmitting the light, and one camera being disposed on one of the optical paths through which the light reflected or transmitted by the optical means passes; A virtual viewpoint image generation apparatus, wherein the one camera acquires image data at a plurality of viewpoint positions.