JP5358137B2 - Image generating apparatus, method and program - Google Patents

Description

  The present invention relates to a technology for constructing a light space for realizing a free viewpoint image.

  Various proposals have been made regarding free viewpoint video technology for reproducing video viewed from an arbitrary viewpoint position based on still images or moving images taken by a plurality of cameras (for example, Patent Document 1 and Non-Patent Document 1). And Non-Patent Document 2, for example).

  In the prior art method, images taken at approximately the same time by a plurality of cameras are arranged in the light space, and an image viewed from a position where no camera is present is generated by interpolation processing from images actually taken around. Thus, the video viewed from an arbitrary viewpoint position is reproduced.

JP 2008-15756 A Takeshi Naemura, et al. "Ray-Based Creation of Photo-Realistic Virtual World", VSMM97, pp. 59-68, September 1997 Michael Drose, et al. “Ray-Space Interpolation based on Filtering in Disparity Domain”, Proc. of 3D Image Conference 2004, pp. 29-30, 2004 C. Nicolas, et al. "A CMOS 3D Camera with Millimetric Depth Resolution", IEEE Custom Integrated Circuits Conference, pp. 705-708, 2004

  When constructing a dense ray space by generating images viewed from positions between actually captured images by interpolation processing, each pixel value in the generated image is generated based on the corresponding pixels of the actually captured image It is important to do. If the accuracy of the corresponding pixel determination is low, the quality of the image generated by the interpolation process is degraded, and thus the quality of the entire free viewpoint video system is degraded. On the other hand, generation of an image by interpolation processing has a high processing load, and it is required to search for a corresponding pixel as easily as possible.

  Therefore, the present invention provides an image generation apparatus that easily and accurately finds corresponding pixels in an actually captured image and constructs an image at a position where no camera actually exists in order to construct a ray space. An object of the present invention is to provide a method and a program for causing a computer to function as the image generation apparatus.

According to the image generation apparatus of the present invention,
An image generation device that determines a pixel value of a third pixel of a third image at a specified viewpoint based on a first image and a second image, wherein each pixel of the first and second images A first image corresponding to the third pixel, based on position information in the light space and real space, a distribution in the light space of points representing light rays from one point in the real space, and the designated viewpoint; The first pixel, the intersection detection means for obtaining the second pixel of the second image, the first search area that is a predetermined range centered on the first pixel, and the second pixel In the second search region that is the same predetermined range, block matching by a block having a predetermined pixel size is applied, and the first block in the first search region and the second block that are the most correlated combination are applied. Corresponding point detecting means for obtaining a second block in the search area; The pixel at the center of one of the blocks, based on the pixel in the center of the second block, and a pixel value calculating means for determining a pixel value of the third pixel, the first search area and the second The range of the search area is reduced as it approaches the position of the third pixel in the light space, and increases as it moves away .

According to another embodiment of the image generation apparatus of the present invention,
Based on the positions and directions of a plurality of cameras and the captured images captured by the plurality of cameras, pre-processing for calibrating camera parameters of the plurality of cameras and determining positional information of each pixel of the captured images in real space Means, a captured image, and storage means for storing a generated image generated from the captured image, wherein the first image and the second image are the captured image or the generated image, respectively. Preferably, each pixel of the generated image is determined by the intersection detection unit, the corresponding point detection unit, and the pixel value calculation unit.

According to another embodiment of the image generation apparatus of the present invention,
It is also preferable that the viewpoints of the first image and the second image are on both sides of a straight line extending in the direction of the viewpoint at the designated viewpoint of the third image.

According to the image generation method of the present invention,
A method for determining a pixel value of a third pixel of a third image at a designated viewpoint in an image generation device based on a first image and a second image, wherein the intersection detection unit includes: Corresponding to the third pixel based on the positional information of each pixel of the second image in the light space and the real space, the distribution of the positions of the real space in the light space, and the designated viewpoint. The step of obtaining the first pixel of the first image and the second pixel of the second image, and the corresponding point detection unit obtains a rectangular first search region that is a range centered on the first pixel. A step of setting, in the first image, a rectangular second search area that is a range centered on the second pixel in the first image, and a corresponding point detection unit within the first search area. And a block of the predetermined pixel size in the second search area. Determining the second block in the second search area corresponding to the first block in the first search area by determining the correlation of And determining the pixel value of the third pixel based on the pixel at the center of the second block, and the range of the first search area and the second search area is defined as a ray space. It is characterized in that it becomes smaller as it gets closer to the position of the third pixel in, and becomes larger as it goes away .

  A program according to the present invention causes a computer to function as the image generation apparatus.

  Each ray from a point in the real space is distributed in a curve or a straight line in the ray space, and this distribution can be obtained based on the camera parameters, the camera position and direction, and the position of each pixel in the real space. . However, each value used for calculating the distribution usually includes an error, and the corresponding pixels of the first and second images cannot be accurately obtained by this distribution. However, as long as the error is not significantly large, the corresponding pixel exists in the vicinity of the pixel obtained from the distribution. Therefore, the corresponding pixel is determined by applying block matching to the search region centered on the obtained pixel. By limiting the search area to which the block matching is applied according to the distribution to a narrow range, it is possible to easily find the corresponding pixel.

First, an outline of an image generation apparatus according to the present invention will be described. As shown in FIG. 4, a case is considered where the cameras 11 to 18 are arranged on the circumference so as to be directed toward the center. The direction is expressed as an angle θ in the counterclockwise direction from the Z axis with the Z axis positive direction as a reference. Each camera receives light rays traveling in the direction determined by the camera position and direction and camera parameters, and acquires image information. As described in Non-Patent Document 1, for example, in a certain horizontal plane, each ray acquired by each camera is represented by its direction and the position of the intersection of the P-axis and the ray orthogonal to the direction. To do. In this case, as shown in FIG. 4 (a), light 90 is one of light rays camera 18 is acquired, since a direction theta _0, is intended to intersect at a position p ₀ and P axis, FIG. 4 As shown in (b), it is represented by a point (p ₀ , θ ₀ ) on the P-θ plane.

  As described above, the light beam that causes each pixel of the image captured by each camera can be calculated from the camera position and direction, and the camera parameters. Therefore, there is a correspondence between each light beam and the pixel. Therefore, among the moving images captured by the cameras, the pixel values of the pixels in the images captured by the cameras at almost the same moment are represented by light rays corresponding to the pixels on the P-θ plane, as shown in FIG. A ray space is constructed by recording in the position. In FIG. 4B, reference numerals 11-1 to 18-1 are images taken by the cameras 11 to 18, respectively.

  However, in the ray space shown in FIG. 4B, the image data exists only at positions corresponding to the rays actually acquired by the camera, and does not exist at other positions. The image generation apparatus according to the present invention generates image data corresponding to other light rays by interpolation processing from actually captured image data.

  FIG. 1 is a block diagram of an image generation apparatus according to the present invention. As shown in FIG. 1, the image generation apparatus includes a storage unit 2, a preprocessing unit 3, and a light space construction unit 4. The storage unit 2 holds, for example, still images or moving images captured by a plurality of cameras 1 in association with information about the cameras that captured the images. The information about the camera includes a camera identifier, a camera position, and camera parameters. In the following embodiment, as shown in FIG. 4, the camera 1 has a circumferential shape and is arranged so that the optical axis faces the center at the same height, that is, the direction of each camera. Can be calculated from the camera position, and the camera parameters of each camera are the same.

  The preprocessing unit 3 obtains from the storage unit 2 images at approximately the same time (hereinafter referred to as “captured images”) captured by the respective cameras, the positions of the respective cameras, and camera parameters. • Adjust the camera parameters by performing calibration. Further, each pixel of the photographed image is determined to be a foreground or a background, each subject in the photographed image, that is, each object is modeled, and its position is determined by parallax information or the like. To generate model information for specifying the position of each pixel. In addition, for each of these processes, known methods for obtaining depth information can be used.

  The ray space constructing unit 4 generates an image viewed from this viewpoint (hereinafter referred to as a generated image) based on information indicating the viewpoint designated from the outside or set in advance in the image generation apparatus. More specifically, at this viewpoint position, when a camera with the direction of the viewpoint is placed, an image at each time that the camera will shoot is generated from images taken at almost the same time taken by the actual camera. To do. Thereafter, information indicating the viewpoint for the generated image is stored in the storage unit 2. The position of the viewpoint is the same height on the same circumference as the actual camera, the direction of the viewpoint is the same as the actual camera, and the image range viewed from the viewpoint is the same as the actual camera. In other words, the camera that is assumed to capture the generated image has the same camera parameters as the actual camera.

  As described above, since the position of the light space of each pixel can be calculated from the camera position, direction, and camera parameters, the storage unit 2 captures each captured image captured by each camera and each generated image generated by interpolation processing. Therefore, it is possible to construct a denser light space by arranging the pixel values of the photographed image and the generated image at the corresponding light space positions. . Note that the number of generated images generated by the interpolation process, that is, the number of virtual camera positions is determined by the resolution of the viewpoint position required for the system.

  FIG. 2 is a block diagram of the light space construction unit 4. According to FIG. 2, the ray sky construction unit 4 includes an intersection detection unit 41, a corresponding point detection unit 42, and a pixel value calculation unit 43. FIG. 3 is a diagram for explaining processing in the light space construction unit 4. In FIG. 3, reference numerals 50 and 60 are captured images, and reference numeral 70 is an image that is to be captured by a virtual camera, that is, a generated image. In order to determine the pixel value of the pixel 71 in the generated image 70, the intersection detection unit 42 obtains a curve 80 passing through the pixel 71 among the curves indicating the same position in the real space on the ray space. The pixel 51 at the intersection with the photographed image 50 and the pixel 61 at the intersection between the curve 80 and the photographed image 60 are determined. Note that the curve 80 is calculated based on the model information of the captured images 50 and 60, more specifically, based on the position information of each pixel of the captured images 50 and 60 in the real space. Can be requested.

For example, as shown in FIG. 5, consider a ray passing through a point 94 at a distance r from the origin at a direction φ. As is clear from FIG. 5, the intersection of the light beam traveling through the point 94 in the θ direction and the P axis is
r × sin (φ−θ) (1)
And the points 94 are distributed in a sinusoidal shape in the ray space. More precisely, each ray passing through the point 94 will be distributed sinusoidally in the ray space. The intersection detection unit 41 obtains the position of each pixel in the light beam space of the captured image based on the camera parameters, position, and direction of each camera, and determines the position of each pixel from the model information, that is, φ and r in Expression (1). The curve 80 passing through the pixel 71 is obtained from the camera direction and camera parameters assumed to be installed at the viewpoint positions designated as these, the pixels 51 and 61 are determined, and the intersection information indicating the pixels 51 and 61 is obtained. Is output to the corresponding point detector 42.

  If the installation position of each camera is accurate, each camera is accurately directed to the center, and further, the camera parameters are completely calibrated and the model information is accurate, the pixel 51 and the pixel 61 are Each of the pixels corresponds to the pixel 71. However, it is usually difficult to completely adjust the position and direction of each camera, and the camera parameter characteristic difference cannot be completely compensated for by calibration. There is also an error. Therefore, normally, the pixel 51 and the pixel 61 are not the corresponding pixels. For this reason, the corresponding point detection unit 42 sets a search area 52 of a predetermined range centered on the pixel 51 and a search area 62 of the same predetermined range centered on the pixel 61, and supports this by block matching in this range. Search for a block to perform. That is, the combination having the highest correlation among the blocks in the search area 52 and the blocks in the search area 62 is searched. Specifically, for example, for each combination of each block in the search area 52 and each block in the search area 62, the sum of the absolute values of the differences in the values of the pixels at the same position or the difference of 2 Find the combination of blocks with the smallest sum of powers. In FIG. 3B, it is determined that the block 53 and the block 63 correspond to each other. Note that the height and width of the search area are determined based on an assumed camera installation position and direction error, a calibration error, and an object position error in the model information.

The corresponding point detection unit 42 corresponds to the pixel 71, the central pixel of the block having the highest correlation, in the case of FIG. 3B, the pixel 54 at the center of the block 53 and the pixel 64 at the center of the block 63. The pixel is determined to be a pixel, and corresponding point information indicating the pixel 54 and the pixel 64 is output to the pixel value calculation unit 43. The pixel value calculation unit 43 calculates the pixel value of the pixel 71 by the following formula based on the corresponding point information.
Pixel value of the pixel 71 = a × pixel value of the pixel 54+ (1−a) × pixel value of the pixel 64 where a is 0 <a <1, and the distance between the pixel 54 and the pixel 71 in the ray space is The value depends on the distance between the pixel 71 and the pixel 64 with respect to the sum of the distance between the pixel 64 and the pixel 71.

  As described above, the ray space construction unit 4 obtains a pixel in the captured image corresponding to the calculation for each pixel of the generated image, and blocks the search area centered on this point. Perform matching, thereby determining the set of pixels to use for the pixel values of the generated image. The curve 80 in FIG. 3 is not a line connecting the same points in the real space due to camera parameters, camera position errors, etc., but if the camera parameters and camera position errors are not significantly large, they are close to the curve 80. There will be a corresponding pixel. In the present invention, a search region is set around a point determined by the curve 80, and a corresponding point is searched by block matching. In this way, by limiting the search range based on the calculated position, the corresponding points can be easily and quickly found.

  As described above, an actual camera that captures a captured image and a virtual camera that is assumed for a generated image are arranged on the circumference so as to face the center, and all camera parameters are the same. As described in the embodiment, the actual and virtual camera installation positions, directions, and camera parameters are used to determine the correspondence between the pixels and the light space. As long as it can be determined, there are no restrictions on the installation position, direction and camera parameters of the actual and virtual cameras. In other words, as long as it is premised that the first space where the subject exists and the second space where there is no subject and the actual camera is placed are clearly distinguished, the actual camera in the second space For example, a straight line, an ellipse, or any other shape can be arranged in the direction of the space, and the viewpoint for the generated image is at an arbitrary position in the first space and the second space. Can be set.

  In the above-described embodiment, the heights and widths of the search areas 52 and 62 are fixed values determined in advance. For example, according to the distance in the light space with respect to the captured images 50 and 60 of the pixels 71. It can also be variable. That is, for example, the search area 52 may be reduced as the pixel 71 approaches the captured image 50 and the search area 52 may be increased as the pixel 71 moves away. Since the error range varies depending on the distance from the pixel 71, the search area can be made variable according to the position of the pixel 71, so that a more appropriate search area can be obtained. Further, the position information of each object may be acquired by a sensor described in Non-Patent Document 3, for example, instead of being determined based on a captured image.

  Note that it is not necessary to determine the pixel values of all the pixels of the generated image based on the same set of captured images. For example, as shown in FIG. 6, cameras 21, 22, and 23 are set and captured images are acquired, and a virtual camera 24 that is assumed to be a viewpoint set between the cameras 21 and 22 is captured from these captured images. Consider the case of generating a generated image. Reference numerals 25 and 26 are subjects, that is, objects. As shown in FIG. 6A, since the light rays from the point 95 on the surface of the object 25 reach the camera 21 and the camera 22, the pixel value of the pixel corresponding to the point 95 of the generated image is determined by the camera 21. And a set of captured images captured by the camera 22 can be generated. However, as shown in FIG. 6B, the light rays from the point 96 on the surface of the object 25 reach the camera 21 and the camera 23, but do not reach the camera 22 because they are shielded by the object 26. . Therefore, the pixel value of the pixel corresponding to the point 96 of the generated image is generated from a set of captured images captured by the camera 21 and the camera 23. The occurrence of so-called occlusion shown in FIG. 6B is determined based on the model information.

  It should be noted that the two captured images used for determining the pixel value of the generated image at a certain viewpoint are preferably selected so that their camera positions sandwich the viewpoint. Specifically, as shown in FIG. 6, when a straight line is drawn in the direction of the viewpoint at the viewpoint position of the generated image, the captured images taken by the two cameras arranged on both sides of the straight line are taken. It is preferable to use it. Of course, the captured image is selected from those including pixels corresponding to the pixels for determining the pixel value of the generated image.

  In the above-described embodiment, the generated image is generated based on the captured image actually captured by the camera. However, the image generation device also has information on the position of each pixel in the light space and the position in the real space for the generated image generated based on the captured image, as in the captured image. And can be used to determine pixel values of pixels of yet other generated images. In this case, the captured image in the above embodiment is replaced with the generated image, and the camera position and direction of the captured image are replaced with the viewpoint position and direction of the generated image. That is, the captured image can be regarded as a generated image with the camera position and direction as the viewpoint.

  The image generation apparatus according to the present invention can be realized by a program that causes a computer to function as each unit in FIGS. 1 and 2. The computer program is stored in a computer-readable storage medium or can be distributed via a network. Furthermore, only a part of each part shown in FIGS. 1 and 2 of the present invention is realized by hardware, and the other part is realized by a computer program, that is, it can be realized by a combination of hardware and software.

1 is a block diagram of an image generation apparatus according to the present invention. It is a block diagram of a light space construction part. It is a figure explaining the process of a light space construction part. It is explanatory drawing of the image generation apparatus by this invention. It is a figure explaining distribution in ray space of one point of real space. It is a figure explaining the relationship of the picked-up image used for determination of the pixel value of a production | generation image.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1, 11-18, 21-23 Camera 11-1 to 18-1 Photographed image 2 Storage part 24 Virtual camera 25, 26 Object 3 Preprocessing part 4 Ray space construction part 41 Intersection detection part 42 Corresponding point detection part 43 Pixel value calculation unit 50, 60 Captured image 51, 61, 54, 64 Captured image pixel 52, 62 Search area 53, 63 Block 70 Generated image 71 Generated image pixel 80 Curve 90 Ray 94, 95, 96 points

Claims (5)

An image generation device that determines a pixel value of a third pixel of a third image at a specified viewpoint based on a first image and a second image,
Based on the position information of each pixel of the first and second images in the light space and real space, the distribution of points indicating light rays from one point in the real space, and the designated viewpoint, Intersection detection means for determining a first pixel of the first image and a second pixel of the second image corresponding to three pixels;
Block matching using a block of a predetermined pixel size is applied in the first search area that is a range centered on the first pixel and the second search area that is a range centered on the second pixel. Corresponding point detection means for obtaining the first block in the first search region and the second block in the second search region, which is the most highly correlated combination;
A pixel value calculating means for determining a pixel value of the third pixel based on the pixel at the center of the first block and the pixel at the center of the second block ;
An image generating apparatus configured such that ranges of the first search area and the second search area are smaller as the position of the third pixel is closer to the position of the third pixel in the light space, and are larger as the distance is increased . Based on the positions and directions of a plurality of cameras and the captured images captured by the plurality of cameras, pre-processing for calibrating camera parameters of the plurality of cameras and determining positional information of each pixel of the captured images in real space Means,
Storage means for storing the captured image and a generated image generated from the captured image;
With
The first image and the second image are the captured image and the generated image, respectively.
Each pixel of the generated image is determined by the intersection detection unit, the corresponding point detection unit, and the pixel value calculation unit.
The image generation apparatus according to claim 1. The viewpoints of the first image and the second image are on both sides of a straight line extending in the direction of the viewpoint at the designated viewpoint of the third image.
The image generating apparatus according to claim 1 or 2. A method of determining a pixel value of a third pixel of a third image at a specified viewpoint in an image generation device based on a first image and a second image,
The intersection detection unit, based on the position information of each pixel of the first and second images in the light space and real space, the distribution of the positions of the real space in the light space, and the designated viewpoint, Determining a first pixel of the first image and a second pixel of the second image corresponding to three pixels;
The corresponding point detection unit sets a rectangular first search region that is a range centered on the first pixel to the first image, and a rectangular second search region that is a range centered on the second pixel. Setting a second image;
The corresponding point detection unit obtains the correlation between each block having a predetermined pixel size in the first search area and each block having the predetermined pixel size in the second search area, thereby obtaining the first Determining a second block in the second search area corresponding to the first block in the search area;
A pixel value calculation unit comprising: determining a pixel value of a third pixel based on a pixel at the center of the first block and a pixel at the center of the second block ;
A method in which the range of the first search area and the second search area is changed so as to become smaller as the position of the third pixel is closer to the position of the third pixel in the ray space and to be larger as the distance is increased . A program that causes a computer to function as the image generation apparatus according to claim 1.

Images