JP2010079506A - Image generating apparatus, method, communication system, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image generating apparatus for quickly creating an image of a designated viewpoint, while suppressing previously prepared data quantity to implement a free visual point video system. <P>SOLUTION: The image generating apparatus stores multiple images and a disparity map associated with a set having two images included in the multiple images. The disparity map shows a correspondence relationship among the pixels in the two associated images, and the image generating apparatus generates an image from a designated viewpoint, with the designated viewpoint being based on the multiple pixels and the disparity map. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a free viewpoint video generation technique.

  Various proposals have been made on free viewpoint video technology for reproducing video at an arbitrary position from moving images taken by a plurality of cameras. (For example, see Patent Document 1, Non-Patent Document 1, and Non-Patent Document 2,).

  The above-described method in the prior art is called image-based rendering. A light space is constructed from images taken by a plurality of cameras, and an image viewed from an arbitrary position is interpolated based on the light space. It is generated by processing.

JP 2008-15756 A Takeshi Naemura, et al. "Ray-Based Creation of Photo-Realistic Virtual World", VSMM97, pp. 59-62, September 1997 Michael Drose, et al. “Ray-Space Interpolation based on Filtering in Disparity Domain”, Proc. of 3D Image Conference 2004, pp. 29-30, 2004

  For example, in order to construct a free viewpoint video system that generates a moving image viewed from a viewpoint when the viewpoint, that is, a position and a direction is specified on a user terminal such as a personal computer, and displays it on the user terminal. Therefore, it is necessary to shorten the time required for moving image generation by the interpolation processing as much as possible. By accumulating images from almost all viewpoint positions in the image generation device by interpolation processing in advance, the time required for moving image generation can be shortened, but a huge amount of storage devices are required. Not right.

  Accordingly, an object of the present invention is to provide an image generation apparatus, a method, a communication system, and a program that causes a computer to function as the image generation apparatus that solve the above-described problems.

According to the image generation apparatus of the present invention,
Storage means for storing a disparity map associated with a set of a plurality of images and two images included in the plurality of images, and indicating a correspondence relationship between pixels in the two images associated with each other; A viewpoint image generation unit that generates a first image that is an image of a specified viewpoint based on a plurality of images and a parallax map is provided.

According to another embodiment of the image generation apparatus of the present invention,
In order to determine the pixel value of the first pixel of the first image, the viewpoint image generation means changes the first pixel and the light ray from the set of images to an image having a viewpoint between the viewpoints of the two images of the set. The first pixel is selected based on the second image and the third image, which are images of the selected set, and the disparity map associated with the selected set. The second pixel of the second image and the third pixel of the third image corresponding to the second pixel are determined, and the pixel value of the first pixel is determined based on the second pixel and the third pixel. preferable.

According to another embodiment of the image generation apparatus of the present invention,
The viewpoint image generation means, when there are a plurality of sets having an image including a pixel at the same position in the light ray space as the first pixel in an image having a viewpoint between the viewpoints of the two images of the set, It is also preferable to select one set based on the position of the subject and determine the pixel value of the first pixel based on the selected set.

According to another embodiment of the image generation apparatus of the present invention,
One or more parallax maps are provided corresponding to parallax map viewpoints that are viewpoints between the viewpoints of two associated images, and each parallax map corresponds to a pixel of an image at the parallax map viewpoint. , Indicating pixels in two associated images, and when there are a plurality of disparity maps associated with a set of the second image and the third image, the viewpoint image generating means selects one disparity map and The second pixel and the third pixel are determined, and the parallax map to be selected has the same position in the ray space as the first pixel, where the parallax map viewpoint is between the viewpoints of the images included in the selected set. It is also preferable that the image is closest to the viewpoint of the image having the pixels.

Furthermore, according to another embodiment of the image generation apparatus of the present invention,
A storage unit that stores a plurality of images and a depth map indicating position information of each pixel of the plurality of images, and generates a first image that is an image of a specified viewpoint based on the plurality of images and the depth map. And a viewpoint image generating means.

Furthermore, according to another embodiment of the image generation apparatus of the present invention,
In order to determine the pixel value of the first pixel of the first image, the viewpoint image generation means changes the first pixel and the light ray from the set of images to an image having a viewpoint between the viewpoints of the two images of the set. Select a set having an image including pixels at the same position in space, and correspond to the first pixel of the first image based on the second image and the third image, which are images of the selected set, and the depth map It is also preferable to determine the second pixel of the second image and the third pixel of the third image, and determine the pixel value of the first pixel based on the second pixel and the third pixel.

According to the program of the present invention,
A computer is caused to function as the image generation apparatus.

According to the system of the present invention,
An image generation device that stores a plurality of images and a parallax map associated with a set composed of two images included in the plurality of images, and can communicate with the image generation device via a communication network And a terminal device that notifies the image generation device of the viewpoint of the first image to be performed, wherein the disparity map indicates a correspondence relationship between pixels in two associated images. .

According to another embodiment of the system of the present invention,
The image generation device selects one set from the set of images to generate the pixel value of the first pixel of the first image in the terminal device, and the second image and the second image that are images of the selected set 3 and the disparity map associated with the selected set to the terminal device, and the terminal device, based on the received disparity map, the second pixel of the second image corresponding to the first pixel and The group of the third image is determined, the pixel value of the first pixel is determined based on the second pixel and the third pixel, and the group selected by the image generation apparatus uses the viewpoint as the viewpoint. It is also preferable that the image is a set having an image including a pixel at the same position in the light ray space as the first pixel.

According to the image generation method of the present invention,
An image generation method in an image generation apparatus for storing a plurality of images and a parallax map associated with a set composed of two images included in the plurality of images, wherein the parallax map includes two associated parallax maps The correspondence between the pixels in the image is shown, the viewpoint input unit obtains the viewpoint of the first image to be generated, and the viewpoint image generation unit generates the first image based on the acquired viewpoint, the plurality of images, and the parallax map. 1 is provided with a step of generating an image.

According to another embodiment of the image generation method of the present invention,
The step of generating the image includes a step of obtaining a set of the images having viewpoints on both sides of the first pixel of the first image and a viewpoint of an image having a pixel at the same position in a light space, and generating a viewpoint image. The second pixel of the second image corresponding to the first pixel based on the second image and the third image, which are images of the obtained set, and the parallax map associated with the obtained set And determining a third pixel of the third image and determining a pixel value of the first pixel based on the second pixel and the third pixel.

  In the present invention, it does not have information about all the light rays in the light space, but is generated by a parallax map or a depth map when necessary. Calculation of pixel values using a parallax map or a depth map is simple, and thus the number of images prepared in advance can be reduced.

  The best mode for carrying out the present invention will be described in detail below with reference to the drawings. In the following description, “viewpoint” is a value determined by the position and direction, and “viewpoint image” or “viewpoint image” is a predetermined centered on the direction indicated by the viewpoint at the position indicated by the viewpoint. It means an image representing a range. Further, the expression “image A” is used to refer to the image at the viewpoint A. When the image at the viewpoint B is actually taken by the camera, the “image B” and the position of the camera are referred to as the “camera position B” in order to refer to this image.

  FIG. 1 is a functional block diagram of an image generation apparatus according to the present invention. According to FIG. 1, the image generation apparatus includes a storage unit 1, a parallax map generation unit 2, a prior image generation unit 3, a transmission processing unit 4, and a viewpoint input unit 5. FIG. 2 is a functional block diagram of the user terminal according to the present invention. According to FIG. 2, the user terminal includes a viewpoint image generation unit 6 and a viewpoint transmission unit 7. Note that the image generation apparatus and the user terminal according to the present invention communicate via a communication network such as the Internet, for example. FIG. 1 and FIG. 2 show only the parts necessary for explaining the present invention.

  The storage unit 1 of the image generation apparatus generates camera parameters, arrangement positions and directions of a plurality of cameras 100, each image (hereinafter referred to as a captured image) that constitutes a moving image captured by each camera 100, and prior image generation. The unit 3 holds an image (hereinafter, referred to as an interpolated image) generated from the captured image by interpolation processing, and the parallax map generated by the parallax map generating unit 2 based on the captured image and the interpolated image. The plurality of cameras 100 are assumed to be an area or space where no subject exists (hereinafter referred to as a first space) for photographing a subject existing in a predetermined area or space (hereinafter referred to as a second space). Are arranged on a predetermined line. In the following description, it is assumed that the predetermined line is a circle or a straight line. When the predetermined line is a circle, the optical axis of each camera faces the center of the circle at the same height. The optical axes of the cameras are assumed to be the same height and face the same direction. In addition, the captured image also holds information indicating the correspondence relationship of which camera has captured. In the following description, when both a captured image and an interpolated image are shown, they are simply referred to as “images”.

  The viewpoint transmission unit 7 of the user terminal transmits viewpoint information indicating the viewpoint designated by the user to the image generation apparatus, and the viewpoint input unit 5 of the image generation apparatus transmits the viewpoint information received from the user terminal. Output to part 4. Based on the viewpoint information, the transmission processing unit 4 generates images and parallax map data necessary for generating a moving image viewed from the viewpoint designated by the user among the images and parallax maps stored by the storage unit 1. The viewpoint image generation unit 6 of the user terminal that transmits to the user terminal generates a moving image viewed from the specified viewpoint based on the received data. In addition, the structure which produces | generates the moving image seen from the viewpoint which the user specified on the image generation apparatus side, and transmits to a user terminal may be sufficient, and in this case, it is the transmission process part 4 of an image generation apparatus. The function of the viewpoint image generation unit 6 is incorporated, and the user terminal performs processing for displaying data indicating the received moving image.

  FIG. 3 is a block diagram of the pre-image generation unit 3. According to FIG. 3, the prior image generation unit 3 includes a preprocessing unit 31, a first interpolation image generation unit 32, a second interpolation image viewpoint determination unit 33, and a second interpolation image generation unit 34. The preprocessing unit 31 performs camera calibration based on the captured images taken by the respective cameras at approximately the same time, adjusts camera parameters, determines whether each pixel is the foreground or the background, and determines the foreground. For each pixel, the object such as a subject is modeled, each object and its position are recognized, and model information indicating the object and its position is generated for each captured image. Various known methods can be used for these processes.

  Subsequently, the first interpolation image generation unit 32 generates an interpolation image with the position between the cameras 100 as the viewpoint on the line where the cameras 100 are arranged. FIG. 4 is a diagram for explaining an interpolation image generated by the first interpolation image generation unit 32. In FIG. 4, the solid line arrow indicates the viewpoint of the captured image, and the dotted line arrow indicates the viewpoint of the interpolation image. For example, FIG. 4A shows a case where the camera 100 is arranged so as to face the same direction along the straight line 10, and FIG. 4B shows a case where the camera 100 is circled along the circumference 11. The case where it arrange | positions so that it may face the center of is shown. Note that the density of viewpoint positions to be set, that is, the density of interpolated images generated between captured images of adjacent viewpoints, is determined by the required viewpoint resolution, and the interpolated image is located at a position close to the viewpoint. A certain captured image, for example, a captured image on both sides is generated using a weighting according to the distance. Model information is also generated for the generated interpolated image.

  The second interpolation image viewpoint determination unit 33 determines the viewpoint of the interpolation image to be further generated in the second space based on the position and size of the target object. FIG. 5 is a diagram for explaining processing performed by the second interpolation image viewpoint determination unit 33. 5A and 5B correspond to the camera arrangements of FIGS. 4A and 4B, respectively, and the dotted arrows indicate the viewpoints to be set. For example, as shown in FIG. 5A, when the camera 100 is arranged along the straight line 10 so as to face the same direction, the object 100 or a region on the opposite side to the straight line 10 of the target object 21 is located. A straight line parallel to the straight line 10 is set, and at least two viewpoints facing the same direction as the camera 100 are set on the set straight line. Note that the number of viewpoints is determined based on the size of the object 21, and increases as the object 21 becomes larger. Preferably, the intersection of the object 21 and the line extending from the viewpoint position to the viewpoint direction of the image with the viewpoint on the straight line 10 is determined as the viewpoint. That is, the viewpoint is set on the straight line set at the position of the object 21 with the same density as the viewpoint on the straight line 10.

  Further, as shown in FIG. 5B, when the camera 100 is arranged along the circumference 11 so as to face the center of the circle, the object 21 is placed on a line along the boundary of the object 21 or On the surrounding circle or ellipse, the viewpoint is set so as to face at least four directions orthogonal to each other. As in the case of arranging in a straight line, the number of viewpoints is determined based on the size of the object 21, and the number of viewpoints increases as the object 21 becomes larger. For example, at the intersection of a line extending from the viewpoint and the object 21 in an image with the viewpoint on the circumference 11, the viewpoint in the direction of the extended line or the curve on the object 21 is the same as the circumference 11. Set the density perspective.

  In addition, as shown in FIG.5 (c), the viewpoint set on the line along the boundary of the target object 21 or on the circle | round | yen or ellipse surrounding the target object 21 is set to face the outside of these closed curves. As shown in FIG. 5 (d), there is a configuration in which it is set to face the inside of the closed curve, both of which are possible.

  The second interpolation image generation unit 34 uses the interpolation image at the viewpoint determined by the second interpolation image viewpoint determination unit 33 as an image having a viewpoint on the straight line 10 or the circumference 11, that is, the captured image captured by the camera 100 and the first image. 1 is generated based on the interpolation image generated by the interpolation image generation unit 32. FIG. 6 is a diagram for explaining generation of an interpolation image in the second interpolation image generation unit 34. FIG. 6 shows a case where the cameras 100 are arranged in a straight line, and an interpolation image at the viewpoint 15 set at the position of the object 21 is generated. For example, in order to generate the interpolated image 15, it is necessary to determine the pixel value at the position of the point 23 of the object 22, but among the images having viewpoints on a straight line, Since the point 23 is shielded by the object 21, the image 12 cannot be used to determine the pixel value of the point 23. Further, since the point 23 is outside the range of the visual field of the viewpoint indicated by reference numeral 13, that is, outside the range of the image at the viewpoint, the image 13 cannot be used for determining the pixel value of the point 23. On the other hand, since the point 23 is within the range of the visual field of the viewpoint indicated by reference numeral 14 and there is no obstruction between the viewpoint 14 and the point 23, the second interpolated image generation unit 34 Based on the pixel value of the pixel corresponding to 23, the pixel value of the pixel corresponding to the point 23 of the interpolated image 15 is determined.

  In this way, the second interpolation image generation unit 34 uses an image having a viewpoint on the straight line 10 or the circumference 11 to be used for generating an interpolation image at the viewpoint determined by the second interpolation image viewpoint determination unit 33. This is determined based on model information for an image having a viewpoint on the straight line 10 or the circumference 11. In other words, interpolation processing is performed by determining an image having pixel value information for each pixel of the interpolation image and having a viewpoint on the straight line 10 or the circumference 11 based on the position of the object indicated by the model information of these images. To generate an interpolated image.

  Returning to FIG. 1, the parallax map generation unit 2 applies 1 to each of the adjacent sets of the image having the viewpoint on the straight line 10 or the circumference 11 and the image having the viewpoint on the straight line or curve set at the position of the object. One or more parallax maps are generated. FIG. 7 is a diagram illustrating a set of images for generating a parallax map. Note that solid arrows in FIG. 7 indicate images at the respective viewpoints. As shown in FIG. 7A, when there are four images on a straight line, one or more parallax maps associated with each of the image sets A, B, and C are generated. Also, as shown in FIG. 7B, when there are four images in a circle, one or more parallax maps respectively associated with the image sets A, B, C, and D are generated. Hereinafter, generation of a parallax map will be described.

  FIG. 8 is a diagram illustrating generation of a parallax map. The parallax map is associated with a set of images of adjacent viewpoints, and is also associated with viewpoints between the viewpoints of the images on a line where the associated images exist. Here, the direction of the associated viewpoint is the same direction as the images on both sides, that is, when the viewpoint of the associated image is in the first space, for example, on the line 10 or the circumference 11. In this case, the direction is the direction determined in the same way as the viewpoint for the interpolation image determined by the second interpolation image generation unit 33 when the object is in the second space. FIG. 8 shows a case where three parallax maps are generated that are associated with the set of the viewpoint 500 image and the viewpoint 600 image and correspond to the viewpoints 500, 600, and 700, respectively. For example, when the viewpoint of the associated image is a curve such as a circumference 11, a parallax map is associated with a viewpoint on a straight line connecting the viewpoints of the associated image instead of the curve. You may create it. In this case, the direction of the viewpoint is a direction close to the viewpoint direction of the images on both sides of the two directions perpendicular to the straight line.

  In the following description, the expression that a straight line is set between images at each viewpoint is used. This is because images at the positions of the viewpoints are parallel to each other and lines connecting pixels at the same position are on the image plane. This is based on the premise that the images are arranged so that the distance between the images corresponds to the distance between the viewpoints on the line where the images exist.

  FIG. 8A is an explanatory diagram for generating a parallax map corresponding to the viewpoint 500. A search range centering on the pixel of the image 600 at the same position as the pixel 51 of the image 500 is set in the image 600, and the pixel of the image 600 corresponding to the pixel 51 is determined from this search range. In the determination of the corresponding pixel, block matching using a block of a predetermined size centered on the pixel 51 is applied to the search range of the image 600, and the highest correlation with the block centered on the pixel 51 is found from the search range of the image 600. Do by finding blocks. That is, for each block within the search range of the image 600, the sum of the absolute values of the differences in the values of the pixels at the same position of the block centered on the pixel 51 or the sum of the squares of the differences in the values is obtained. The smallest block is found from the search range of the image 600. In FIG. 8A, it is determined that the pixel 61 corresponds. In this case, the shift Δa from the same position as the pixel 51 of the image 600 is recorded in the parallax map corresponding to the viewpoint 500 as the parallax information for the pixel 51 as shown in FIG. Note that the parallax information is actually a two-dimensional value indicating a difference in position between the horizontal direction and the vertical direction. The search range is determined based on the distance between the viewpoint 500 and the viewpoint 600. In this manner, parallax information is determined for all pixels of the image 500.

  FIG. 8B is an explanatory diagram for generating a parallax map corresponding to the viewpoint 600. Similar to the generation of the parallax map corresponding to the viewpoint 500, the search range is set to the image 500, and the pixel of the image 500 corresponding to the pixel 62 is determined from this search range. In FIG. 8B, it is determined that the pixel 52 corresponds. In this case, the deviation Δb from the same position as the pixel 62 of the image 500 is recorded in the parallax map corresponding to the viewpoint 600 as the parallax information for the pixel 62 as shown in FIG.

  FIG. 8C is an explanatory diagram for generating a parallax map corresponding to the viewpoint 700. When it is assumed that there is an image at the viewpoint 700, a pixel 73 of this image is assumed, and search ranges centered on a pixel at the same position as this pixel 73 are set to the image 500 and the image 600, respectively. Subsequently, a straight line that passes through the pixel 73 and intersects the image 500 and the image 600 within each search range is obtained, and the corresponding pixel is obtained by block matching from the set of pixels at the intersection of the obtained straight line and the image 500 and the image 600. Determine. Specifically, the correlation of a block of a predetermined size centered on each of the two intersections of the image 500 and the image 600 is obtained for each straight line, and the central pixel of the block with the highest correlation is set as the corresponding pixel. In FIG. 8C, pixels 53 and 63 are obtained. In this case, the deviation Δc1 from the same position as the pixel 73 is recorded in the parallax map at the viewpoint 700 as the parallax information for the pixel 73 as shown in FIG.

  Since the pixels 53, 63, and 73 exist on a straight line, Δc2 may be recorded instead of Δc1. Further, as shown in FIGS. 8A to 8C, in the disparity information, each pixel of the image at the corresponding viewpoint of the disparity map corresponds to which pixel of the set of images associated with the disparity map. For example, it may be specified by another method such as a slope of a line connecting corresponding pixels.

  8A and 8B are special forms in which the viewpoint corresponding to the parallax map is in the same position as the viewpoints 500 and 600 of the associated image. When the pixel of the image is used as a reference, the pixel determined to correspond most to the reference pixel among the pixels of the associated image is specified. In addition, since the parallax information is searched based on different viewpoints, for example, the corresponding pixels of the images 500 and 600 indicated by the parallax maps of the viewpoints 500, 600, and 700 do not normally match. In the present invention, preferably, two disparity maps are generated in correspondence with both viewpoints of the associated images. Preferably, further, a parallax map at a viewpoint at an intermediate position of the associated image is generated.

  Next, generation of an image of a designated viewpoint (hereinafter referred to as a viewpoint image) in the transmission processing unit 4 and the viewpoint image generation unit 6 will be described. FIG. 10 is a diagram illustrating generation of a viewpoint image when the designated viewpoint is in the same direction as each image on a line where the viewpoint of each image is located. In FIG. 10, reference numerals 500, 600, 700, 800, and 900 indicate viewpoints, and the storage unit 1 stores the viewpoint 500 associated with the viewpoint 500 image, the viewpoint 600 image, and the set of the image 500 and the image 600. , 600 and 700, respectively, and viewpoint images at viewpoints 800 and 900 are generated.

  First, the transmission processing unit 4 determines which of the viewpoints of the parallax map the viewpoint of the viewpoint image is close to. Subsequently, the pixels of the image 500 and the image 600 used to calculate the pixel value of each pixel of the viewpoint image are determined based on the closest parallax map. For example, in FIG. 10, since the viewpoint 600 is closest to the viewpoint 900 among the viewpoints 500, 600, and 700, the viewpoints 500, 600, and 700 are determined in order to determine each pixel value of the image at the viewpoint 900. It is determined that the parallax map at the viewpoint 600 is used among the parallax maps at. Below, the determination of the pixel value in the viewpoint image generation part 6 is demonstrated concretely.

First, based on the parallax map, the pixels of the image 500 and the image 600 corresponding to the pixel 94 of the image at the viewpoint 900 are determined. For this reason, a search range of a predetermined range centering on the pixel corresponding to the pixel 94 in the image 600 at the viewpoint 600 that is the parallax map to be used is set in the image 600. This search range is determined based on the distance between the viewpoint 900 and the viewpoint 600, similarly to the generation of the parallax map shown in FIG. Subsequently, a straight line is drawn from each pixel in this search range to the corresponding pixel in the image 500 according to the parallax information of each pixel in the parallax map at the viewpoint 600, and a pixel that passes through the pixel 94 is searched. In FIG. 10, a straight line from the pixel 64 to the pixel 54 passes through the pixel 94. In this case, the viewpoint image generation unit 6 determines the pixel value of the pixel 94 as follows.
Pixel value of the pixel 94 = a × pixel value of the pixel 54+ (1−a) × pixel value of the pixel 64 where a is 0 <a <1, and the viewpoint 900 with respect to the distance between the viewpoint 500 and the viewpoint 600 is The value depends on the distance of the viewpoint 600. When there are a plurality of straight lines passing through the pixel 94, for example, the straight line having the smallest inclination is selected and the above formula is applied. If there is no straight line passing through the pixel 94, the straight line passing through the position closest to the pixel 94 is selected and the above formula is applied.

  Similarly, in FIG. 10, since the viewpoint 700 is closest to the viewpoint 800, for each pixel of the image at the viewpoint 800, the corresponding pixels of the image 500 and the image 600 are determined based on the parallax map at the viewpoint 700. To do. As in the case of the pixel 94, the search range is set around the pixel at the viewpoint 700 corresponding to the pixel 85 at the viewpoint 800. Subsequently, based on the disparity information of each pixel of the disparity map at the viewpoint 700, a straight line that passes through each pixel in the search range is drawn between the image 500 and the image 600, and among these straight lines, the one that passes through the pixel 85 is drawn. Explore. In FIG. 10, a straight line drawn based on the parallax information for the pixel 75 at the viewpoint 700 passes through the pixel 85, and this straight line intersects the pixel 55 of the image 500 and the pixel 65 of the image 600. Therefore, the value of the pixel 85 is determined based on the pixels 55 and 65 as in the case of the pixel 94. The processing when there are a plurality of straight lines passing through the pixel 85 or when there is no straight line is the same as that described for the pixel 94.

  Next, a general viewpoint image generation method in the viewpoint image generation unit 6, that is, a method other than the above-described case will be described. FIG. 11 is an explanatory diagram of a viewpoint image generation method. In FIG. 11A, reference numerals 43 and 44 denote images of viewpoints on a straight line or a circumference where the camera 100 is arranged, that is, an image directly acquired by the camera 100 or the first interpolation image generation unit 32. The generated image is shown. Reference numerals 41 and 42 indicate images generated by the second interpolation image generation unit due to the presence of the object 21. A dotted line 40 is a viewpoint image to be generated. As shown in FIG. 11 (c), in the representation of images in reference numerals 40 to 44, the plane 47 is the screen of the image, the point 48 is the viewpoint position, and the normal direction is the point 48 to the plane 47. Represents the direction of the viewpoint. In this case, for example, the pixel value of the pixel 49 on the surface 47 is determined by the light ray 27 that passes through the pixel 49 and reaches the viewpoint 48.

  In order to generate the viewpoint image 40, the viewpoint image generation unit 6 needs to determine, for example, the pixel value of the light beam indicated by reference numeral 25 in FIG. For this reason, the transmission processing unit 4 selects a set of the image 43 and the image 44 and the parallax map thereof having the viewpoint positions on both sides of the line obtained by extending the ray 25, and the viewpoint image generation unit 6 selects the selected set and its Based on the parallax map, the pixel value of the pixel corresponding to the light ray 25 of the viewpoint image 46 is calculated by the method described with reference to FIG. 10, and this is set as the pixel value corresponding to the light ray 25 of the viewpoint image 40.

  That is, the transmission processing unit 4 obtains a point in the light space corresponding to the pixel to be obtained, for example, the light ray 25 in FIG. 11B, and among the sets of images associated with the viewpoint map, the image of the set. A group including pixels by the light ray 25 is selected in an image having a viewpoint between the viewpoints. That is, the images 43 and 44 are selected in FIG. Thereafter, the pixel value of the pixel by the light ray 25 of the viewpoint image 40 is determined based on the found set of images and the parallax map of the set. The pixel value may be corrected according to the distance between the viewpoint positions 40 and 46.

  Similarly, the viewpoint image generation unit 6 needs to determine the pixel value of the pixel by the light beam indicated by reference numeral 24 in FIG. For this reason, the viewpoint image generation unit 6 calculates the pixel value of the pixel corresponding to the light ray 24 of the viewpoint image 45 based on the set of the image 41 and the image 42 and the parallax map thereof using the method described with reference to FIG. This is the pixel value corresponding to the light ray 24 of the viewpoint image 40. In addition, although the light ray shown with a dotted line and the light ray shown with the code | symbol 24 are the same points in light ray space, the viewpoint image generation part 6 is not the light ray shown with a dotted line from model information, but the light ray 24 of the image 40. Determine that it is necessary for generation.

  In other words, the viewpoint image generation unit 6 is a pixel that is the same position in the ray space as the pixel to be obtained in an image having a viewpoint between the viewpoints of the images of the set among the sets of images associated with the viewpoint map. If a plurality of sets are found, one set is selected based on the position of the subject and the viewpoint of the viewpoint image 40, and the pixel value is determined based on the selected set of images. To do.

  In order to realize a free viewpoint video, first, a space is divided into a second space where a subject exists and a first space where no subject exists, and a plurality of subjects are directed toward the subject in the first space. Although shooting is performed with a camera installed, the number of cameras that can be used is usually limited, so an image whose viewpoint is the position where the camera is not installed is created by interpolation processing. Become. However, all images corresponding to the required resolution of the viewpoint position are created in advance by interpolation processing and saved in the storage unit 1, that is, all rays in the light space are created and saved in advance. It is not realistic in terms of storage capacity and processing load. For this reason, a parallax map is generated for a set of images, and pixels corresponding to light rays not prepared in advance are generated based on the parallax map. In other words, it does not have information about all the light rays in the light space, but generates them with a parallax map when necessary. The calculation of the pixel value using the parallax map is simple, and thus the number of images prepared in advance can be reduced. For this reason, the arrangement position of the camera in the first space, the viewpoint position of the interpolated image with the viewpoint in the first space created based on the image captured by the camera, and the position corresponding to the viewpoint of the parallax map are determined in advance. Decide it. In the above-described embodiment, a straight line or a circle is used, but it is only necessary to determine the position of each image pixel in the light space, and the position of each image pixel in the light space is calculated based on the installation position and direction of the camera and camera parameters. Since this is possible, the present invention can define an arbitrary line in the first space, place a camera on the line, generate an interpolated image, and generate a parallax map.

  However, a captured image captured by a camera arranged on a line set in the first space, and an interpolation image generated from these captured images and having a viewpoint position on the same line as the camera are located at a position shielded by the object. Pixel values cannot be calculated quickly. For example, since the light ray 24 shown in FIG. 11B is shielded by the object 21, the intersection of the light ray indicated by a dotted line in the drawing and the line where the camera 100 is arranged is an extension of this light ray 24. The captured image and the interpolated image in the vicinity do not have information on the light ray 24, and the information on the light ray 24 is an image having pixel information corresponding to the light ray 24 based on model information or the like as described with reference to FIG. Must be generated by searching. Since these processes take time, in the present invention, an interpolated image and a parallax map are generated in advance even at the position of the object in the second space. As a result, it is possible to quickly generate an image at a viewpoint beyond the object as seen from the position of the camera.

  In FIG. 8, the search range is set to a predetermined range centered on the corresponding point. This corresponds to a form in which the cameras are arranged in a circumferential shape. For example, when the cameras are arranged in a straight line, a shift occurs only in one direction. Therefore, as shown in FIG. The half in the horizontal direction in the form shown in FIG.

  In the above-described embodiment, the interpolation image is generated at the position of the object. However, as another embodiment, a line for setting a viewpoint in advance is set in the second space regardless of the position of the object. It may be in a predetermined form. For example, in FIG. 13, reference numeral 200 indicates a boundary between the first space and the second space, and a plurality of circumferences 16 that are concentric with the circumference 11 in which the camera set in the first space is arranged. 17 may be set in advance in the second space, and an interpolation image may be generated with the center direction as the viewpoint in this circumferential shape. In this case, the generation of the image of the portion shielded by the object is inferior to the above-described embodiment, but the second interpolation image position determination unit 33 is not necessary, and the image generation load prepared in advance is not necessary. Can be lightened. In addition, the forms of (c) and (d) in FIG. 5 bring about a result that images generated when an object is present at a viewpoint position are different.

  Furthermore, the role of the parallax map in the above-described embodiment can be replaced with, for example, a depth map created from model information generated by the preprocessing unit 31. Here, the depth map indicates information indicating the object and its position. In this case, as shown in FIG. 14, light rays 24 and 25 corresponding to the point 23 are obtained based on the positions of the images 43 and 44 and the position 23 of the object 22, and the point 23 in the viewpoint image 46 is obtained. The viewpoint image is generated by obtaining the light ray 26 corresponding to the pixel value. In addition, the generated image is used as it is, but it can also be used after being converted so as to correct the deviation of the optical axis based on the camera calibration result.

  The image generation apparatus according to the present invention can be realized by a program that causes a computer to function as the above-described image generation apparatus. The computer program is stored in a computer-readable storage medium or can be distributed via a network. Furthermore, only a part of each functional block of the present invention is realized by hardware, and the other part is realized by a computer program, that is, can be realized by a combination of hardware and software.

It is a functional block diagram of the image generation apparatus by this invention. It is a functional block diagram of the user terminal by this invention. It is a block diagram of a prior image generation part. It is a figure explaining the interpolation image produced | generated by a 1st interpolation image production | generation part. It is a figure explaining the determination of the viewpoint in a 2nd interpolation image viewpoint determination part. It is a figure explaining the production | generation of the interpolation image in a 2nd interpolation image generation part. It is a figure explaining the group of the image which produces | generates a parallax map. It is a figure explaining the production | generation of a parallax map. It is a figure which shows the exemplary form of a parallax map. It is a figure explaining the determination of the pixel value of the viewpoint image by a parallax map. It is explanatory drawing of viewpoint image generation. It is a figure which shows the other form of the setting of a search range. It is a figure which shows the other form of the viewpoint determination of a 2nd interpolation image. It is a figure explaining the determination of the pixel value of the viewpoint image by model information.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Storage part 2 Parallax map generation part 3 Prior image generation part 4 Transmission processing part 5 Viewpoint input part 6 Viewpoint image generation part 7 Viewpoint transmission part 10 Straight line 12, 13, 14, 41, 42, 43, 44 Image 11, 16, 17 Circumference 15, 48, 500, 600, 700, 800, 900 Viewpoint 21, 22 Target 23, 49 Point 31 Pre-processing unit 32 First interpolation image generation unit 33 Second interpolation image viewpoint determination unit 34 Second interpolation image Generation unit 40, 45, 46 Viewpoint image 47 Surface 24, 25, 26, 27 Ray 51-56, 61-65, 73, 75, 85, 94 Pixel 100 Camera

Claims (11)

A storage unit that stores a disparity map that is associated with a set of a plurality of images and two images included in the plurality of images, and that indicates a correspondence relationship between pixels in the associated two images;
Viewpoint image generation means for generating a first image that is an image of a specified viewpoint based on the plurality of images and the parallax map;
An image generation apparatus. In order to determine the pixel value of the first pixel of the first image, the viewpoint image generation means changes the first pixel and the light ray from the set of images to an image having a viewpoint between the viewpoints of the two images of the set. The first pixel is selected based on the second image and the third image, which are images of the selected set, and the disparity map associated with the selected set. Determining a second pixel of the second image and a third pixel of the third image corresponding to, and determining a pixel value of the first pixel based on the second pixel and the third pixel;
The image generation apparatus according to claim 1. The viewpoint image generation means, when there are a plurality of sets having an image including a pixel at the same position in the light ray space as the first pixel in an image having a viewpoint between the viewpoints of the two images of the set, One set is selected based on the position of the subject, and the pixel value of the first pixel is determined based on the selected set.
The image generation apparatus according to claim 2. One or more parallax maps are provided corresponding to the parallax map viewpoint that is the viewpoint between the viewpoints of two associated images,
The disparity map indicates pixels in the two associated images corresponding to the pixels of the image at the disparity map viewpoint,
When there are a plurality of parallax maps associated with the set of the second image and the third image, the viewpoint image generation unit selects one parallax map, determines the second pixel and the third pixel,
The parallax map to be selected is the one closest to the viewpoint of an image having a pixel located at the same position in the light space as the first pixel, where the parallax map viewpoint is between the viewpoints of the images included in the selected set. ,
The image generation apparatus according to claim 2. Storage means for storing a plurality of images and a depth map indicating position information of each pixel of the plurality of images;
Viewpoint image generation means for generating a first image that is an image of a specified viewpoint based on the plurality of images and the depth map;
With
Image generation device. In order to determine the pixel value of the first pixel of the first image, the viewpoint image generation means changes the first pixel and the light ray from the set of images to an image having a viewpoint between the viewpoints of the two images of the set. Select a set having an image including pixels at the same position in space, and correspond to the first pixel of the first image based on the second image and the third image, which are images of the selected set, and the depth map Determining a second pixel of the second image and a third pixel of the third image, and determining a pixel value of the first pixel based on the second pixel and the third pixel;
The image generation apparatus according to claim 5.   A program that causes a computer to function as the image generation apparatus according to claim 1. An image generation device that stores a plurality of images and a parallax map associated with a set including two images included in the plurality of images;
A terminal device capable of communicating with the image generation device via a communication network and notifying the image generation device of a viewpoint of a first image to be generated;
A communication system comprising:
The parallax map indicates a correspondence relationship between pixels in two associated images.
Communications system. The image generation device selects one set from the set of images to generate the pixel value of the first pixel of the first image in the terminal device, and the second image and the second image that are images of the selected set 3 images and the disparity map associated with the selected set are transmitted to the terminal device,
The terminal device determines, based on the received parallax map, the second pixel of the second image and the third pixel of the third image corresponding to the first pixel, and the second pixel and the third pixel Determining the pixel value of the first pixel based on the pixel;
The set selected by the image generation device is a set having an image including a pixel at the same position in the light ray space as the first pixel in an image whose viewpoint is between the viewpoints.
The communication system according to claim 8. An image generation method in an image generation apparatus for storing a plurality of images and a parallax map associated with a set composed of two images included in the plurality of images,
The parallax map indicates a correspondence relationship between pixels in two associated images;
A viewpoint input unit acquiring a viewpoint of the first image to be generated;
A step in which a viewpoint image generation unit generates an image based on the acquired viewpoint, the plurality of images, and the parallax map;
A method comprising: Generating the image comprises:
A step in which a viewpoint image generation unit obtains a set of images having viewpoints on both sides of a viewpoint of an image having a first pixel of the first image and a pixel at the same position in a light space;
The second image and the third image, which are images of the obtained set, and the second image of the second image corresponding to the first pixel based on the parallax map associated with the obtained set. Determining a second pixel and a third pixel of the third image and determining a pixel value of the first pixel based on the second pixel and the third pixel;
The method of claim 10 comprising:

Images