JP5011168B2 - Virtual viewpoint image generation method, virtual viewpoint image generation apparatus, virtual viewpoint image generation program, and computer-readable recording medium recording the program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a new technology for generating a virtual viewpoint image with high subjective quality with a small amount of calculation when the virtual viewpoint image is generated seeing the subject from a different viewpoint from a camera when using an image for the subject photographed by the camera. <P>SOLUTION: In a case using a method for generating the virtual viewpoint image by the small amount of calculation using simple geometrical information which is the distance from cameras to the subject in a multiviewpoint image and with each element in the multiviewpoint image, when the virtual viewpoint image is generated by synthesizing sampled beam information by the multiviewpoint image, a boundary region in which a plurality of beam information with different distances from the cameras is mixed is determined. Then, the sampled beam in the boundary region among the sampled beams by the multiviewpoint image is treated as the beam with lower intensity than the beam sampled in other regions and the virtual viewpoint image is generated by carrying out its synthesis. <P>COPYRIGHT: (C)2009,JPO&amp;INPIT

Description

  The present invention relates to a virtual viewpoint image generation method for generating an image (virtual viewpoint image) when the subject is viewed from a position or orientation (virtual viewpoint) different from that of the camera using an image of the subject photographed by the camera, and The present invention relates to an apparatus, a virtual viewpoint image generation program used to realize the virtual viewpoint image generation method, and a computer-readable recording medium on which the program is recorded.

  In the field of computer graphics (CG) and virtual reality (VR), images of objects (subjects) appearing in images taken with cameras are viewed from different positions and orientations of those cameras. The technology to generate is actively researched.

  Hereinafter, the position and orientation of the image to be generated are referred to as a virtual viewpoint, and the generated image is referred to as a virtual viewpoint image. This conceptual diagram is shown in FIG.

  There are two types of methods for generating a virtual viewpoint image: a geometric base and an image base.

  In a typical geometry-based virtual viewpoint image generation method, after obtaining a geometric three-dimensional model of a subject expressed using polygons and voxels from images taken by multiple cameras called multi-viewpoint images. The virtual viewpoint image is generated by performing projective transformation corresponding to the virtual viewpoint camera on the three-dimensional model (see, for example, Non-Patent Document 1).

  This method of acquiring a 3D model and generating a virtual viewpoint image has the advantage that the process of generating an arbitrary virtual viewpoint image from the 3D model is very lightweight, but the 3D model is highly accurate. It has a problem that it is very difficult to acquire in a real environment. In addition, since it is converted into one three-dimensional model, there is a problem that the difference in appearance depending on the viewpoint acquired by the multi-viewpoint image is lost, and the quality of the generated image is poor.

  On the other hand, the image-based virtual viewpoint image generation method is called image-based rendering, and based on the multi-viewpoint image, the virtual viewpoint image is obtained by using the two-dimensional image processing without explicitly obtaining the object model. Is generated (see, for example, Non-Patent Document 2).

  Theoretically speaking, in the image-based virtual viewpoint image generation method, the camera is considered to be a device that samples light information in real space, and the space where the subject exists from light information discretely obtained as a multi-viewpoint image. A virtual viewpoint image is generated by sampling the ray space with a virtual viewpoint camera.

  The image-based virtual viewpoint image generation method can be used to generate extremely high-quality images, but the required multi-viewpoint image has a high viewpoint position density and requires a large number of cameras. Not only is there a need to process huge amounts of data. This enormous amount of data affects not only the processing time but also storage / transmission, so it becomes a big problem regardless of real-time processing / offline processing.

  Therefore, a method of generating a virtual viewpoint image using a multi-view image and simple geometric information such as a distance from the camera to the subject in each pixel of the multi-view image instead of a three-dimensional model of the subject has been proposed (for example, Non-Patent Document 3).

  Specifically, a multilayer plane is assumed at a certain position of the subject, and the distance information is used to appropriately map the light ray information sampled to each pixel of the multi-viewpoint image, and then the virtual plane is virtualized from the multilayer plane. A virtual viewpoint image is generated by estimating light ray information sampled by the viewpoint camera.

  In this method, since the light source (subject position) can be known from the distance information, a large number of cameras are not required unlike the image-based method, and information sampled from a multi-viewpoint image is 1 Instead of combining them into one model, each is handled as light ray information emitted from a specific position and orientation, so that the quality of the generated virtual viewpoint image is not extremely bad. Further, since the sampling process of the ray information in the multilayer plane by the virtual viewpoint camera can be realized by a simple rendering process as in the geometry-based method, the virtual viewpoint image can be generated at a very high speed.

  On the other hand, in this method, it is assumed that light ray information sampled by one pixel of the multi-viewpoint image is generated from a certain point on the subject.

  However, in an actual camera, a light beam passing through a certain area in real space is acquired for one pixel. For this reason, in some cases, a plurality of light rays generated from different subjects are sampled by one pixel at the boundary portion of the subject.

  Since the light sampled at each pixel is mapped on the multilayer plane using one distance, if the mapping is performed using the distance of the foreground object to this mixed light, the mixed light Of course, the rays of the subject in the background will be mapped to the wrong position, and conversely, if the mapping is performed using the distance of the subject of the background, the rays of the foreground subject out of the mixed rays will be in the wrong position. Will be mapped.

  This deviation causes noise such as a ghost in the generated virtual viewpoint image, and significantly reduces the subjective quality of the virtual viewpoint image.

In order to solve this problem, the following Non-Patent Document 4 describes a method of obtaining and separating foreground and background distances and light rays from light rays sampled by one pixel and separately mapping them on a multilayer plane. Has been taken.
PEDebevec, CJTaylor, and J. Malik, "Modeling and rendering architecture from photographs: A hybrid geometry- and image-based approach," In Proc. ACM Annu. Computer Graphics Conf., Aug. 1996, pp. 11-20. Marc Levoy and Pat Hanrahan: “Light Field Rendering,” In Proc. SIGGRAPH, 1996, pp.34-41. Yutaka Kunida et al., “Real-time system for generating arbitrary viewpoint images using multi-lens cameras,” IEICE Journal, Vol.J84-DII, No.1, 2001, pp.129-138. A. Fitzgibbon, Y. Wexler, A. Zisserman, "Image-based rendering using image-based priors," In International Conference on Computer Vision, vol.2, 2003, pp.1176-1183.

  As in Non-Patent Document 4, ray information in one pixel is separated into foreground and background, and each is mapped to a multilayer plane using appropriate distances, and rays sampled from the multilayer plane by a virtual viewpoint camera. By estimating the information, a virtual viewpoint image with high subjective quality can be generated.

  However, this method has a problem that a lot of calculations are required because the foreground and background rays that best represent the ray information sampled by the multi-viewpoint image must be obtained. there were.

  The present invention has been made in view of such circumstances, in the case of generating a virtual viewpoint image when a subject reflected in an image captured by a camera is viewed from a virtual viewpoint different from the viewpoint of the camera, An object of the present invention is to provide a new virtual viewpoint image generation technique that enables generation of a virtual viewpoint image with high subjective quality with a small amount of calculation.

[1] First Invention [1-1] Configuration of First Invention In order to achieve the above object, the first invention of the present invention provides n images obtained by photographing a subject from a plurality of cameras with different viewpoints. A multi-viewpoint image composed of a plurality of subject images, and distance information corresponding to each of the n subject images from a camera that has photographed the multi-viewpoint image to a certain point on the subject reflected in each pixel of the subject image. A virtual viewpoint image generation method for generating a virtual viewpoint image that is an image of a subject viewed from a virtual viewpoint specified in advance, and (a) for each of the n subject images, A boundary region setting step for setting a boundary region of a subject for each of the subject images based on the distance information; and (b) coordinates and virtual viewpoint images on the subject images for each of the n subject images. Pair with top coordinate The relationship, said a corresponding relationship determining step of determining by using the distance information corresponding to each object image, for each (c) the n pieces of each object image, by using the correspondence determined in correspondence decision step, the subject A composite image generation step for generating a composite subject image (image that is a composition source of the virtual viewpoint image) from the image, and (d) the correspondence relationship determined in the correspondence relationship determination step is set for each subject image. A combined boundary region setting step for setting a combined boundary region (a boundary region of a subject in the combined subject image) for each of the combined subject images based on the information on the boundary region; and (e) n combined subject images. , whether the weighted addition in accordance with the information of the synthesis boundary area set for each synthetic subject image, wherein the synthetic boundary set for each synthesis object image And weighted addition in accordance with the information of the virtual viewpoint information area, characterized in that it comprises a virtual viewpoint image generation step of generating a virtual viewpoint image.

Here, the virtual viewpoint image generation device of the present invention that realizes the first invention captures a multi-view image composed of n subject images obtained by capturing a subject from a plurality of cameras having different viewpoints, and the multi-view image. This is an image of a subject viewed from a virtual viewpoint designated in advance using distance information corresponding to each of the n subject images from a camera to a certain point on the subject reflected in each pixel of the subject image. However, in order to generate a virtual viewpoint image, (a) for each of the n subject images, a boundary region for setting a subject boundary region for each subject image from the subject image and its distance information And (b) for each of the n subject images , a correspondence relationship between the coordinates on the subject images and the coordinates on the virtual viewpoint image is determined using distance information corresponding to the subject images. Do And response relationship determining means, and (c) for each of said n pieces of each object image, using the determined correspondence relationship relationship determining means (Synthesis originated the image of a virtual viewpoint image) synthesized subject image from the subject image Using the composite image generation means to be generated and (d) the correspondence determined by the correspondence determination means, the information on the boundary region set for each subject image is combined with each composite subject image. A composite boundary region setting means for setting a boundary region (a subject boundary region in the composite subject image); and (e) n composite subject images of the composite boundary region set for each composite subject image. or weighted addition in accordance with the information, and weighted addition the following information and the information and the virtual viewpoint of the synthesis boundary area set for each synthetic subject image, the temporary generating a virtual viewpoint image And a viewpoint image generation unit.

[1-2] About the first invention In the first invention of the present invention, a boundary region in which light ray information of a plurality of subjects having different distances from the camera is mixed is determined, and a multi-viewpoint image is displayed only at the time of weighted addition. Among the light rays sampled by the above, the light ray sampled in the boundary region is treated as a light ray having a lower intensity than the light ray sampled in the other region to generate a virtual viewpoint image.

  As a result, the influence of light rays mixed with light ray information from a plurality of subjects can be reduced, and a high-quality virtual viewpoint image can be generated. In addition, in this method, the process for obtaining the ray at the virtual viewpoint can be performed simply by changing the ratio at the time of synthesis according to whether or not the ray is sampled in the boundary region. It does not require enormous operations such as a process of separating the foreground and background light rays that are most appropriately represented and combining the light rays in the virtual viewpoint image therefrom.

  Any method may be used to determine the boundary region. As the simplest example, a portion where the distance information changes rapidly can be determined as the boundary region. Further, not only distance information but also edge information in a multi-viewpoint image can be used for determination.

[2] Second Invention [2-1] Configuration of Second Invention The second invention of the present invention is similar to the first invention, in the virtual viewpoint image generation step, the image of the boundary region of the multi-viewpoint image Is differentiated from images in other regions and weighted addition is performed. However, if there is at least one composite subject image that is not a composite boundary region in each pixel of the generated virtual viewpoint image, only those composite subject images Is used for weighted addition.

[2-2] Regarding the second invention That is, in the second invention of the present invention, the intensity of the light beam in the boundary region is not weakened but is treated as a light beam with low reliability. Then, when obtaining a ray at a pixel with a virtual viewpoint, if there is even one ray sampled in an area other than the boundary area, that is, a ray with low reliability, only the ray is used for the virtual viewpoint. Find the rays.

  Since the multi-viewpoint image is a group of images obtained by photographing the subject from a plurality of positions and orientations, the region in the real space that becomes the boundary region in the background subject is different for each sampled subject image. For this reason, light rays determined as boundary regions at a certain point and light rays that are not so are sampled. In such a place, it is possible to generate a high-quality virtual viewpoint image by obtaining the light ray at the virtual viewpoint without using the light ray determined to be a boundary region in which the light rays of a plurality of subjects are mixed.

[3] Third Invention [3-1] Configuration of Third Invention In addition to the first invention and the second invention, the third invention of the present invention is a multi-viewpoint that corrects a subject image in a boundary region. An image correction step is provided, and in the composite image generation step, a composite subject image is generated using the subject image corrected in the multi-viewpoint image correction step.

[3-2] About 3rd invention The light ray sampled in the boundary region is also mapped to the light ray in three-dimensional space using the given one distance information. At this time, if the distance information to be used corresponds to the foreground subject, more accurate rays are used when generating the virtual viewpoint image by removing the influence of the rays depending on the background subject from the sampled rays. It will be done. If the distance information used corresponds to the subject in the background, more accurate rays were used when generating the virtual viewpoint image by removing the influence of the rays depending on the foreground subject from the sampled rays. It will be.

  Therefore, in the third invention of the present invention, in the boundary region, correction is made to the light beam appropriately sampled according to the distance information, thereby generating a higher-quality virtual viewpoint image. Can do.

  In the method according to the third aspect of the present invention, the region where the processing corresponding to the subject is applied to the light ray is limited to the boundary region, so that an enormous amount of calculation is not required. In addition, instead of separating the rays of the foreground subject and background subject, only correction is made so that they are close to the rays of the subject according to the distance information. And not.

[4] Fourth Invention [4-1] Configuration of Fourth Invention In addition to the first invention, the second invention, and the third invention, the fourth invention of the present invention is distance information in the boundary region. A distance information correction step for correcting the relationship, and in the correspondence setting step, the correspondence is determined using the distance information corrected in the distance information correction step.

[4-2] Fourth Invention As a method of obtaining distance information, there are a method of estimating using a multi-viewpoint image and a method of obtaining using a special device. Whichever method is used, noise is included. For example, a distance that is too different from the surrounding distance may be given.

  In the fourth aspect of the present invention, by removing such noise, it becomes possible to map the sampled light beam to a light beam in an appropriate three-dimensional space, and generate a higher-quality virtual viewpoint image. be able to.

  On the other hand, in order to remove the impulse noise as described above, the distance information may be smoothed beforehand. In addition, when estimating distance information from a multi-viewpoint image, there is a case where a process is performed in which the distance does not change greatly in the adjacent region in consideration of the continuity of the subject in real space. In such a case, the distance information is supposed to be a boundary area, and the distance information is incorrect, and the generated virtual viewpoint image is distorted.

  In the fourth invention of the present invention, by performing the process of sharpening the edge in such a region, or by performing the process of matching the edge obtained from the multi-viewpoint image and the edge of the distance image, A higher-quality virtual viewpoint image can be generated without being mapped to a light beam in a three-dimensional space in which the background and the foreground subject are connected.

[5] Fifth Invention [5-1] Configuration of Fifth Invention In addition to the first invention, the second invention, the third invention, and the fourth invention, the fifth invention of the present invention includes: (A) a virtual viewpoint distance information generating step for generating virtual viewpoint distance information as distance information of a virtual viewpoint image from distance information corresponding to each of the n subject images ; (b) virtual viewpoint distance information; A virtual viewpoint boundary region setting step for setting a virtual viewpoint boundary region that is a boundary region of a subject with respect to the virtual viewpoint image from the virtual viewpoint image; and (c) each pixel in the virtual viewpoint boundary region of the virtual viewpoint image. And a virtual viewpoint image correction step of changing the pixel value according to information on surrounding pixels.

[5-2] About the fifth invention As described above, the light rays sampled in the boundary region of the multi-viewpoint image are mixed with the light rays of a plurality of subjects. Therefore, even in the generated virtual viewpoint image, it is natural that the light rays of the foreground subject and the background subject are mixed in the boundary region. Actually, when the rays of the background subject are not mixed in the portion near the foreground subject in the boundary area, it looks as if the foreground subject is floating. This can also be seen from the fact that even if a photograph of a landscape is combined with a clip of a person's photograph, the image becomes uncomfortable.

  According to the fifth aspect of the present invention, a boundary area is obtained for the virtual viewpoint image, and the pixel values of the pixels in the area are determined using the pixel values of surrounding pixels that are considered to be another subject. By correcting, it is possible to reduce the above-mentioned uncomfortable feeling and generate a virtual viewpoint image with high subjective quality.

[6] The first related inventions [6-1] first relevant aspect of structure present invention of the first related invention, the subject image where an image obtained by photographing a certain subject, were taken the subject image A virtual viewpoint image that generates a virtual viewpoint image that is an image of a subject viewed from a virtual viewpoint specified in advance using distance information from the camera to a certain point on the subject that appears in each pixel of the subject image (A) a boundary region setting step for setting a boundary region of a subject with respect to the subject image from the subject image and distance information; and (b) a distance information correction step for correcting distance information in the boundary region. And (c) a subject image correction step for correcting the subject image in the boundary region, and (d) a distance information correction step that shows the correspondence between the coordinates on the subject image and the coordinates on the virtual viewpoint image. And (v) a virtual viewpoint that generates a virtual viewpoint image from the subject image corrected in the subject image correction step using the correspondence relationship determined in the correspondence relationship determination step. An image generation step.

Here, the virtual viewpoint image generation device that realizes the first related invention is shown in a subject image that is an image obtained by photographing a certain subject and each pixel of the subject image from the camera that photographed the subject image. In order to generate a virtual viewpoint image that is an image of the subject viewed from the virtual viewpoint specified in advance using the distance information to a certain point on the subject, (a) from the subject image and the distance information, Boundary area setting means for setting a boundary area of a subject with respect to an image; (b) distance information correction means for correcting distance information in the boundary area; and (c) subject image correction means for correcting a subject image in the boundary area. (D) correspondence determining means for determining the correspondence between the coordinates on the subject image and the coordinates on the virtual viewpoint image using the distance information corrected by the distance information correcting means; Using the determined correspondence relationship relationship determining means, and a virtual viewpoint image generation means for generating a virtual viewpoint image from the corrected subject image of the subject image correction means.

[6-2] In the first related aspect of the present invention for the first associated invention, to determine the boundary region where light information is mixed different subjects of the distance from the camera is sampled only in that region The virtual ray image is generated by correcting the ray information to the ray of the subject according to the distance information.

  As a result, the influence of light rays mixed with light ray information from a plurality of subjects can be reduced, and a high-quality virtual viewpoint image can be generated. In addition, since this method newly sets information called boundary areas and performs only the process of correcting the light rays only in those areas, the calculation for obtaining the foreground and background rays that best represents the sampled ray information. Does not require a huge amount of operations.

[7] The second associated invention [7-1] second related aspect of construction the invention of the second associated invention, in addition to the first associated invention, corresponding determined in (i) correspondence relationship determining step Using the relationship, the virtual viewpoint boundary region setting step for setting the virtual viewpoint boundary region (the subject boundary region in the virtual viewpoint image) from the information on the boundary region of the subject image, and (b) the virtual viewpoint boundary region setting step A virtual viewpoint image correction step of changing the pixel value of each pixel on the virtual viewpoint image generated in the virtual viewpoint image generation step which is defined as a boundary region by the virtual viewpoint boundary region, according to information on surrounding pixels. To do.

[7-2] Second related invention According to the second related invention of the present invention, the boundary region in the virtual viewpoint image is determined using the virtual viewpoint boundary region generated from the boundary region of the subject image, and the inside of the region is determined. By correcting the pixel values of the pixels with the pixel values of neighboring pixels that are considered to be another subject, the discomfort that the foreground subject is separated from the background is reduced, and the subjective quality High virtual viewpoint images can be generated.

[8] distance third related invention [8-1] Third related aspect of construction the invention of the third associated invention, in addition to the first related inventions, corrected by (i) the distance information correction step A virtual viewpoint distance information generating step for generating virtual viewpoint distance information, which is distance information of the virtual viewpoint image generated in the virtual viewpoint image generating step, and (b) a virtual viewpoint generated in the virtual viewpoint distance information generating step. A virtual viewpoint boundary region that sets a virtual viewpoint boundary region that is a boundary region of a subject with respect to the virtual viewpoint image generated in the virtual viewpoint image generation step from the distance information and the virtual viewpoint image generated in the virtual viewpoint image generation step A virtual viewpoint image generated in the virtual viewpoint image generation step that is set as a boundary region by the setting step and (c) the virtual viewpoint boundary region set in the virtual viewpoint boundary region setting step The pixel value of each pixel of the upper, characterized in that it comprises a virtual viewpoint image correction step of changing in accordance with information of the surrounding pixels.

[8-2] Third related invention According to the third related invention of the present invention, a virtual viewpoint boundary in which a boundary region in a virtual viewpoint image is generated from virtual viewpoint distance information (generated from distance information of a subject image) The foreground subject is separated from the background by determining using the region and correcting the pixel values of the pixels in that region using the pixel values of surrounding pixels that are considered to be another subject. It is possible to reduce a sense of discomfort and to generate a virtual viewpoint image with high subjective quality.

  According to the present invention, when generating a virtual viewpoint image, by treating the light ray information acquired in the boundary region of the subject as a light beam with low intensity or a light beam with low reliability, a virtual viewpoint image with high subjective quality can be obtained. It can be generated with a small amount of calculation.

  Hereinafter, the present invention will be described in detail according to embodiments.

  Here, in the embodiment described below, a multi-viewpoint image captured by N cameras and a camera that captures the image required for each image of the multi-viewpoint image are reflected in each pixel. A method of generating a virtual viewpoint image when the subject is viewed from a specified virtual viewpoint based on distance information (defined for each pixel) to a point on the subject will be described.

  The distance information may be input using information estimated using a multi-viewpoint image, or may be input using information measured using a special device. However, it must match the coordinate system definition of the camera parameters of the multi-viewpoint image to be used.

[1] First Embodiment FIG. 1 illustrates an embodiment of a virtual viewpoint image generation apparatus 100 according to the present invention.

  As shown in this figure, the virtual viewpoint image generation device 100 according to the present embodiment includes a virtual viewpoint input unit 101, an image input unit 102, a distance information input unit 103, an input information analysis unit 104, and a correspondence setting. Unit 105, composite image generation unit 106, composite image memory 107, composite boundary region setting unit 108, composite boundary region memory 109, virtual viewpoint image generation unit 110, virtual viewpoint distance information generation unit 111, virtual A viewpoint boundary region setting unit 112, a virtual viewpoint image correction unit 113, and a virtual viewpoint image output unit 114 are provided.

  The virtual viewpoint input unit 101 inputs information on a virtual viewpoint to be generated. The image input unit 102 inputs a multi-viewpoint image. The distance information input unit 103 inputs distance information on each pixel of the multi-viewpoint image (distance information from the camera to a point on the subject shown in each pixel of the multi-viewpoint image).

  The input information analysis unit 104 analyzes the input multi-viewpoint image and the distance information, corrects the input information, and sets a boundary region (a subject boundary region) using them. The correspondence setting unit 105 sets a correspondence (a correspondence between pixels) between each image of the multi-viewpoint image and the virtual viewpoint image.

  The composite image generation unit 106 generates a composite image that is a synthesis source of the virtual viewpoint image using each image of the multi-viewpoint image according to the correspondence relationship obtained by the correspondence relationship setting unit 105. The composite image memory 107 stores the composite image group generated by the composite image generation unit 106.

  The composite boundary region setting unit 108 uses the boundary region set for each image of the multi-viewpoint image according to the correspondence obtained by the correspondence relationship setting unit 105, and determines the position where the boundary region appears on the virtual viewpoint image. A composite boundary region to be shown (a boundary region on the composite image) is obtained. The composite boundary area memory 109 stores information on the composite boundary area obtained by the composite boundary area setting unit 108.

  The virtual viewpoint image generation unit 110 generates a virtual viewpoint image from the composite image group generated by the composite image generation unit 106. The virtual viewpoint distance information generation unit 111 generates distance information for the virtual viewpoint image (distance information from the camera of the virtual viewpoint to the point on the subject reflected in each pixel of the virtual viewpoint image). The virtual viewpoint boundary area setting unit 112 obtains the boundary area of the subject in the virtual viewpoint image. The virtual viewpoint image correcting unit 113 corrects the virtual viewpoint image. The virtual viewpoint image output unit 114 outputs the virtual viewpoint image finally generated by the virtual viewpoint image correction unit 113.

  FIG. 2 illustrates a flowchart executed by the virtual viewpoint image generation device 100 of the present embodiment configured as described above. Here, this flowchart shows processing when generating an image at a specified virtual viewpoint from a multi-viewpoint image photographed in synchronism with the camera by the virtual viewpoint image generation apparatus 100 of the present embodiment.

  Below, according to this flowchart, the process which the virtual viewpoint image generation apparatus 100 of the example of this embodiment performs is demonstrated in detail. It is assumed that camera parameters of each camera that has captured a multi-viewpoint image are given.

[Overview of Step A1 to Step A11]
First, the virtual viewpoint input unit 101 inputs virtual viewpoint information [step A1]. Here, it is assumed that an internal parameter A and external parameters R and t of the camera are designated as the virtual viewpoint.

  In addition, you may designate with the relative positional relationship etc. with respect to the camera of the multiview image which image | photographed. In that case, the camera internal parameter A and the external parameters R, t at the virtual viewpoint are calculated from the designated relative relationship and the camera parameters of the camera of the multi-viewpoint image.

  Next, in this embodiment, the process of generating a composite image / composite boundary region is repeated for each image constituting the multi-viewpoint image.

  In the present embodiment example, an index indicating each image constituting the multi-viewpoint image is denoted as View, and the number of images constituting the multi-viewpoint image is denoted as numViews. Therefore, after initializing View to 0 [Step A2], while adding 1 to View [Step A10], the following processing [Step A3 to Step A9] is repeated until View becomes numViews [Step A11]. Thus, a composite image / composite boundary region is generated for each image constituting the multi-viewpoint image.

  In the following, since confusion does not occur, a camera that captures a multi-viewpoint image indicated by the index View may be represented by View. For convenience of description, View may be described as view in the formula.

[Step A3, Step A4 Processing]
In the process [Step A3 to Step A9] that is repeated for each image constituting the multi-viewpoint image, first, the image input unit 102 inputs an image corresponding to View [Step A3]. Hereinafter, the input image is represented as ORG _view [•]. Further, by adding position information sandwiched between symbols [] to an image, an image signal of a specific region is represented.

Subsequently, the distance information input unit 103 inputs the distance information corresponding to the View image [step A4]. Hereinafter, the distance information is represented as D _view [•]. Further, by adding position information sandwiched between the symbols [] to the distance information, the distance information of a specific area of the corresponding image is represented.

[Process of Step A5]
Subsequently, the input information analysis unit 104 analyzes and corrects the input image and distance information.

The input information analysis unit 104 first sets the boundary region of the object using the input distance information and generates the boundary region information [step A5]. In the following, the boundary area information is represented as BOUND _view [•]. Further, by adding position information sandwiched between symbols [] to the boundary area information, the boundary area information of a specific area of the image is represented. Details of the processing for setting the boundary area information will be described in detail later.

The boundary area information may be expressed in any way, but here, if the pixel at the position p is not the boundary area of the subject, BOUND _view [p] ≠ 0, and if it is the boundary area, BOUND _view [p] = 0 The boundary area information is as follows. Further, when it is not a boundary region, it may include information indicating how far away from the boundary. For example, a value determined by adding 0 to a minimum value of boundary area information of an adjacent pixel is 0 for a pixel determined not to be continuous with an adjacent pixel, and a pixel determined to be continuous with all adjacent pixels. This is boundary area information. Even in this case, it is not necessary to indicate the distance from the boundary in all the pixels, and the maximum value MaxDistance of the boundary area information may be determined to represent the distance only for a certain range of pixels.

  In this embodiment, boundary region information including information indicating how far away from the boundary is generated, and the maximum value is 3.

[Process of Step A6]
The input information analysis unit 104 corrects the image and distance information input in the boundary area after setting the boundary area information [step A6].

  In the distance information correction executed in step A6, distance information that is erroneously smoothed in the boundary region, distance information that does not match the image, and the like are corrected. Here, when the edge position of the distance information and the edge position of the image do not match in integer pixel units in the boundary region, a process of rewriting the distance information is performed so that the edges match.

  Further, in the image correction executed in step A6, a state in which the light rays of a plurality of objects that have been generated as a result of sampling light rays with finite and discrete pixels in the boundary region are mixed is corrected. . Here, processing is performed to rewrite the image information of the target pixel with the image information of the pixel having distance information close to the target pixel in the surrounding pixels.

  Here, a pixel that is determined to belong to the same region by performing region division from the distance information and the image information, not from the image information of the pixels having the distance information close to the distance information of the target pixel among the surrounding pixels The image information of the target pixel may be rewritten with the image information.

  Furthermore, instead of simply rewriting with the average value of the image information of the surrounding pixels, the weight calculation is performed using the distance from the target pixel and the distance information of the pixel, and the image information of the surrounding pixels is weighted using the weight. The image information of the target pixel may be rewritten using the value calculated by addition. Examples of weight calculation formulas include the following formulas (1) to (4).

In these mathematical expressions, p represents the position of the target pixel, and p _i represents the pixel position of the weight calculation target. Exp () is a function that returns the exponent of the number of Napiers. ‖‖ represents the norm. Here, σ _I and σ _D are control parameters, and any values can be used, but by setting values depending on the statistical properties of the image, higher quality image information can be generated. become. Note that when these weights are used, the total value of the weights does not become 1 depending on the number or position of pixels to be added, and thus it is necessary to normalize and use them.

[Process of Step A7]
After the input information analysis unit 104 corrects the input image and distance information in the boundary region, the correspondence setting unit 105 uses the camera parameter and the distance information to input the input image ( The correspondence relationship between the pixels of the multi-viewpoint image pointed to by the index View is calculated [step A7].

  The processing here is conceptually described because the mathematical formula varies depending on the camera parameter. First, each pixel of the input image is back-projected into a three-dimensional space using the camera parameter for the input image. At this time, by using the distance information, it is possible to obtain a corresponding point of the pixel in a straight line connecting the camera focus and the pixel to be reverse projected. Once the three-dimensional point can be calculated, the coordinate value on the virtual viewpoint image can be calculated by reprojecting the point onto the image plane using the camera parameters for the virtual viewpoint image. A correspondence relationship between the pixels and the pixels of the virtual viewpoint image is obtained.

  Specifically, when the correspondence between the image coordinate m and the world coordinate M uses the camera parameter expression obtained by the following equation (5), the correspondence is calculated using the following equation (6). can do. Note that the tilde symbol represents a homogeneous coordinate that allows arbitrary scalar multiplication.

  Here, A, R, and t represent the internal parameter matrix, rotation matrix, and translation vector of the camera, respectively. Also, the subscript indicates which camera information the parameter is, and New (for convenience of description, New may be described as new in the formula) represents a virtual viewpoint camera. A and R are 3 × 3 matrices, and t is a three-dimensional vector. (U, v) represents a position on the virtual viewpoint image, and (x, y) represents a position on the input image.

D _view [x, y] represents distance information from the camera View (the camera that captured the input image) to a point on the subject that is captured in each pixel of the input image, and D _{new, view} [u, v ] Represents the distance information from the virtual viewpoint camera New to the point on the subject shown in each pixel of the virtual viewpoint image. Note that the subscript view is described in D _{new, view} [u, v] in order to indicate from which input image the distance information is obtained.

  In this conversion, since a plurality of pixels of the input image are associated with one pixel of the virtual viewpoint image, a large number of pixels for which effective distance information cannot be obtained may occur. However, distance information is not obtained outside the occlusion area. Therefore, in consideration of continuity between pixels in the input image, corresponding points may be obtained for the pixels of the virtual viewpoint image.

  As a conversion method considering continuity, for example, it is assumed that the subject is continuous in the real space if the distance information is within a certain range with respect to adjacent pixels on the input image. There is a method of interpolating between the obtained pixels on the virtual viewpoint using the decimal pixel positions on the input image after obtaining the corresponding pixels on the virtual viewpoint image. In addition, after generating a three-dimensional point, a polygon model may be constructed from continuity, and the corresponding model on the input image may be obtained for the pixel of the virtual viewpoint image by reprojecting the stereo model.

  In the description of the above formula (6), occlusion is not considered, but when a plurality of pixels of the input image correspond to the same pixel of the virtual viewpoint image, the value of the obtained distance information is used to calculate the virtual viewpoint. Processing that assumes that the correspondence with the pixels close to the camera is the correct correspondence is necessary. On the contrary, there are some pixels whose corresponding points cannot be obtained on the input image for the pixels of the virtual viewpoint image, but such pixels need to be distinguished as having no corresponding pixels.

As can be seen from the above equation (6), in this processing, in addition to the correspondence between the input image and the virtual viewpoint image (correspondence between (x, y) and (u, v)), the input viewpoint View The distance information D _{new, view} [u, v] in the virtual viewpoint estimated from the image of the camera (ie, the viewpoint of the camera View that captured the input image) is calculated at the same time. This value calculates the distance information of the virtual viewpoint. Is sent to the virtual viewpoint distance information generation unit 111 for use.

[Process of Step A8]
When the correspondence relationship between the input image and the virtual viewpoint image is calculated by the correspondence relationship setting unit 105, the composite image generation unit 106 then combines the input image and the corresponding relationship from the composite image (synthesis of the virtual viewpoint image). (Original image) is generated, and the generated composite image is stored in the composite image memory 107 [step A8].

  The processing here is performed by copying the pixel value of the input image to each pixel of the virtual viewpoint image using a given correspondence relationship. That is, a composite image that is an image at the virtual viewpoint estimated from the image information observed at the input viewpoint View is generated. For a pixel for which no corresponding pixel exists, information that there is no image information is copied.

When the correspondence is given in decimal pixel units, the input image is subjected to filter processing, the image information at the decimal pixel position of the input image is generated and copied, and finally obtained virtual The viewpoint image is of higher quality. In the following, a composite image generated using the input image View is represented as SYN _view [•]. In addition, by adding position information sandwiched between symbols [] to the composite image, the pixel value of the composite image in a specific region of the image is represented.

[Process of Step A9]
Subsequently, the composite boundary region setting unit 108 sets a composite boundary region on the composite image where the boundary region in the input image exists, generates composite boundary region information representing the position, and generates the generated composite boundary region information. Is stored in the composite boundary area memory 109 [step A9].

The process here is performed by copying the value of the boundary area information of the input image to each pixel of the virtual viewpoint image using the given correspondence. That is, information on the position where the boundary region observed at the input viewpoint View appears on the composite image is generated. Since only the value to be copied is different from the processing in step A8, it may be performed simultaneously. Hereinafter, the combined boundary area information generated using the boundary area information for the input image View is represented as SBOUND _view [•]. Further, the value of the combined boundary area information of a specific area of the image is represented by adding position information sandwiched between the symbols [] to the combined boundary area information.

[Summary of Step A1 to Step A11]
In this way, by executing the processing of Step A1 to Step A11, when the processing that is repeated for each image constituting the multi-viewpoint image is completed, the composite image at the numViews virtual viewpoints is stored in the composite image memory 107. The composite boundary area memory 109 stores numViews pieces of composite boundary area information corresponding to each composite image.

[Overview of Step A12 to Step A17]
Subsequently, using these pieces of information, the virtual viewpoint image generation unit 110 generates a virtual viewpoint image, and the virtual viewpoint distance information generation unit 111 generates virtual viewpoint distance information [Step A12 to Step A17]. .

  The process for generating the virtual viewpoint image is performed by repeating the process for generating the pixel value for each pixel of the virtual viewpoint image, and the process for generating the virtual viewpoint distance information is performed for each pixel of the virtual viewpoint image. This is done by repeating the process of generating. That is, if the index indicating each pixel of the virtual viewpoint image is represented by Pix and the total number of pixels constituting the virtual viewpoint image is represented by numPixs, Pix is initialized to 0 [Step A12], while adding 1 to Pix [ Step A16], and the next process [Step A13 to Step A15] is repeated until Pix reaches numPixs [Step A17].

[Process of Step A13]
In the process of repeating for each pixel of the virtual viewpoint image [Step A13 to Step A15], first, the virtual viewpoint image generation unit 110 determines a weight for each composite image [Step A13].

  In this embodiment, boundary area information is used in determining the weight. Various methods are conceivable. For example, as the simplest method, there is a method of determining by the following equation (7).

Here, W _view represents the weight for the composite image generated using the input image View (the input image indicated by the index View).

If the weight W _view determined by the equation (7) is specifically described, for example, a multi-viewpoint image is composed of five images, and SBOUND _view [Pix] of the pixel indicated by Pix is (0, 0, 0, 0, 0), sum_bound_info = 0, so all the weights W _view are equal and W _view = 1/5. On the other hand, for example, when SBOUND _view [Pix] of the pixel indicated by Pix is (0, 0, 1, 2, 3), sum_bound_info = 6, and therefore, each of the five weights W _view is W _view = 0, 0, 1/6, 2/6, 3/6.

In the above equation (7), the case where there is no corresponding point in the input image View in the virtual viewpoint image is not considered. Whether or not there is a corresponding point is grasped because a valid value is not assigned to an area where no corresponding point exists at the time of generating a composite image in step A8 or generating composite boundary area information in step A9. It is possible. When such combined boundary area information exists in Pix, the corresponding combined boundary area information is excluded from the calculation of the sum when calculating the above sum_bound_info, and W _view = 0 is set.

  The above equation (7) treats all input images equally when there are only pixels belonging to the boundary region, but if not, information from the input images belonging to the boundary region is It means to exclude and set the weight according to the distance from the boundary region. Note that the above equation (7) is merely an example, and instead of ignoring information from the input image belonging to the boundary region, a weight smaller than the weight for the input image not belonging to the boundary region is set. It doesn't matter.

  Also, a weighting term may be added in consideration of the similarity between the virtual viewpoint and the viewpoint of the input image that is the basis of the composite image. That is, the weight for the viewpoint whose distance and direction between the viewpoints are close to the virtual viewpoint may be increased. When this similarity is taken into consideration, the above equation (7) can be transformed into the following equation (8).

  Here, α and β are control parameter variables, S is a function that returns the similarity between two given viewpoints, and New represents a virtual viewpoint to be generated.

In this embodiment, distance information for the composite image is not taken into consideration, but D _{new, view} [u, v] generated by the above-described equation (6) in step A7 represents the distance information for the composite image. This value may be accumulated in the process of step A7 and weighted using this value.

For example, if the minimum value of D _{new, view} [Pix] for all Views (the value closest to the virtual viewpoint camera) is MinD, the difference between MinD and D _{new, view} [Pix] seems to exceed a certain threshold. If there is, there is a method of setting the weight for View to zero. In this case, since the distance information represents the context, when there is a shielding object, the object located in the background is not considered.

[Process of Step A14]
When the calculation of the weights is completed in this way, the virtual viewpoint image generation unit 110 generates a virtual viewpoint image in Pix [Step A14].

The process here is a weighted addition of a simple synthesized image SYN _view [Pix] using the weight W _view determined in step A13, and is expressed by the following equation (9).

  Here, Gen [•] represents a virtual viewpoint image, and represents the pixel value of a specific region of the virtual viewpoint image by adding position information sandwiched between symbols [] to the virtual viewpoint image.

[Process of Step A15]
Subsequently, the virtual viewpoint distance information generation unit 111 generates virtual viewpoint distance information in Pix [Step A15].

The processing here is also weighted addition of simple distance information D _{new, view} [Pix] using the weight W _view determined in step A13, and is expressed by the following equation (10).

In addition to the method of generating the virtual viewpoint distance information from the distance information input in this way, after the generation of the virtual viewpoint image is completed (between step A17 and step A18), the input image and the stereo matching method, etc. Alternatively, a method of generating virtual viewpoint distance information using the distance information generation method described above may be used. However, in that case, enormous operations may be required. For this reason, a hybrid technique may be used in which both are combined and a distance image is generated by the stereo matching method only for pixels with a large variance of D _{new, view} [Pix].

[Process of Step A18]
In this way, by executing the processing of Step A12 to Step A17, when the processing that is repeated for each pixel constituting the virtual viewpoint image is completed, the virtual viewpoint boundary region setting unit 112 subsequently generates the processing in Step A15. A virtual viewpoint boundary region is set using the virtual viewpoint distance information thus obtained [step A18].

  This process can use the same process as the process of setting the boundary area in the input image performed by the input information analysis unit 104 (the process of step A5: the process of setting the boundary area using distance information). is there. Although details of the processing in step A5 will be described later, in step A18, this processing is executed by replacing the index View for the input image with the virtual viewpoint New.

[Process of Step A19]
Subsequently, the virtual viewpoint image correcting unit 113 corrects the virtual viewpoint image using the virtual viewpoint boundary region information generated in Step A18 [Step A19].

In this process, a process of correcting the pixel value in the boundary region using the pixel value of the foreground object and the pixel value of the background object is performed. Various processing methods are conceivable, but the simplest method is to apply a low-pass filter in the boundary region. It may be applied only to the boundary region, or may be applied to the periphery of the boundary region. For example, there is a method in which the filter strength becomes stronger as the value of BOUND _new becomes smaller without the filter in the portion where the value of BOUND _new is MaxDistance (the maximum value of the boundary area information described above).

There is also a method in which the pixel value of the background object is estimated from the peripheral pixels, and the pixel value of the background object is mixed with the pixel value of the virtual viewpoint image using a mixing ratio according to the value of BOUND _new . There is a method of using the pixel value of a pixel having virtual viewpoint distance information indicating that it is the farthest from the camera among the neighboring pixels as the estimated pixel value of the background object, and the inter-pixel distance or virtual viewpoint distance from the pixel value of the peripheral pixel There is also a method of calculating the weighted average using information. In the latter method, for example, the weight may be determined using the following equation (11).

Here, q represents the position of the target pixel, and q _i represents the pixel position of the surrounding pixels. i is an index for identifying peripheral pixels, and weight _i represents a weight for the peripheral pixel of index i. σ ' _I and σ' _D are control parameters, and any numerical value can be used, but it is possible to generate image information with higher image quality by setting values that depend on the statistical properties of the image. .

When the virtual viewpoint distance information of the correction target pixel is smaller than the virtual viewpoint distance information of the surrounding pixels (closer to the camera), the mixture ratio corresponding to the value of BOUND _new indicates that the correction target pixel is a foreground object. Considering this, the larger the value of BOUND _new, the smaller the ratio of the pixel value of the background object corresponding to the fact that it enters the foreground object side. Conversely, the virtual viewpoint distance information of the correction target pixel Is larger than the virtual viewpoint distance information of the surrounding pixels (far from the camera), considering that the correction target pixel is a foreground object, the larger the value of BOUND _new , the closer to the foreground object side. The ratio of the pixel values of the background object is set so as to increase correspondingly.

  Further, in the virtual viewpoint image correction process, in addition to the correction in the boundary region, a process of generating an effective pixel value for a pixel for which an effective pixel value is not obtained may be performed. Since such an area is an area that cannot be observed due to occlusion from the input image, there is no foreground object. Therefore, the pixel value of the background object is estimated. For the specific processing, the above-described processing for estimating the pixel value of the background object in the boundary region can be used.

[Process of Step A20]
Finally, the virtual viewpoint image output unit 114 outputs the virtual viewpoint image that has been corrected by the virtual viewpoint image correction unit 113 [step A20].

[Details of processing in step A5]
Next, details of the boundary area information generation processing executed in step A5 will be described with reference to the flowchart of FIG.

In the process of generating the boundary area information, first, for every pixel position p of the image indicated by the index View, initialization is performed by substituting MaxDistance into the boundary area information BOUND _view [p] [step B1]. . Here, in the flowchart of FIG. 3, a set of pixel positions is represented as Image.

  Next, for each pixel, it is determined whether the pixel is included in the boundary region. In this embodiment, the pixel index is represented as idPix, and the number of pixels included in the image is represented as numPix. Therefore, after initializing idPix to 0 [Step B2], adding 1 to idPix [Step B6], and until idPix reaches numPix [Step B7], the following processing is repeated [Step B3 to Step B5]. .

In the boundary region determination process (step B3 to step B5) that is repeated for each pixel, after substituting the pixel position represented by idPix for p [step B3], the difference from the distance information of p is determined from a predetermined threshold TH_SAME_PLANE. It is checked whether or not there is at least one pixel adjacent to p having distance information that increases (step B4). If such an adjacent pixel exists, it is determined that p is a boundary region, and 0 is substituted into BOUND _view [p] [step B5]. Note that the adjacent pixels may target only four pixels that are vertically and horizontally adjacent, or may be eight pixels including diagonal directions.

  After the determination of the boundary region is completed, the distance from the boundary region is set as boundary region information for pixels that are not boundary regions.

  In the present embodiment example, pixels having a short distance from the boundary region are sequentially specified, and appropriate boundary region information is generated for pixels separated by a distance smaller than MaxDistance. Accordingly, after initializing the target distance value distance to 1 [Step B8], until the distance becomes MaxDistance [Step B9], while adding 1 to the distance [Step B18], the following processing is repeated [ Step B10 to Step B17].

  In the process of setting the boundary area information of a pixel whose distance from the boundary area is distance, it is determined for each pixel whether the distance from the boundary area of the pixel is distance. Therefore, after initializing the pixel index idPix with 0 [Step B10], while adding 1 to idPix [Step B16], until idPix becomes numPix [Step B17], the following processing is repeated [Step B11 to Step B11] B15].

In the distance determination process repeated for each pixel, after substituting the pixel position represented by idPix for p [Step B11], the boundary area information of p is smaller than the distance being determined [Step B12]. Then, the determination for the pixel is finished. Otherwise, the determination process is continued. That is, the minimum value of BOUND _view of the pixel adjacent to p is substituted into min_distance [Step B13], and if min_distance is smaller than MaxDistance [Step B14], min_distance + 1 is substituted into BOUND _view [p] [Step B15]. In addition, the adjacent pixels in this process may be targeted only for four pixels that are vertically and horizontally adjacent, or may be targeted for 8 pixels including an oblique direction.

  This flowchart is one example for setting the boundary area information, and information indicating the boundary area for each subject can be set by using another method.

  In this way, according to the present embodiment, the light ray information acquired in the boundary region of the subject is treated as a light beam with low intensity or a light beam with low reliability, thereby reducing the virtual viewpoint image with high subjective quality. Will be able to produce in quantity.

[2] Second Embodiment In the first embodiment described above, the input information analysis unit 104 corrects the image and distance information input in the boundary region in step A6 of the flowchart of FIG. Although processing has been performed, this correction processing is not necessarily executed.

  That is, in the first embodiment described above, when a virtual viewpoint image is generated by combining light rays sampled by a multi-viewpoint image (as described above, a combined image is generated and combined), When a boundary area where light information of a plurality of subjects is mixed is determined and a pixel of a virtual viewpoint image belonging to the boundary area is determined to belong to a pixel of a multi-viewpoint image that is associated with the pixel (which belongs to the boundary area) Of the light sampled by the boundary region as a light beam having a lower intensity than the light sampled by the non-boundary region. By performing the synthesis, the influence of mixing light ray information from a plurality of subjects is reduced.

  In this way, since the light beam sampled in the boundary region is treated as a light beam having a lower intensity than the light beam sampled in the non-boundary region, the light beam is synthesized, so that the image and distance executed in step A6 of the flowchart of FIG. Even if information correction processing is not performed, it is possible to reduce the influence caused by mixing light ray information from a plurality of subjects.

  On the other hand, when the image and distance information correction processing executed in step A6 of the flowchart of FIG. 2 is performed, the pixels of the multi-viewpoint image associated with the same pixel of the virtual viewpoint image are corrected. If you sample the foreground object rays, it is more likely that all those pixels will sample the foreground object rays, and conversely if those pixels sample the foreground object rays. Is more likely that all those pixels will sample the rays of the foreground object.

  From now on, when the image and distance information correction processing executed in step A6 of the flowchart of FIG. 2 is performed, when the virtual viewpoint image is generated by synthesizing the light rays sampled by the multi-viewpoint image, those light rays are used. Even if it is made to synthesize | combine with the same weight about, it can reduce the influence by the light ray information from several subjects being mixed.

  Here, this concept of correction processing is not only applicable when generating a virtual viewpoint image from a multi-viewpoint image, but also applied when generating a virtual viewpoint image from a subject image taken by one camera. It can be done.

  In the case of adopting this configuration (second embodiment), the virtual viewpoint image generation device 100 of the present invention executes the flowchart of FIG. 4 instead of executing the flowchart of FIG.

  That is, when realizing the second embodiment, the virtual viewpoint image generation device 100 according to the present invention is first similar to step A1 in step C1 as shown in the flowchart of FIG. By executing the process, information on the virtual viewpoint is input, and the index View is initialized to 0 in the subsequent step C2.

  Subsequently, in step C3, an image is input by executing the same processing as in step A3 described above. Subsequently, in step C4, distance information is input by executing the same processing as in step A4 described above. Subsequently, in step C5, the boundary region of the object is set by executing the same processing as in step A5 described above. Subsequently, in step C6, the same process as in step A6 described above is executed to correct the image and distance information input in the boundary region. Subsequently, in step C7, the same processing as in step A7 described above is executed to calculate the correspondence relationship between the pixels of the virtual viewpoint image and the pixels of the input image. Subsequently, in step C8, a composite image is generated by executing the same process as in step A8 described above. Subsequently, in step C9, a combined boundary region on the combined image is set by executing the same processing as in step A9 described above.

  Subsequently, in step C10, the value of the index View is incremented by one. In the subsequent step C11, it is determined whether or not the value of the index View has reached numViews, and it is determined that the value has not reached numViews. Returning to the process of step C3, when it is determined that numViews has been reached, the process proceeds to step C12, where the index Pix is initialized to zero.

  Subsequently, in step C13, the weights of the respective synthesized images are made the same, and the average value of the pixel values of the synthesized image generated in step C8 is obtained for the pixel indicated by the index Pix, and a virtual viewpoint image is generated by copying the average value. To do.

  Subsequently, in step C14, the value of the index Pix is incremented by one. In the subsequent step C15, it is determined whether or not the value of the index Pix has reached numPixs, and it is determined that the value has not reached numPixs. The process returns to step C13.

  On the other hand, when it is determined that the value of the index Pix has reached numPixs, the process proceeds to step C16, the weight of each composite boundary area information is made the same, and the average value of the composite boundary area information generated in step C9 is obtained. Is copied to set a virtual viewpoint boundary region corresponding to the virtual viewpoint boundary region set in step A18.

  Subsequently, in step C17, the virtual viewpoint image is corrected using the virtual viewpoint boundary region set in step C16 by executing the same processing as in step A19, and in step C18, By executing the same processing, the virtual viewpoint image that has been corrected is output.

  Thus, also in the second embodiment, a virtual viewpoint image with high subjective quality can be generated with a small amount of calculation.

  Here, when realizing the second embodiment, it is possible to adopt a configuration in which the flowchart of FIG. 5 is executed instead of the flowchart of FIG.

  That is, in the flowchart of FIG. 5, the process of step C9 of the flowchart of FIG. Step C100 for generating viewpoint distance information is provided. Further, instead of Step C16 in the flowchart of FIG. 4, the virtual viewpoint boundary area is set using the virtual viewpoint distance information generated in Step C100, and the same as Step A18 described above. Step C200 for executing the processing is provided, but the second embodiment can also be realized by executing the flowchart of FIG.

  The processing described above can also be realized by a computer and a software program, and the program can be provided by being recorded on a computer-readable recording medium or can be provided through a network.

  In the above embodiment, the virtual viewpoint image generation device has been described mainly. However, the virtual viewpoint image generation method of the present invention can be realized by steps corresponding to the operations of the respective units of these virtual viewpoint image generation devices.

  As mentioned above, although embodiment of this invention has been described with reference to drawings, the said embodiment is only the illustration of this invention, and it is clear that this invention is not limited to the said embodiment. . Accordingly, additions, omissions, substitutions, and other modifications of components may be made without departing from the spirit and scope of the present invention.

It is an apparatus block diagram of the virtual viewpoint image generation apparatus of this invention. It is a flowchart which the virtual viewpoint image generation apparatus of this invention performs. It is a flowchart of the production | generation process of boundary area information. It is a flowchart which the virtual viewpoint image generation apparatus of this invention performs. It is a flowchart which the virtual viewpoint image generation apparatus of this invention performs. It is a figure explaining a virtual viewpoint.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Virtual viewpoint image generation apparatus 101 Virtual viewpoint input part 102 Image input part 103 Distance information input part 104 Input information analysis part 105 Correspondence setting part 106 Composite image generation part 107 Composite image memory 108 Composite boundary area setting part 109 Synthetic boundary area memory DESCRIPTION OF SYMBOLS 110 Virtual viewpoint image generation part 111 Virtual viewpoint distance information generation part 112 Virtual viewpoint boundary area setting part 113 Virtual viewpoint image correction part 114 Virtual viewpoint image output part

Claims (8)

From a multi-viewpoint image consisting of n subject images taken from a plurality of cameras with different viewpoints, to a certain point on the subject appearing in each pixel of the subject image from the camera that took the multi-viewpoint images , A virtual viewpoint image generation method for generating a virtual viewpoint image that is an image of a subject viewed from a virtual viewpoint designated in advance using distance information corresponding to each of the n subject images ,
A boundary region setting step for setting a boundary region of a subject for each of the subject images from the subject image and its distance information for each of the n subject images ;
A correspondence determining step of determining , for each of the n subject images , a correspondence relationship between coordinates on each subject image and coordinates on the virtual viewpoint image using distance information corresponding to each subject image ; ,
A composite image generation step for generating a composite subject image from the subject image using the determined correspondence for each of the n subject images;
Using pre Symbol determined relationship, from said information of the boundary area set for each object image, a synthesis boundary region setting step of setting a synthesis boundary region with respect to each synthetic subject image,
The n pieces of the synthetic subject image, or the weighted addition in accordance with the information of the synthesis boundary area set for each synthetic subject image, and wherein the synthesis boundary area of the information set for each synthesis object image A virtual viewpoint image generation step of generating the virtual viewpoint image by weighting and adding according to the virtual viewpoint information,
A featured virtual viewpoint image generation method. The virtual viewpoint image generation method according to claim 1,
In the virtual viewpoint image generation step, if any one of the synthesized subject images that are not the synthesized boundary region exists in each pixel of the virtual viewpoint image, weighted addition is performed using only those synthesized subject images. The
A featured virtual viewpoint image generation method. The virtual viewpoint image generation method according to claim 1 or 2,
A multi-viewpoint image correcting step for correcting the subject image in the boundary region;
In the composite image generation step, generating the composite subject image using the corrected subject image,
A featured virtual viewpoint image generation method. The virtual viewpoint image generation method according to any one of claims 1 to 3,
A distance information correction step for correcting the distance information in the boundary region;
In the correspondence determination step, the correspondence is determined using the corrected distance information.
A featured virtual viewpoint image generation method. The virtual viewpoint image generation method according to any one of claims 1 to 4,
A virtual viewpoint distance information generating step of generating virtual viewpoint distance information that is distance information of the virtual viewpoint image from distance information corresponding to each of the n subject images ;
A virtual viewpoint boundary region setting step for setting a virtual viewpoint boundary region that is a boundary region of a subject with respect to the virtual viewpoint image from the virtual viewpoint distance information and the virtual viewpoint image;
A virtual viewpoint image correction step of changing a pixel value of each pixel in the virtual viewpoint boundary region of the virtual viewpoint image according to information of surrounding pixels,
A featured virtual viewpoint image generation method. From a multi-viewpoint image consisting of n subject images taken from a plurality of cameras with different viewpoints, to a certain point on the subject appearing in each pixel of the subject image from the camera that took the multi-viewpoint images , A virtual viewpoint image generation device that generates a virtual viewpoint image that is an image of a subject viewed from a virtual viewpoint specified in advance using distance information corresponding to each of the n subject images ,
Boundary region setting means for setting a boundary region of a subject for each subject image from the subject image and distance information thereof for each of the n subject images ;
Correspondence determining means for determining , for each of the n subject images , the correspondence between the coordinates on the subject images and the coordinates on the virtual viewpoint image using distance information corresponding to the subject images ; ,
Composite image generation means for generating a composite subject image from the subject image using the determined correspondence for each of the n subject images;
Using pre Symbol determined relationship, from said information of the boundary area set for each object image, a synthesis boundary area setting means for setting a synthesis boundary region with respect to each synthetic subject image,
The n pieces of the synthetic subject image, or the weighted addition in accordance with the information of the synthesis boundary area set for each synthetic subject image, and wherein the synthesis boundary area of the information set for each synthesis object image A virtual viewpoint image generation means for generating the virtual viewpoint image by weighting and adding according to the virtual viewpoint information;
A featured virtual viewpoint image generation device. A virtual viewpoint image generation program for causing a computer to execute the virtual viewpoint image generation method according to any one of claims 1 to 5 . Virtual viewpoint image generation program computer-readable recording medium recording a for executing a virtual viewpoint image generation method according to the computer in any one of claims 1 to 5.

Images