JP4993355B2 - Occlusion processing method for free-viewpoint video generation with local region division - Google Patents

Description

The present invention relates to an occlusion processing method in local area division free viewpoint video generation, and in particular, when the surface of a subject is on the boundary line between two adjacent local areas, the generated free viewpoint video has a mesh shape. The present invention relates to an occlusion processing method for preventing noise.

  In recent years, with the development of image processing technology and image communication technology, three-dimensional free viewpoint images have attracted attention as next-generation image content. Therefore, a technique for generating an all-around free viewpoint image using a multi-viewpoint image captured by arranging a video camera around the subject has been researched and developed.

  In this case, if cameras are densely arranged around the subject so as to correspond to an arbitrary viewpoint, the number of cameras increases, resulting in high costs, which is not practical. Therefore, the cameras are arranged sparsely around the subject, but if this is done, images between the video cameras cannot be obtained.

  In order to solve this problem, conventionally, a method has been proposed in which an image from a viewpoint that is not photographed by a camera is generated by interpolating an image between images using image-based rendering.

  As a typical image-based rendering method for performing interpolation between multi-viewpoint images, there is a “light space method”, and the following patent document is a technical document for explaining generation of an interpolated image using the light space method. There are those shown in 1 and 2.

  However, when the subject is distributed over a wide range in the subject space and the parallax due to the difference in depth (depth parallax) is large, the interpolation processing of the light ray information becomes complicated when trying to correct the depth parallax by the conventional technique. There was a problem that it was not practical.

  Therefore, the present inventors divide the subject space into small local regions in which depth parallax can be ignored, and invent an invention that performs interpolation by applying an image-based rendering method to each local region individually, A patent application was filed (Japanese Patent Application No. 2006-185648). According to the present invention, even if the subject is distributed over a wide range in the subject space and the parallax due to the difference in depth is large, the object in the local region is approximated by a plane passing through the central axis of the local region. Therefore, the interpolation process of the ray information is facilitated, and an image of an arbitrary viewpoint on the subject space close to reality can be generated.

For example, when there is a real viewpoint image as shown in FIG. 8A, it is divided into a large number of local regions. Then, pay attention to a person in one local area as shown in FIG. Assuming that this person is photographed by a plurality of video cameras arranged sparsely around the person, an image as shown in FIG. 6C obtained by each camera is obtained. When these images are processed using, for example, image-based rendering, the ray information of a person's waist line, for example, becomes as shown in FIG. It becomes a spiral as shown in FIG. Although not shown, the ray information of each part from the person's head to the toe is also obtained as shown in Fig. (D), and the ray space composed of the ray information of each part is interpolated in the same figure ( It becomes like f). This figure (f) is the figure which interpolated the light space which consists of light information, such as a head, a shoulder, a thigh, a leg, etc. of a person. Therefore, in order to obtain an image viewed from an arbitrary angle, the ray information of the angle is cut out and synthesized from FIGS. (E) and (f) to obtain an arbitrary viewpoint image of the entire person in FIG. If this is returned to the original image position, the virtual viewpoint image of FIG.
JP 2004-258775 A JP-A-10-111951

  In the above invention, in the process of integrating the free viewpoint images of each local area, the local area far from the virtual viewpoint position selected by the user is drawn first, and the near local area is overwritten later. is doing. For this reason, as shown in FIG. 9, occlusion of the subject 51 by the local region 61 in front is generated in the subject 50 drawn from the local region 60 in the back as viewed from the camera position. . In such normal occlusion, the occlusion relation between local regions is obtained, the occlusion part is obtained for each multi-viewpoint image, the ray information in the occlusion part is erased, and the interpolation of the ray information is performed, thereby removing the occlusion part Since it can be restored to a normal image, no particular problem occurs.

  However, for example, as shown in FIG. 10, when the surface 53 of the subject 52 is on the boundary line between two adjacent local regions 62 and 63, drawing is performed from the local region 62 at the back as viewed from the virtual viewpoint. The subject surface thus formed is regarded as occlusion by the local region 63 in the foreground, and is overwritten with the surface of the subject drawn from the local region 63 in the foreground. That is, an erroneous determination that the subject 52 is hidden by a part 53 of the subject is generated. Then, there is a positional shift between the subject surfaces drawn from the local regions 62 and 63 because the object in the local region is approximated by a plane. When the range drawn from the local region 63 in front is smaller than the previous range, there is a problem that mesh-like noise is generated in the generated free viewpoint video.

The present invention has been made to solve the above-described problems of the prior art, and the object is to be generated even when the surface of the subject is on the boundary line between two adjacent local regions. An object of the present invention is to provide an occlusion processing method in local area division type free viewpoint video generation that prevents or reduces the mesh noise in a free viewpoint image.

In order to achieve the above-described object, the present invention divides a subject space into small local regions where depth parallax can be ignored, and performs local region interpolation to generate a free-viewpoint video by interpolating each local region. This is an occlusion processing method for video generation. When the subject space is divided into a plurality of local areas, the local areas are arranged so as not to overlap each other, and free viewpoint images are generated in the respective local areas. It is characterized in that it is synthesized with.

  Also, when the subject space is divided into a plurality of local areas, images from free viewpoints are interpolated in each local area, and when they are combined, an occlusion area image cannot be generated in a single local area Another feature is that the image is overwritten with an image of another local area.

  According to the present invention, it is possible to prevent mesh-like noise from being generated in the generated free viewpoint video, and to generate a highly accurate free viewpoint video.

  Hereinafter, the present invention will be described in detail with reference to the drawings. First, an outline of a method for generating a free viewpoint image by local region division, which is a premise of the present invention, will be described. Details are described in the invention of the present application (Japanese Patent Application No. 2006-185648).

  FIG. 1 is a conceptual diagram showing a positional relationship between a subject and a video camera (hereinafter simply referred to as a camera) used in the method for generating a free viewpoint image by local region division.

  As shown in the drawing, a plurality of cameras 3a, 3b whose optical axes are horizontal on the outer periphery (for example, on the outer periphery of a polygon such as a circle, an ellipse, a quadrangle, etc.) of the subject space 1 surrounding the subject that is the subject 2 to be photographed. 3c,..., 3n are arranged at sparse intervals. It is assumed that the subject 2 has a depth parallax that cannot be ignored. The present invention combines images captured by the cameras 3a, 3b, 3c,..., 3n to obtain a free viewpoint image viewed from a virtual viewpoint (eg, 3X) where no camera is placed. Is.

  Next, an outline of a free viewpoint image generation method according to an embodiment of the present invention will be described with reference to the block diagram of FIG.

  In step S1, multi-viewpoint images are taken by the plurality of cameras 3a, 3b, 3c,. In step S2, voxel data is acquired from the multi-viewpoint image. A voxel means a section obtained by dividing a three-dimensional space in a lattice pattern. In step S3, the subject space is divided into local regions. For example, as shown in FIG. 3 and FIG. 4, it is divided into cylindrical local regions 4 a, 4 b, 4 c,. In step S4, an occlusion relationship between the local regions is obtained. Next, in step S5, an occlusion portion is obtained for each multi-viewpoint image.

  On the other hand, in step S6, the light ray information is stored in the light ray space in each local region (see FIGS. 8C and 8D). In step S7, the ray information of the occlusion portion obtained in step S5 is deleted. In step S8, the interpolation of the ray information and the filling of the ray space are performed (see FIGS. 8E and 8F). Next, the process proceeds to step S9, where the ray information of the virtual viewpoint is cut out from the ray space, and a local free viewpoint image is generated (see FIG. 8 (g)). Note that the position information of the virtual viewpoint is given by the user.

  In step S10, the free viewpoint image in the back local area as viewed from the virtual viewpoint is overwritten with the free viewpoint image in the local area in front. In step S11, the entire free viewpoint image is generated. In step S12, the free viewpoint image obtained by the above process is output. The steps S2, S3, S4, S5, S10, and S11 are processes repeatedly performed for each frame, and the steps S6, S7, S8, and S9 are performed for constructing a ray space for each local region. Process. Thin lines represent data exchange.

  The processes in steps S8 to S11 are performed on the ray information obtained by erasing the ray information of the occlusion portion in step S7. Therefore, as shown in FIG. The normal occlusion that the subject 50 drawn from the local region 60 is hidden is eliminated, but the surface 53 of the subject 52 is on the boundary line between two adjacent local regions 62 and 63 as shown in FIG. Yes, when the subject surface drawn from the local region 62 at the back as viewed from the virtual viewpoint is regarded as occlusion by the local region 63 in the foreground, that is, the subject 52 is hidden by a part 53 of the subject. When an erroneous determination occurs, there arises a problem that mesh-like noise of different colors occurs in the generated free viewpoint video.

  Therefore, a first embodiment that addresses the occlusion problem will be described below.

  In the present embodiment, when the subject space is divided into local areas in the process of dividing into local areas in step S3 (see FIG. 2), as shown in FIG. 5, the local areas 4a, 4b, 4c,. .. Make sure that the surface of the subject does not appear on the boundary of the local area. However, when viewed from either the shooting viewpoint or the virtual viewpoint (for example, a, b, etc.), all the pixels of the image (virtual image) when viewed from the captured image or the virtual viewpoint are any one or more. The interval and the size of the local area are adjusted so that the local area is projected. For example, when the cylindrical local region is arranged in a honeycomb shape, the radius of the cylinder is larger than √3 / 4 (≈0.433) times the distance R between the central axes of adjacent cylinders, and more than 1/2 times. Small is enough.

  If the local area is divided as described above, the surface of the subject 52 is within one local area 4b as shown in FIG. 6, and is not positioned on the boundary with the other local area 4a. Therefore, it is not erroneously determined that the subject 52 is hidden by a part of the subject. That is, there is no concern that the subject surface drawn from the deep local area viewed from the virtual viewpoint is considered as occlusion by the local area in front.

  After the local region is divided as described above, the processing in step S4 and subsequent steps in FIG. 2 is performed, but description of this processing is omitted. In this embodiment, since the interval and size of the local area are adjusted as described above, even if the surface of the subject exists in the gap between the local areas, the pixel information is projected onto any of the local areas. Will not be missing.

  Next, a second embodiment of the present invention will be described.

  In this embodiment, first, a local region is set only in a region where a subject exists in the subject space. For example, for each subject, a cylindrical local region including the subject is set with the vertical line passing through the center of gravity as the central axis. Therefore, in each local area, the pixel information of the subject included in the local area is projected onto the local area.

  According to the present embodiment, since the subject is included in the cylindrical local region, the surface of the subject 52 is within one local region and is not positioned on the boundary of the local region.

  Next, a third embodiment of the present invention will be described. In this embodiment, the subject space is divided into a plurality of local regions, and an image from a free viewpoint is generated by interpolation in each local region, and an image of the occlusion region is generated in a single local region when they are combined. Only when it is not possible, overwrite the image with another local area.

  First, the subject space is divided into local areas arranged in close contact with each other without a gap. Next, pixel information of the actual captured image is processed in each local region. Since the local area is arranged in close contact with no gap, the pixel information is projected onto any of the local areas.

  Subsequently, the occlusion portion in each local region is grasped from the positional relationship between the local regions, and the pixel information of the portion determined to be occlusion is deleted. However, it is determined whether an image of the occlusion area can be generated in a single local area. If it can be generated, these pieces of pixel information are not erased. Then, the pixel information projected on the local area is interpolated.

  According to this embodiment, since it is determined that the subject 52 as shown in FIG. 10 can be generated in one local region 62, the surface 53 of the subject 52 is not regarded as occlusion, and the step S7 (see FIG. 2). Thus, the surface 53 is not erased. Therefore, the problems of the prior art are solved.

  Next, a fourth embodiment of the present invention will be described.

  In this embodiment, in the above third embodiment, even when an image of the occlusion area cannot be generated in a single local area, pixels with a ratio exceeding a predetermined threshold (for example, 95%) can be generated. Alternatively, when a volume of a specific subject that exceeds a predetermined threshold (for example, 80%) belongs to the local region, as shown in FIG. The overwriting by the local areas 11 to 16 adjacent to the area 10 is not performed.

  According to this embodiment, for example, the surface 53 in FIG. 10 is not overwritten by an adjacent local region, so that no mesh noise is generated on the surface.

  It should be noted that an image may be preferentially generated by interpolation from a local area having a large area occupied by the object among objects and spaces in the subject space.

It is a conceptual diagram of the shooting site of a multi-view video in which cameras are arranged at sparse intervals. It is a flowchart which shows the process sequence of the free viewpoint image generation method to which this invention is applied. It is explanatory drawing of an example which divides | segments to-be-photographed object space into a local area | region. It is a figure at the time of seeing the divided | segmented local area | region from the top. It is explanatory drawing of the local region division | segmentation of 1st Embodiment of this invention. It is a figure explaining the surface of a to-be-photographed object being settled in one local region, and not being located on the boundary with another local region. It is explanatory drawing of the occlusion process of 3rd, 4th embodiment of this invention. It is explanatory drawing of the outline of the free viewpoint image generation method of application invention. It is a figure explaining generation | occurrence | production of the occlusion in a local area | region division. It is a figure explaining the subject of a prior art.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Subject space, 3a, 3b, 3c ... Video camera, 3X ... Virtual viewpoint position, 4a, 4b, 4c, ..., 5, 10-16, 60-63 ... Local region , 50, 51, 52... Subject, 53.

Claims (7)

A occlusion processing method in a local region division type free viewpoint video generation dividing the object space into small local region negligible depth parallax to generate a free viewpoint video performs interpolation for each local region,
Means for shooting multi-viewpoint images;
Means for dividing the subject space into a plurality of local regions so that each local region does not overlap ;
Obtaining an occlusion part for each of the multi-viewpoint images, and erasing the occlusion part;
Means for generating a free viewpoint image in each local region;
Means for synthesizing the free viewpoint image in the whole space ;
An occlusion processing method characterized by comprising: In the occlusion processing method according to claim 1,
An occlusion processing system characterized in that local regions are arranged so that all pixels on an actual captured image are projected onto any one or more local regions. In the occlusion processing method according to claim 1,
An occlusion processing method characterized in that each local region is arranged in accordance with the position of the subject. A occlusion processing method in a local region division type free viewpoint video generation dividing the object space into small local region negligible depth parallax to generate a free viewpoint video performs interpolation for each local region,
Means for shooting multi-viewpoint images;
Means for dividing the subject space into a plurality of local areas so that the local areas are closely contacted and the local areas do not overlap ;
Obtaining an occlusion part for each of the multi-viewpoint images, and erasing the occlusion part;
Means for generating an interpolation image from a free viewpoint in each local region ;
Means for overwriting an image of an occlusion area with an image of another local area only when the image of the occlusion area cannot be generated in a single local area when the interpolated image is synthesized ;
An occlusion processing method characterized by comprising: In the occlusion processing method according to claim 4,
When the ratio of the area of the part that is interpolated using the light ray information of the other part in the area of the part determined to be occlusion from the virtual viewpoint is larger than a predetermined threshold Is an occlusion processing method characterized by not performing overwriting by a local region adjacent to the local region. In the occlusion processing method according to claim 4,
An occlusion processing method characterized in that an image is preferentially interpolated and generated from a local area having a large area occupied by an object among objects and spaces in the subject space. In the occlusion processing method according to any one of claims 1 to 6,
An occlusion processing method characterized by generating a ray space in each local region and interpolating and generating a free viewpoint image thereon.