KR100989435B1 - Method and apparatus for processing multi-viewpoint image - Google Patents

Abstract

Disclosed are a multiview image processing method and apparatus. Multi-view image processing method according to the present invention comprises the steps of obtaining a multi-view depth map by three-dimensional warping the depth map obtained from the depth camera to the viewpoint of the multi-view camera; Generating a multiview trimap corresponding to a multiview image obtained from the multiview camera using the multiview depth map; Generating a multiview alphamat corresponding to the multiview image using the multiview trimap; And extracting a multiview foreground by applying the multiview alphamat to the multiview image. According to the present invention, it is possible to automatically perform high quality image matting on a multiview image.

Foreground separation, image matting, multiview image, image synthesis

Description

Method and apparatus for multiview image processing {Method and apparatus for processing multi-viewpoint image}

The present invention relates to a multiview image processing method and apparatus, and more particularly, to a multiview image processing method and apparatus capable of extracting a multiview foreground using a depth map obtained from a depth camera and a multiview image obtained from a multiview camera. It is about.

With the development of multimedia technology, interest in realistic media has recently increased. Among them, a multi-view image means a set of images acquired by using two or more cameras (multi-view cameras) having different viewpoints. Unlike conventional single-view video, multi-view video can generate a 3D dynamic image by photographing a scene through multiple cameras. Such a multi-view image may provide a user with a free view and a 3D effect through a wide screen. Recently, a variety of multi-view video applications have been studied, and free-view TV (FTV), 3-D TV, surveillance, and immersive teleconference are being actively researched.

On the other hand, the film industry and broadcasting stations require image synthesis technology for special effects. In particular, it is no exaggeration to say that it is essential to secure excellent image synthesis technology these days when blockbusters with advanced three-dimensional computer graphics technology are pouring in.

In general, an image may be divided into a foreground object and a background. Extracting the foreground object from the background is called a foreground matte and removes the background from the original image in consideration of an alpha value representing pixel opacity of the image. This technique is called image matting. Image synthesis, on the other hand, means combining the foreground matt with an arbitrary background using an alpha value.

In order to extract a foreground from a multiview image and synthesize a background of another image, the foreground must be extracted for each image of each viewpoint. In the past, a manual matting method in which a user or an editor directly intervenes to perform image matting for each image of each viewpoint is used. That is, when performing multiview image mating consisting of N images, N image matting should be performed daily. Therefore, as the number of images to be processed like a video sequence increases, more time and effort are required, and thus, a method for automatically performing high quality image matting is required.

SUMMARY OF THE INVENTION The present invention has been made in an effort to provide a multiview image processing method and apparatus capable of automatically performing high quality image matting on a multiview image.

In order to solve the above technical problem, the multi-view image processing method according to the present invention comprises the steps of: (a) obtaining a multi-view depth map by three-dimensional warping the depth map obtained from the depth camera to the viewpoint of the multi-view camera; generating a multiview trimap corresponding to a multiview image obtained from the multiview camera using the multiview depth map; (c) generating a multi-view alphamat corresponding to the multi-view image using the multi-view trimap; And (d) extracting a multiview foreground by applying the multiview alphamat to the multiview image.

In generating the trimaps constituting the multi-view trimap in the step (b), each image constituting the multi-view image and each three-dimensional warped depth map constituting the multi-view depth map, (b1) determining a foreground area in the three-dimensional warped depth map and filling a hole generated by the three-dimensional warping in the foreground area; And (b2) generating the trimap from the depth map filled with the holes.

The step (b1) may include: (b11) generating a segmented image by performing color segmentation on the image; (b12) determining, as the foreground region, a set of segments among the segments of the segmented image having a depth value in the three-dimensional warped depth map more than a predetermined ratio; And (b13) filling a hole generated due to the three-dimensional warping in the foreground area of the three-dimensional warped depth map.

In the step (b13), the hole may be filled by setting an average of depth values of available pixels among the pixels around the hole as the depth value of the hole.

In the step (b2), the trimap may be generated by applying an erosion and expansion operation to the depth map filled with the holes.

In the step (a), the three-dimensional warping may be performed using camera parameters of the depth camera and camera parameters of each of the cameras constituting the multi-view camera.

The multi-view image processing method may further include synthesizing the extracted multi-view foreground with a multi-view background to generate a multi-view composite image.

In order to solve the above technical problem, a multi-view image processing apparatus according to the present invention includes: a three-dimensional warping unit which obtains a multi-view depth map by three-dimensional warping a depth map obtained from a depth camera to a viewpoint of a multi-view camera; A trimap generation unit generating a multiview trimap corresponding to a multiview image obtained from the multiview camera using the multiview depth map; An alpha matte generator configured to generate a multiview alphamat corresponding to the multiview image using the multiview trimap; And a foreground extracting unit extracting a multi-view foreground by applying the multi-view alphamat to the multi-view image.

The trimap generator generates each trimap constituting the multiview trimap, each image constituting the multiview image and each three-dimensional warped depth map constituting the multiview depth map. The foreground region may be determined from the 3D warped depth map, the hole generated due to the 3D warping may be filled in the foreground region, and the trimap may be generated from the depth map filled with the hole.

The trimap generator may include a segment unit configured to generate a segmented image by performing color segmentation on the image; A foreground region determiner configured to determine, as the foreground region, a set of segments in which a depth value in the 3D warped depth map is greater than a predetermined ratio among the segments of the segmented image; And a hole removing unit filling a hole generated by the three-dimensional warping in the foreground area of the three-dimensional warped depth map. And a trimap generator for generating the trimap from the depth map filled with the holes.

The hole removing unit may fill the hole by setting an average of depth values of available pixels among the pixel values around the hole as the depth value of the hole.

The trimap generator may generate the trimap by applying an erosion and expansion operation to the depth map filled with the holes.

The three-dimensional warping unit may perform the three-dimensional warping by using camera parameters of the depth camera and camera parameters of cameras constituting the multi-view camera.

The multi-view image processing apparatus may further include a background synthesizer configured to generate the multi-view composite image by synthesizing the extracted multi-view foreground with a multi-view background.

In order to solve the above technical problem, there is provided a computer-readable recording medium having recorded thereon a program for executing a multi-view image processing method according to the present invention.

According to the present invention described above, a multi-view depth map is obtained by three-dimensional warping a depth map obtained from a depth camera to a viewpoint of a multi-view camera, and a multi-view corresponding to a multi-view image obtained from a multi-view camera using a multi-view depth map. After generating a trimap, a multiview alphamat corresponding to a multiview image may be generated using a multiview trimap to automatically perform high quality image matting on the multiview image.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description and the accompanying drawings, substantially the same components are denoted by the same reference numerals, and redundant description will be omitted. In addition, in the following description of the present invention, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted.

1 is a diagram illustrating a configuration of a multiview image processing apparatus according to an embodiment of the present invention. The multi-view image processing apparatus according to the present embodiment includes a multi-view camera 110 and a depth camera 120, a three-dimensional warping unit 130, a trimap generator 140, an alphamat generator 150, and a foreground. It comprises an extraction unit 160.

The multi-view camera 110 is composed of a plurality of cameras (first to n-th camera) as shown. The viewpoints of the plurality of cameras are different from each other according to the position of the camera, and thus a plurality of images having different viewpoints are bundled together to be called multiview images. The multi-view image obtained from the multi-view camera 110 includes color information for each pixel on the 2D image constituting each image, but does not include depth information on the 3D image.

The depth camera 120 obtains a depth map having depth information on a 3D image. The depth camera 120 is a device capable of illuminating a laser or an infrared ray to an object or a target area, obtaining a return ray, and obtaining depth information in real time. The depth camera 120 includes a depth sensor for sensing depth information through a laser or infrared ray.

FIG. 2 is a three-dimensional view of an example in which a multiview camera 110 and a depth camera 120 are disposed. As illustrated, the multiview camera 110 is configured in a form in which cameras are arranged in a line. The depth camera 120 may be placed at the bottom (or top) of any one of them (eg, the center camera). The arrangement of the multiview camera 110 and the arrangement of the depth camera 120 may be modified according to each viewpoint of the multiview image to be obtained.

3 shows an example of a multiview image obtained from the multiview camera 110 and one depth map obtained from the depth camera 120.

The depth map obtained from the depth camera 120 is input to the 3D warping unit 130, and the 3D warping unit 130 3D warps the depth map to the viewpoint of each camera forming the multiview camera 110. We obtain a multiview depth map consisting of n three-dimensional warped depth maps. Here, the 3D warping of the depth map uses camera parameters of each camera constituting the multiview camera 110 and camera parameters of the depth camera 120.

4 is a reference diagram for explaining a process of three-dimensional warping a depth map obtained from the depth camera 120 to a viewpoint of an arbitrary camera of the multiview camera 110.

Cameras generally have internal and external parameters that are unique to the camera. Internal parameters refer to the focal length and image center coordinates of the camera, and external parameters refer to their translation and rotation relative to the world coordinate system. The base matrix P of the camera according to the internal and external parameters is obtained by the following equation.

Here, the first matrix on the right side is a matrix consisting of internal parameters, and the second matrix is a matrix consisting of external parameters.

)에 깊이 카메라(120)의 기반 행렬의 역행렬(

)을 적용하여 월드 좌표계로 변환한 후, 변환된 깊이맵에 다시점 카메라(110)를 이루는 한 카메라(

)의 기반 행렬(

)을 적용하여 카메라(

)의 시점으로 투영(projection)한다. As shown in FIG. 4, first, a depth map obtained from the depth camera 120 (

) Is the inverse of the base matrix of the depth camera 120

After converting to the world coordinate system by applying a), one camera (1) to form a multi-view camera 110 on the converted depth map

Base matrix of

) To apply the camera (

Projection to the point of view.

This three-dimensional warping is performed for each of the cameras forming the multi-view camera 110, and as a result, a multi-view depth map composed of n three-dimensional warped depth maps corresponding to each of the images forming the multi-view image is obtained. Obtained. FIG. 5 illustrates a depth map in which the depth map illustrated in FIG. 3 is three-dimensional warped to a viewpoint of each camera. Referring to FIG. 5, it can be seen that due to the 3D warping, a plurality of holes, which are pixels having no defined depth value, are generated.

Referring back to FIG. 1, the multi-view image obtained from the multi-view camera 110 and the multi-view depth map from the 3D warping unit 130 are input to the trimap generator 140 and the trimap generator 140. ) Generates a multiview trimap consisting of trimaps corresponding to each of the images forming the multiview image using the multiview depth map.

The trimap generator 140 uses a target image and a three-dimensional warped depth map to generate a trimap corresponding to each image forming a multiview image. FIG. 6 is a diagram illustrating a specific embodiment of the trimap generator 140. Hereinafter, a trimap is generated with a three-dimensional warped depth map corresponding to a target image forming a multi-view image. The process of doing this is explained in detail.

As illustrated, the trimap generator 140 includes a segment 141, a foreground area determiner 142, a hole remover 143, and an erosion / expansion calculator 144.

The segment unit 141 generates segmented images by performing color segmentation on each image forming a multiview image. Color segmentation refers to a process of dividing an image into regions (segments) of similar colors.

The segment unit 141 sets segments between pixels whose luminance or chrominance difference between adjacent pixels is less than or equal to a threshold in the target image to which color segmentation is to be performed. That is, the segment unit 141 divides the image into segments by grouping pixels having similar luminance or color information in the target image. In this case, it may be assumed that the depth value is minutely changed in the divided segment, and the discontinuity point of the depth value is generated at the boundary of the segment. In order to satisfy this assumption, it is desirable that the segment be subdivided to be as small as possible, so that the threshold can be made small. As the threshold increases, the range of similar luminance or chrominance increases, which may increase the size of the segment, thereby increasing the probability of including a discontinuity point of the depth value in the segment. An example of an image in which an image in the center of the multi-view images of FIG. 3 is segmented by the segment unit 141 is illustrated in FIG. 7.

In the depth map obtained from the depth camera 120, the depth value does not exist at all or is very sparse in an area far from the depth camera 120 according to the setting of the depth camera 120. In the area nearer to), all depth values exist or are very dense. The same is true of the three-dimensional warped depth map, except that a hole, which is a pixel whose depth value is not defined, is generated due to the three-dimensional warping. As described above, since the discontinuity point of the depth value may occur at the boundary of the segment, it may be determined that the region having a large depth value among the segments is the foreground region, and the other region is the background region.

Accordingly, the foreground region determiner 142 segments the three-dimensional warped depth map corresponding to each segment of the image by the segment unit 141, and the depth value is present at a predetermined ratio or more in the segment. Determine the set of segments as the foreground area. The ratio may be arbitrarily determined appropriately, for example, to determine the foreground area as a set of segments having a depth value of 70% or more.

As described above, there are a plurality of holes whose depth values are not defined in the three-dimensional warped depth map, and the hole removing unit 143 displays pixel values surrounding the holes in the foreground area of the three-dimensional warped depth map. Fill it with the appropriate depth value. Parts outside the foreground area are no longer of interest, so the hole is filled only in the foreground area.

In this embodiment, a 3x3 block is set around each hole in the foreground area, and the average of depth values of available pixels among the eight pixels adjacent to the hole, that is, pixels having a depth value, is used as the depth value of the corresponding hole. Decide For example, as shown in FIG. 8, the depth values of the pixels having the depth value among the eight pixels adjacent to the center pixel corresponding to the hole are 200, 212, and 207, and therefore their average (200 + 212 + 207/3). = 206.33) to make 206 the depth value of the pixel. Repeating this process will fill all the holes in the foreground area. FIG. 9 illustrates a depth map obtained by determining a foreground area from one of the three-dimensional warped depth maps shown in FIG. 5 and filling all holes of the foreground area.

Referring back to FIG. 6, a three-dimensional warped multiview depth map in which the foreground area is divided and all of the holes are input to the erosion / expansion calculator 144, and the erosion / expansion calculator 144 inputs the multiview depth map. A multiview trimap is generated by applying erosion and expansion operations to the. The trimap is the image before obtaining the alpha matte, which is essential for matting, divided into the foreground and the background, and the unknown region whether it is the foreground or the background. The erosion / expansion calculating unit 144 makes each depth map constituting the input multiview depth map into a binary image according to a predefined threshold, and then erodes the edge of the binary image inwards and expands outwards. Create a multiview trimap by creating an unknown region. For example, the trimap may be generated to have a pixel value of 0 for the background, 255 for the foreground, and 128 for the unknown area. The size of the unknown area may be predefined.

FIG. 10 illustrates a multiview trimap generated from the three-dimensional warped multiview depth map illustrated in FIG. 5 by the operation of the trimap generator 140 described above.

Referring back to FIG. 1, the multi-view trimap from the trimap generator 140 is input to the alphamat generator 150, and the alphamat generator 150 again uses the input multiview trimap. Create a multiview alphamat corresponding to the point image. That is, the trimap generator 140 generates an alpha matte using each trimap constituting the multiview trimap to generate a multiview alphamat including alphamats corresponding to each image constituting the multiview image. do. The method of generating an alphamat from a trimap can use one of the known techniques. An example will be described.

Alphamat means an image composed of α (alpha) values used in the matting equation I = αF + (1-α) B (I, F, and B are composite images, foreground images, and background images, respectively). The matting equation combines the foreground and background pixels. Although not shown, the alphamat generator 150 may include a Cb converter, an edge extractor, an unknown region edge extractor, and an edge labeler.

The Cb converter converts the RGB color space into the YCbCr color space in the multiview image, and applies a Gaussian filter to Cb to reduce noise around the foreground boundary. The edge extractor extracts a boundary from the image converted by the Cb converter. The unknown region edge extractor extracts the unknown region edges of each viewpoint image based on the shared trimap unknown region information and the boundary of the multiview image extracted by the edge extractor. The edge labeling unit deletes edges having a predetermined length or less based on the found edge lengths and performs labeling using only the remaining edges. The edge labeling unit finds both ends of the labeled edges and connects the nearest endpoints to complete the closed curve. As a result, the inside of the final closed curve is the foreground and the outside is the background.

In this way, a multiview alphamat is completed by generating an alphamat corresponding to each image of a multiview image. FIG. 11 shows a multiview alphamat generated from the multiview trimap shown in FIG. 10.

The multiview alphamat from the alphamat generator 150 is input to the foreground extractor 160, and the foreground extractor 160 applies a multiview alphamat to the multiview image from the multiview camera 110. Extract multi-view foreground. That is, a multiview foreground consisting of images extracted from the foreground is obtained by applying each alphamat forming a multiview alphamat to each image forming a multiview image.

Although not shown, the apparatus for processing a multiview image according to the present embodiment may further include a background synthesis unit generating a multiview composite image. The background synthesizer receives a multiview foreground image from the multiview foreground extractor 160, and generates a multiview composite image by combining the multiview background taken in the same camera environment as the multiview camera 110.

12 is a flowchart illustrating a multiview image processing method according to an embodiment of the present invention. The multi-view image processing method according to the present embodiment includes steps processed by the multi-view image processing apparatus described above. Therefore, even if omitted below, the above description of the multi-view image processing apparatus is applied to the multi-view image processing method according to the present embodiment.

In operation 1210, the depth map obtained from the depth camera 120 is three-dimensional warped to the viewpoint of the multiview camera 110 to obtain a multiview depth map.

In operation 1220, a multiview trimap corresponding to a multiview image is generated using the multiview depth map.

In operation 1230, a multiview alphamat corresponding to a multiview image is generated using the multiview trimap.

In operation 1240, a multiview foreground is extracted by applying a multiview alphamat to the multiview image. Here, the extracted multiview foreground may be synthesized with a multiview background taken in the same camera environment as the multiview camera 110 to generate a multiview composite image.

13 is a flowchart illustrating a more specific embodiment of step 1220.

In operation 1310, segmented images are generated by performing color segmentation on each image constituting the multi-view image.

In operation 1320, a set of segments having a depth value greater than or equal to a predetermined ratio among the segments for each segmented image is determined as the foreground area.

In operation 1330, the hole generated by the 3D warping is filled in the corresponding foreground area of the multiview depth map.

In operation 1340, a multiview trimap is generated from the multiview depth map filled with holes.

Meanwhile, the above-described embodiments of the present invention can be written as a program that can be executed in a computer, and can be implemented in a general-purpose digital computer that operates the program using a computer-readable recording medium. The computer-readable recording medium may be a magnetic storage medium (for example, a ROM, a floppy disk, a hard disk, etc.), an optical reading medium (for example, a CD-ROM, DVD, etc.) and a carrier wave (for example, the Internet). Storage medium).

So far I looked at the center of the preferred embodiment for the present invention. Those skilled in the art will appreciate that the present invention can be implemented in a modified form without departing from the essential features of the present invention. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the scope will be construed as being included in the present invention.

1 is a diagram illustrating a configuration of a multiview image processing apparatus according to an embodiment of the present invention.

2 is a diagram three-dimensionally showing an example of a form in which a multiview camera 110 and a depth camera 120 are arranged.

3 shows an example of a multiview image obtained from the multiview camera 110 and one depth map obtained from the depth camera 120.

4 is a reference diagram for explaining a process of three-dimensional warping a depth map obtained from the depth camera 120 to a viewpoint of an arbitrary camera of the multiview camera 110.

FIG. 5 illustrates a depth map in which the depth map illustrated in FIG. 3 is three-dimensional warped to a viewpoint of each camera.

6 is a block diagram illustrating a specific embodiment of the trimap generator 140.

7 is an example of an image in which an image in the center of the multi-view images of FIG. 3 is segmented by the segment unit 141.

8 is a reference diagram for explaining an example of a method of filling a hole in a 3D warped depth map.

FIG. 9 illustrates a depth map obtained by filling all of the holes of the foreground area in one of the three-dimensional warped depth maps shown in FIG. 5.

FIG. 10 illustrates a multiview trimap generated from the 3D warped multiview depth map illustrated in FIG. 5 by the operation of the trimap generator 140.

FIG. 11 illustrates a multi-view alphamat generated from the multi-view trimap shown in FIG. 10.

12 is a flowchart illustrating a multiview image processing method according to an embodiment of the present invention.

FIG. 13 is a flowchart illustrating a more specific embodiment of step 1220 of FIG. 12.

Claims (17)

(a) three-dimensional warping the depth map obtained from the depth camera to the viewpoint of the multiview camera to obtain a multiview depth map; generating a multiview trimap corresponding to a multiview image obtained from the multiview camera using the multiview depth map; (c) generating a multi-view alphamat corresponding to the multi-view image using the multi-view trimap; (d) extracting a multiview foreground by applying the multiview alphamat to the multiview image; In generating the trimaps constituting the multi-view trimap in the step (b), each image constituting the multi-view image and each three-dimensional warped depth map constituting the multi-view depth map, (b1) determining a foreground area in the three-dimensional warped depth map and filling a hole generated by the three-dimensional warping in the foreground area; And (b2) generating the trimap from the depth map filled with the holes. delete The method of claim 1, Step (b1), (b11) generating segmented images by performing color segmentation on the images; (b12) determining, as the foreground region, a set of segments among the segments of the segmented image having a depth value in the three-dimensional warped depth map more than a predetermined ratio; And and (b13) filling a hole generated due to the three-dimensional warping in the foreground area of the three-dimensional warped depth map. The method of claim 3, In the step (b13), the hole is filled by setting an average of depth values of available pixels among the pixels around the hole as the depth value of the hole. The method of claim 4, wherein The pixels around the hole are eight pixels adjacent to the hole. The method of claim 1, The step (b2), the multi-view image processing method characterized in that for generating the trimap by applying the erosion and expansion operations to the depth map filled with the hole. The method of claim 1, In the step (a), the three-dimensional warping is performed using the camera parameters of the depth camera and the camera parameters of each of the cameras constituting the multiview camera. The method of claim 1, And synthesizing the extracted multi-view foreground with a multi-view background to generate a multi-view composite image. A computer-readable recording medium having recorded thereon a program for executing the multiview image processing method according to any one of claims 1 to 10. A three-dimensional warping unit which three-dimensional warps the depth map obtained from the depth camera to the viewpoint of the multi-view camera to obtain a multi-view depth map; A trimap generation unit generating a multiview trimap corresponding to a multiview image obtained from the multiview camera using the multiview depth map; An alpha matte generator configured to generate a multiview alphamat corresponding to the multiview image using the multiview trimap; And A foreground extracting unit extracting a multiview foreground by applying the multiview alphamat to the multiview image; The trimap generator generates each trimap constituting the multiview trimap, each image constituting the multiview image and each three-dimensional warped depth map constituting the multiview depth map. Multi-view image processing comprising determining a foreground area from a three-dimensional warped depth map, filling a hole generated by the three-dimensional warping in the foreground area, and generating the trimap from the depth map filled with the hole. Device. delete The method of claim 10, The trimap generation unit, A segment unit generating color segments by performing color segmentation on the image; A foreground region determiner configured to determine, as the foreground region, a set of segments in which a depth value in the 3D warped depth map is greater than a predetermined ratio among the segments of the segmented image; And A hole removal unit filling a hole generated by the three-dimensional warping in the foreground area of the three-dimensional warped depth map; And And a trimap generator configured to generate the trimap from the depth map filled with the holes. The method of claim 12, And the hole removing unit fills the hole by setting an average of depth values of available pixels among the pixel values around the hole as the depth value of the hole. The method of claim 13, The pixels around the hole are eight pixels adjacent to the hole. The method of claim 12, The trimap generator generates the trimap by applying an erosion and expansion operation to the depth map filled with the holes. The method of claim 10, And the three-dimensional warping unit performs the three-dimensional warping by using camera parameters of the depth camera and camera parameters of cameras constituting the multi-view camera. The method of claim 10, And a background synthesizer configured to synthesize the extracted multiview foreground with a multiview background to generate a multiview composite image.

Images