KR100737808B1 - Method for efficiently compressing 2d multi-view images - Google Patents

Abstract

The present invention relates to a method of compressing a multiview image from a 2D camera array, the method comprising: generating a synthetic parallax image including parallax information of an object with respect to a reference image of a multiview image, and totally from the multiview image Predicting parallax; synthesizing a composite texture image of each viewpoint by adjusting a reference image according to the predicted total parallax; and using a synthesized parallax image and a composite texture image, a spatial axis from the original image of each viewpoint And simultaneously performing prediction and time-base prediction from a previously input image of the same view. According to the present invention, it is possible to reduce the amount of data for transmitting a multiview image through the time axis and the spatial side prediction of the multiview image, and to accurately predict the hidden image according to the viewpoint of each camera.

Multiview, Parallax, Depth, Compression, Prediction, Synthesis, Dense disparity, Depth, View

Description

Multi-view image compression method of two-dimensional structure {METHOD FOR EFFICIENTLY COMPRESSING 2D MULTI-VIEW IMAGES}

1 is a conceptual diagram of a multi-view prediction method according to the prior art.

2 is an exemplary diagram of triangulation for performing spatial prediction from two or more camera inputs in a multiview image compression method according to an embodiment of the present invention.

3 is a conceptual diagram of a method of predicting a current disparity vector using a vector value of an adjacent block in an object in a multiview image compression method.

4 is an overall flowchart of a multi-view image compression method according to an embodiment of the present invention.

5 and 6 conceptually illustrate a multi-view image compression method according to the flowchart of FIG.

The present invention is a method of reducing the amount of image data compared to the conventional compression method by compressing the spatial axis (time difference) and temporal using the image synthesized and reconstructed from the image from the two-dimensional camera array during compression transmission of the multi-view image It is about.

Recently, with the advent of the era of high-definition interactive multimedia, the movement to standardize the technology of receiving, compressing, transmitting, and restoring images from a multi-view camera system in connection with the production of 3D stereoscopic images is required. / Progress in IEC MPEG. Accordingly, many companies, schools, and research institutes are conducting research on this.

However, such a multi-view image compression algorithm is a cascade method of directly compressing a multiview image by predicting it only on the time axis, a method of compressing the image by predicting it from an input from a time axis and an adjacent camera, and compressing the image from the time axis and the adjacent camera. The unidirectional prediction method, and a synthesis algorithm using a high pass filter and a low pass filter using a wavelet algorithm have been proposed.

For example, Korean Patent Application No. 10-2003-0073241 "Adaptive multiplexing / demultiplexing apparatus and method thereof for three-dimensional multi-view multimedia processing" (hereinafter, referred to as "prior art 1") is a cascade method for input from various cameras. A compression / restore method using multiplexing / demultiplexing (Mux / Demux) is disclosed.

That is, according to the prior art 1, after receiving images from several cameras, the preprocessor corrects the parameters due to the difference in the characteristics of the cameras, and then removes the redundancy of the images at each viewpoint as much as possible. It compresses into video streams (Elementary Streams: ES), and inputs them to a multi-view video multiplexing device to generate each independent video stream into one multiplexed stream. Accordingly, in the multi-view video demultiplexing apparatus, the receiver demultiplexes the multiplexed multi-view video and inputs each video image stream to a video decoder corresponding to each view present in the multi-view video decompressor in the receiver. Restore each compressed image. Subsequently, the multiview video synthesizer generates an intermediate image between the restored multiview images and outputs the intermediate image to the display.

As described above, the cascade method of predicting and compressing an image on a time axis is the most basic method for compressing / restore an image input from a multiview camera, and the size of the compressed image is a capacity proportional to the number of input cameras. Therefore, the efficiency of compression does not improve at all. In addition, even if the compressed data is transformed into one video stream, the decoder and encoder are reconfigured according to the stream configuration. This is not an algorithm for improving compression efficiency, but a modification of existing compression standards to form Mux / Demux.

As a technique for improving the above-described Cascade method, Korean Patent Application No. 10-2003-0002116, "Compression / Restoration apparatus and method of multi-view image" (hereinafter referred to as "prior art 2") is an image input from a peripheral camera A method of predicting and predicting by the time axis is proposed, as shown in FIG. 1.

According to the space-time prediction method of the prior art 2, the central image encoder estimates the motion of the input central image and encodes the central image while compensating for the motion to generate the central image data stream and restore the center image. Provide an image as a reference image. The left image encoder estimates the movement of the input left image and compensates for the movement, and performs shift estimation and variation compensation by referring to the reconstructed center image provided by the center image encoder as a reference image and encodes the left image. The left image data stream is generated, and the right image encoder estimates the movement of the input right image and compensates for the movement, and estimates the variation and the variation by referring to the reconstructed center image generated by the center image encoder as a reference image. While performing the compensation, the right image is encoded to generate a right image data stream.

For example, as shown in FIG. 1, after the central image encoder encodes the central image of the multiview image as an I image, the encoding is repeated for the B image, the B image, and the P image, and the B image is repeated. The restored center image is restored to the left image encoder and the right image encoder when encoding is performed on the B image and the P image. In addition, the left image encoder and the right image encoder encode the left image and the right image, respectively, as an I image, and then encode the P image while referring to the reconstructed center image provided by the center image encoder as a reference image. Repeat what you do.

As described above, the spatiotemporal prediction method predicts each video stream temporally and increases efficiency by predicting from another camera in overlapping portions of the images. When the prediction is performed only on the time axis, due to parallax occurring in the overlapping area of the image and the arrangement of the cameras, an area that is not present in the image input from another camera is generated at both ends of the image. This area can increase efficiency by performing spatial prediction according to the camera arrangement. This method is slightly better than the one predicted in time and space compared to the one predicted on the time base.

However, when predicting an image, instead of predicting the cognitively accurate position, the comparison function predicts the least errored block, so it is not very efficient and the characteristics of each camera are different. In actual prediction, the prediction ratio between cameras is not high. In addition, spatial prediction between cameras has a lot of errors because three-dimensional rotational elements are excluded and predicted during the prediction between images. In the method of downsampling and resampling each sequential image by the high pass filter and the low pass filter in order, the data after the high pass and the low pass pass loses more image data than the original data. There is a disadvantage that the quality of the reconstructed image is not so high.

In order to solve the above-described problems, the present invention reduces the amount of data for the transmission of a multiview image through the time axis and the spatial side prediction of the multiview image, and can accurately predict the hidden image according to the viewpoint of each camera. The purpose is to provide a point image compression method.

In order to achieve the above object, according to an aspect of the present invention, there is provided a method of compressing a multiview image, generating a synthetic parallax image including parallax information of an object with respect to the reference image of the multiview image; Predicting total parallax from a multiview image, adjusting a reference image according to the predicted total parallax, synthesizing a composite texture image of each viewpoint, and using a circle of each viewpoint using the synthesized parallax image and the composite texture image A spatial axis and a time axis prediction step of simultaneously performing a spatial axis prediction from an image and a time axis prediction from a previously input image of the same viewpoint are included.

In this case, preferably, the space axis and time axis predicting step comprises: estimating the object rotation of the composite texture image according to the parallax information of the synthesis parallax image to predict the image of each viewpoint on the space axis; And a time axis prediction step of performing time axis prediction from the original image of each viewpoint using the compensated composite texture image.

The generating of the disparity image may include generating the disparity image by object-based disparity prediction, and converting the depth information into the disparity information when the multiview image includes depth information. If not, the parallax information of the object may be predicted by triangulation.

According to still another aspect of the present invention, there is provided a computer-readable recording medium having recorded thereon instructions for performing each step of the above-described multi-view image compression method.

Hereinafter, exemplary embodiments of the present invention will be described with reference to the accompanying drawings.

2 conceptually illustrates triangulation for performing spatial prediction from two or more camera inputs in a multi-view image compression method according to an embodiment of the present invention.

Unlike the inter prediction method, which has a short time difference in the same camera, the spatial prediction between cameras must include a three-dimensional concept. That is, in the inter prediction, the concept of rotation movement should be included, not simple translation movement. This is because when two cameras simultaneously acquire an image from one object, the object input to each camera will have the image rotated by the trigonometry, rather than simply moving the background and the object. Therefore, accurate and efficient prediction is not possible with simple spatial prediction.

In FIG. 2, b represents a baseline, and the baseline represents a distance between cameras. In general, the human eye has a baseline of 6.5 cm. xl and xr mean coordinates in an image inputted by two cameras at a point in real time and space. Cx means the center of the image, which is related to the size of the image. Depth Z in real space (distance from real focus to image plane) can be expressed as follows.

Z = f-f × b / (xl-xr)
Where f represents a focal length.

delete

As described above, when two different cameras look at a point in the object, the image projected on the camera generates a rotation, not a simple movement, and thus requires a calculation of the rotation when predicting parallax.

To this end, in the present invention, a synthesized disparity image was used. The use of parallax images not only suggests parallax values in all pixels, but also includes information about the rotational motion of the object. Synthetic disparity image is obtained by object-based disparity prediction method. This is not simply to find the prediction point of the unit with less error as a vector, but to predict under the assumption that adjacent vectors in the object have a similar size, the parallax vector in the object has a similar value and can find the correct vector.

3 illustrates an algorithm for predicting a current disparity vector using a vector value of an adjacent block in an object in a multiview image compression method.

First, in order to obtain the object-based parallax vector, it is necessary to distinguish between the inside and the outside of the object, and the technique applied at this time is an object extraction method using neighboring blocks. In other words, a vector in an object has a similar vector. In this case, instead of looking for a vector having a low error, a vector similar to a neighboring block and a low error block is not found.

When obtaining the vector value of the current block by using the vector value already obtained from the neighboring blocks, the conditions are as follows. When the median value of the vectors obtained from the neighboring blocks is determined as a candidate and the vector is obtained using the candidate vector, if the value is small enough, it is regarded as a parallax vector. Otherwise, if the size of the neighboring block vectors is the same and the parallax value is obtained using the vector, the error is small enough. If all of the above conditions are not satisfied, the entire search method is used.

More specifically, condition 1 first compares the vector (blue) of the block to be obtained with the vector (purple) of the neighboring block, and the error is below the threshold when predicted using the median (median). If it is a vector of the current block. Next, Condition 2 recognizes as a vector of the current block if all surrounding vectors have the same value and an error using the vector is less than or equal to a threshold. Finally, if Condition 3 does not satisfy the above conditions 1 and 2, it does not recognize as a block in the same object and rescans.

Next, FIG. 4 is an overall flowchart of a multiview image compression method according to an embodiment of the present invention, and FIGS. 5 and 6 conceptually illustrate a multiview image compression method according to the flowchart of FIG. 4.

First, global disparity is required in order to create an accumulated image. The total parallax is a parallax generated by the arrangement between the cameras. In the background, the left image has more left portions than the reference image. The composite reference image includes such a portion, and predicts an image from each camera by using the information of the synthesized parallax image.

Referring to FIG. 4, first, a stream of a multi-view video is input (S410), and data may be based on a parallel camera model or a convergent model. . Subsequently to step S410, it is determined whether the stream has a depth map (S420), and if it is available, depth information is converted into parallax (S430). At this time, the parallax can be obtained from the depth information according to Equation 1 described above. If the depth map is not available in the above-described step (S420), disparity estimation is performed (S440), and the parallax prediction method of FIG. 3 can be used.

Subsequent to the above-described step S430 or S440, a frame of an accumulated dense disparity image including parallax information of an object with respect to the reference image is formed, that is, an accumulated disparity frame (S450). .

Next, in relation to the configuration of the texture (RGB) image, the global parallax is obtained through the global disparity estimation of the multi-view image stream of the above-described step S410, and based on the reference image, By adjusting an image according to each viewpoint (S460), an synthesized texture frame or an synthesized texture image is synthesized (S470). In this case, in the case of a 3 × 3 camera array, when the center image is used as the reference image, a total of eight composite texture images corresponding to each viewpoint of the surroundings can be obtained.

Finally, time-base prediction (previous MPEG prediction method) and inter-image (for each camera input) using the synthesized disparity frame and the accumulated texture frame generated using the total parallax. Spatial) prediction (Inter-view Prediction) is performed (S480).

More specifically, as shown in FIG. 5, by using the parallax information of the object included in the synthesized parallax image to compensate for the rotation of the object in the accumulated texture image (Accumulated disparity frame) to the image of each view The T'th segment frame is predicted on the spatial axis, and at the same time, the image of the past time, that is, the (T-1) th frame (T-1), is already compressed with respect to the image of each viewpoint on the time axis. T-th segment frame prediction is performed by compensating for motion from 'th frame'. That is, the spatial axis prediction and the time axis prediction result are reflected together in the prediction of the T-th segment frame with respect to the image of each viewpoint. The predicted image is made of the same number of images as the input image.

On the other hand, in the case of a 3 X 3 camera array, as shown in Figure 6, when the surrounding image is immediately predicted by using the center image as a reference image, the occlusion area of the image becomes large, and in the image This may occur when the object is not accurately predicted by the rotation phenomenon when looking at another image. Therefore, in a preferred embodiment of the present invention, by using the parallax compensation according to the rotation information of the object obtained from the parallax information and the viewpoint adjustment according to the entire parallax, the accumulated texture image is predicted, and from each viewpoint image ( predict the view.

As described above, according to a preferred embodiment of the present invention, a composite texture image that becomes a composite reference image from each camera input is synthesized by using global disparity and accumulated dense disparity image. The spatial axis prediction is performed from the original image, and at the same time, the time axis prediction is performed from the image of the past time (frame) which is already compressed.

When simple spatial prediction is performed from the central reference image to each camera input, the parallax between the cameras and the parallaxes of objects close to the camera become very large, and the prediction of the invisible region resulting from the global disparity Since an error occurs, the degradation of the image quality and the increase of the computation amount occur when the entire image is predicted. When predicting the image input from each camera using the predicted composite image from the global disparity and the accumulated dense disparity image, the amount of computation is reduced during prediction due to the reduction of the parallax Since it includes a hidden image that does not have a reference image, accurate prediction and prediction errors are reduced.

Although the preferred embodiment according to the present invention has been described above, this is merely exemplary and those skilled in the art will understand that various modifications and equivalent other embodiments are possible therefrom. Therefore, the protection scope of the present invention should be defined by the following claims.

As described above, according to the present invention, it is possible to reduce data and reduce network occupancy for transmitting a multiview image through prediction between inputs of a composite image and a camera, and also to display a hidden image without a reference image. Predictive errors are reduced because they can be predicted accurately. Furthermore, since the image can be modeled using the synthetic parallax image, various application fields such as object extraction, 3D rendering, and blue screen effect can be made.

Claims (7)

As a method of compressing a multiview image, Generating a synthetic parallax image including parallax information of an object with respect to a reference image of the multiview image; Predicting total parallax from the multi-view image; Synthesizing a composite texture image of each viewpoint by adjusting a reference image according to the predicted total parallax; A space axis and time axis prediction step of simultaneously performing the spatial axis prediction from the original image of each viewpoint using the synthesized parallax image and the composite texture image and the time axis prediction from the input image of the same viewpoint Compression method of a multi-view image comprising a. The method of claim 1, wherein the space axis and time axis prediction step Compensating the rotation of the object of the composite texture image according to the parallax information of the composite parallax image to predict the image of each viewpoint on the spatial axis, and at the same time perform the time axis prediction from the previously input image of the same viewpoint. Compression method. The method of claim 1 or 2, wherein the generating of the synthesized parallax image The multi-view image compression method of generating the synthesized parallax image by object-based parallax prediction. The method of claim 3, wherein the object-based disparity prediction The method of claim 1, further comprising extracting an object by finding a block that is similar to a neighboring block and has less error. The method of claim 1 or 2, wherein the generating of the synthesized parallax image And when the multiview image includes depth information, converting the depth information into the parallax information. The method of claim 5, wherein the generating of the synthesized parallax image And if the multiview image does not include depth information, predicting parallax information of the object by triangulation. A computer-readable recording medium having recorded thereon instructions for performing each step of the multi-view image compression method according to claim 1.

Images