KR100731981B1 - The method and system of multi-view video encoding based on grid type pyramid gop structure - Google Patents

Abstract

The present invention provides a multi-view image compression encoding method for constructing an image including one or more GOPs (Group of Pictures) composed of a plurality of frames for individual viewpoints, and obtaining and compressing images of the same object from a plurality of viewpoints. A grid configured such that each GOP comprises an I frame, a B frame, and an RB frame, wherein two GOPs at adjacent points in time are spaced apart by 1/2 of the number of frames constituting the GOP; A multi-view video compression encoding method for forming and decoding a shape GOP structure.

GOP, Multiview, Compression Coding, H.264, Grid

Description

METHOD AND SYSTEM OF MULTI-VIEW VIDEO ENCODING BASED ON GRID TYPE PYRAMID GOP STRUCTURE}

1 is a view for explaining a conventional multi-view video encoding method.

2 is a diagram illustrating a lattice-type GOP structure arranged for multi-view image coding according to the present invention.

3 is a diagram illustrating a process of performing multi-view image encoding according to an embodiment of the present invention.

4 is a diagram showing the hierarchy of each frame in the encoding method according to the present invention;

5 is a diagram illustrating a process of multiview image encoding according to another embodiment of the present invention.

The present invention relates to multiview video, and more particularly to efficient compression of multiview video.

A multi-view image is a field of 3D image processing that geometrically corrects images taken by a plurality of cameras and provides a user with various viewpoints in various directions through spatial synthesis. Such a multi-view image is characterized by increasing freedom of view to a user. This has the advantage of providing a user with a free view and a three-dimensional effect through a wide frame, but there is a problem that requires an overlapping condition, an increase in the amount of instantaneous processing data, and expensive equipment that are essential when acquiring an image.

As shown in FIG. 1, the latest multi-view image compression is performed by independently compressing each view using H.264, which is the most efficient compression method of the related art.

1 shows an example of eight viewpoints. An image of one viewpoint is composed of several GOPs (Group of Pictures). Here, the GOP refers to a set of frames composed of one I frame and a plurality of P frames or B frames. In FIG. 1, one GOP includes one I frame, three P frames, and eight B frames. It consists of a frame.

The I frame of each GOP is a location that can be accessed randomly during video playback, and appears every 0.5 seconds in the conventional method. P frames are predictively encoded using I frames as reference frames, and in the case of B frames, they are predictively encoded using I frames and P frames as reference frames.

However, the conventional method takes a method of encoding images of each viewpoint independently. In the case of H.264 used for independent encoding of each view, encoding parameters are set as follows. The bit rate control is used and the basic unit of the bit rate control is the number of macro blocks corresponding to one line of the image size to be encoded.

Using the rate-distortion optimization technique, the search range for motion prediction used for P frame or B frame prediction is ± 32. The number of reference frames is five, and temporal random access should be possible every 1 second for 15 fps and 0.5 seconds for 25/30 fps. In other words, an I frame must be inserted in each unit.

The structure of the GOP uses the IBBP ..... structure. Loop filter is used, and in the case of VLC, context-based adaptive binary arithmetic coding (CABAC) is used.

  Such a method for multi-view coding should basically have good coding efficiency and have a small delay in time. In addition, consistency of image quality between viewpoints should be maintained, and temporal and spatial random access should be possible.

In the case of the conventional method, the spatial correlation in one frame is considered, but the spatial correlation in each view is not considered.

In this case, the spatial correlation refers to the similarity between images of adjacent viewpoints at the same time. In the case of adjacent viewpoints, only the position of the camera has changed slightly, so that the images of the adjacent viewpoints of the same time are very similar, and if they are actually used in the prediction encoding process, they show better performance than conventional methods in terms of bit rate and image quality. Can be. However, in the conventional method, images of respective viewpoints are independently encoded without considering neighboring viewpoint images, and the encoding efficiency is dependent only on the temporal correlation of individual viewpoints. However, considering multi-view video coding, spatial correlation and spatial correlation need to be considered for better coding efficiency.

SUMMARY OF THE INVENTION In order to solve the above-mentioned problems of the prior art, an object of the present invention is to provide a compression encoding method that considers temporal and spatial correlation between viewpoints in multi-view video compression encoding.

According to the present invention for achieving the above technical problem, to configure an image including one or more GOP (Group of Pictures) consisting of a plurality of frames for each viewpoint, and to obtain the image of the same object from a plurality of viewpoints In the multi-view video compression encoding method for compressing, each GOP includes an I frame, a B frame, and an RB frame, wherein two GOPs at adjacent points in time are separated by a half of the number of frames constituting the GOP. A multiview image compression encoding method is provided in which a lattice-shaped GOP structure configured to arrange the I frames is formed and encoded. Each GOP is configured in the order of I, B, B, RB, B, B, RB, B, B, RB, B, B frames, and first encoding the I frame of each GOP, and I at the same time. Forming an RB frame by using a frame, I frames of adjacent time points of the same time zone, and an RB frame positioned between the I frame and the RB frame by using an I frame at the same time point and an RB frame at the same time point. And forming a B frame using the RB frame and the I frame.

In a multi-view video compression encoding system according to another aspect of the present invention, a plurality of image capturing apparatuses of the same object from a plurality of viewpoints, and the image obtained for each of the image capturing apparatus includes I frames, B frames and RB frames Compression encoding means for forming a compressed image consisting of one or more GOPs (Group of Pictures) consisting of a plurality of frames, wherein the compression encoding means comprises a GOP for an image acquisition device adjacent to each other A multi-view video compression encoding system is provided, which forms and encodes a grid-shaped GOP structure configured to arrange the I frames with a time difference of 1/2 of the number of frames.

The compression encoding means first encodes an I frame of each GOP, forms an RB frame using I frames of the same view, and I frames of two or more adjacent views of the same time zone, and I frames of the same view. And a process of forming an RB frame located between the I frame and the RB frame using the RB frame at the same time point, and forming a B frame using the RB frame and the I frame. Perform video compression coding.

EMBODIMENT OF THE INVENTION Hereinafter, with reference to an accompanying drawing, the structure of this invention is demonstrated in detail.

2 is a diagram illustrating a lattice GOP structure for performing multi-view video compression coding with reference to a frame of an adjacent view according to an exemplary embodiment of the present invention. As shown, basically, each GOP is composed of I, B, B, RB, B, B, RB, B, B, RB, B, and B frames in order, and the positions of the I frames of each viewpoint are adjacent to each other. A GOP structure is provided in which a delay is applied to a GOP corresponding to an even time point by half the number of frames constituting one GOP so as not to coincide with.

The reason for adopting such a novel and advanced GOP structure of the present invention is as follows. The coding efficiency for a point in time is most dependent on the I frame. Since the I frame is used as a reference frame of another frame, it has a great influence on the coding efficiency until the next I frame comes out. Therefore, the narrower the interval between the I frames, the better the image quality. However, the more I frames, the more bits are generated, which results in a loss of compression efficiency.

In order to effectively improve the coding efficiency while compensating for this disadvantage, the method according to the present invention defines a new frame called an RB frame. This is a reference B frame. The RB frame is at the same position as the P frame of the conventional method and is the same. However, it has a characteristic of being encoded like a B frame and used as a reference frame of another B frame. In the proposed algorithm, RB frames are encoded with high quality as much as I frames. In this way, if a plurality of high quality frames are encoded using fewer bits and used as reference pictures, the encoding efficiency of other frames is improved accordingly.

FIG. 3 is a diagram for describing how each frame is encoded in the lattice GOP structure of FIG. 2. In FIG. 3, the method of the present invention will be described by taking the encoding of two viewpoints located at the far left on the screen as an example.

First, the first I frame [I (1)] of the left view is encoded, and the I frame [I (2)] of the adjacent right view is encoded. The RB frames (RB (3) and RB (4)) are encoded with these two frames as reference pictures. Then, other RB frames (RB (5), RB (6)) are encoded with reference frames before and after the same RB frame and I frame, respectively.

Each arrow in FIG. 3 shows the reference relationship described above.

As described above, in the case of an RB frame, an RB frame located at the same time zone as the I frame is encoded using an I frame at the same time as an I frame at an adjacent time point in the same time zone, and an RB frame not located at the same time zone as the I frame. The RB frame and I frame at the same time are used as reference pictures. The remaining B frames are encoded by using RB frames or I frames positioned before and after, as in the conventional method, as reference pictures.

As with the RB frame, a reference relationship for the B frames is shown on the right side of FIG. 3.

  As mentioned above, in the proposed method, the GOP structure is basically arranged in a grid so that the efficiency of I frame can be divided evenly in the first half of multi-view image. To this end, in the case of images of adjacent viewpoints located in the same time zone as the I frame, time and space efficient prediction is performed using not only I frames of the same viewpoint but also I frames of adjacent viewpoints as a reference screen, so that the image quality is similar to that of the I frame with fewer bits. The coding resulted in an I frame appearing at half the actual interval. In addition, the P frame of the conventional method is replaced with the RB frame, and the coding order is adjusted to enable the use of all images in the front and rear time zones, so that the coding efficiency due to the temporal correlation can be effectively improved.

In the above-described embodiment, the quantization parameter (QP) of the RB frame is set to be the same or similar to the QP of the I frame to make the image quality of the RB frame the same or similar to the image quality of the I frame.

In addition, by compressing and encoding QP in the order of I frames, RB frames, and B frames, compression efficiency can be increased while minimizing degradation of image quality.

4 is a diagram illustrating a hierarchical structure of each frame. 4 shows a pyramid GOP structure for the case where the intra interval is 12. Create an intermediate RB scene using the reference I scenes as shown. Create two RB scenes using these RB and I scenes, and then create B scenes using the RB and I scenes.

  Through this, it is possible to maximize the coding efficiency of the B scene by referring to the previous scene and the subsequent scene in time. In this process, images of the I scene position, which is a reference, are encoded in high definition and used by increasing the QP toward the lower layer.

The proposed method can be extended as follows. As shown in the figure below, not only two points of view can be extended but also three or more points. It is also possible to adjust the length of the GOP.

FIG. 5 is a diagram illustrating a GOP structure and a reference relationship between frames when a multiview image is encoded by linking three viewpoints.

As shown, I frames (I (1), I (2), I (3)) of the GOP for three viewpoints are first encoded, and RB frames (eg, RB (4)) are three I frames. Encoding is performed using [I (1), I (2), I (3)]. Other frames may be encoded as described above, and may be encoded using an I frame or an RB frame of an adjacent view.

Although the configuration of the present invention has been described in detail with reference to the preferred embodiments of the present invention and the accompanying drawings, this is only an example and the scope of the present invention is not limited thereto. That is, those skilled in the art will be able to make various modifications and changes within the scope of the present invention. Therefore, the protection scope of the present invention should be defined by the following claims.

As described above, according to the present invention, in consideration of the spatial correlation that has been pointed out as a disadvantage in the conventional method, the structure is modified efficiently in the conventional method in accordance with the characteristics of the multi-view image coding to efficiently improve the spatial and spatial correlation. Can be considered. Lattice GOP placement and RB scenes maximize the effects of the I-scene, which plays a large role in determining image coding efficiency.

Compared to the conventional method, since the interval between I scenes is theoretically half-effective and uses fewer bits, the image quality and bit generation rate can be gained. In addition, in the case of the image of the same view, the temporal order is slightly modified to use an improved temporal correlation.

As a result, it is possible to have a better image quality with fewer bits than the conventional multi-view image coding method. In addition, the number of unnecessary reference pictures is reduced to show good performance in terms of encoding time.

Claims (11)

A multi-view image compression encoding method for constructing an image including one or more GOPs (Group of Pictures) composed of a plurality of frames for individual viewpoints, and obtaining and compressing images of the same object from a plurality of viewpoints, Each GOP is composed of an I frame, a B frame, and an RB frame, and two GOPs of adjacent time points are arranged in a lattice form in which the I frames are arranged with a time difference of 1/2 of the number of frames constituting the GOP. A multi-view video compression encoding method for forming and encoding a GOP structure. The method of claim 1, Wherein each GOP consists of I, B, B, RB, B, B, RB, B, B, RB, B, B frames. The method of claim 1, First encoding the I frames of each GOP, Forming an RB frame using I frames at the same time and I frames at adjacent time points in the same time zone; Forming an RB frame located between the I frame and the RB frame by using an I frame at the same time and an RB frame at the same time; Forming a B frame using the RB frame and the I frame How to include. 4. The method of claim 3, wherein the quantization parameter (QP) of the RB frame is the same as or similar to the QP of the I frame. The method of claim 3, wherein the compression is performed by increasing the QP in the order of the I frame, the RB frame, and the B frame. In the multi-view image compression encoding method for forming an image including one or more GOP (Group of Pictures) consisting of a plurality of frames for each viewpoint, and obtaining and compressing the same object from a plurality of viewpoints, Each GOP comprises an I frame, a B frame, and an RB frame, wherein GOPs in the same time zone at three or more adjacent time points are spaced apart by 1/2 of the number of frames constituting the GOP. And forming and encoding a lattice-shaped GOP structure configured to be arranged. The method of claim 6, First encoding the I frames of each GOP, Forming an RB frame by using an I frame at the same time and two or more adjacent frames at the same time. Forming an RB frame located between the I frame and the RB frame by using an I frame at the same time and an RB frame at the same time; Forming a B frame using the RB frame and the I frame How to include. 8. The method of claim 7, wherein the quantization parameter (QP) of the RB frame is the same as or similar to the QP of the I frame. The method of claim 7, wherein the compression is performed by increasing the QP in the order of the I frame, the RB frame, and the B frame. In a multi-view video compression coding system, A plurality of image acquisition apparatuses for the same object from a plurality of viewpoints, Compression encoding means for forming a compressed image including one or more group of pictures (GOPs) including a plurality of frames including an I frame, a B frame, and an RB frame; Wherein the compression encoding means is to form and encode a grid-shaped GOP structure configured such that the GOPs for the adjacent video acquisition apparatus by the time difference by 1/2 of the number of frames constituting the GOP is arranged to place the I frame Point video compression coding system. The method of claim 10, wherein the compression encoding means Encoding the I frame of each GOP first, Forming an RB frame using I frames at the same time and two or more adjacent time frames at the same time; Forming an RB frame located between the I frame and the RB frame by using an I frame at the same time and an RB frame at the same time; And performing a process of forming a B frame by using the RB frame and the I frame.

Images