KR20070074495A - Method and apparatus for inter-viewing reference in multi-viewpoint video coding - Google Patents

Abstract

An inter-view frame reference method in multi-view moving picture coding and an apparatus therefor are provided to reduce a residual signal and enhance a compression rate by referring to a frame of a different viewpoint of the same time or a frame of a different viewpoint of a different time when multi-view moving picture data are encoded. A multi-view GOP(Group Of Picture) arranging unit(110) includes GOPs corresponding to at least one or more viewpoints, and arranges the respective view points between the respective GOPs to have the same coding structure. A mutual viewpoint reference unit(120) refers in the picture order within a spatially close viewpoint among pictures of different viewpoints previously encoded in the respective GOPs.

Description

Method and apparatus for inter-view frame reference in multi-view video coding {method and Apparatus for inter-viewing reference in multi-viewpoint video coding}

1 is a block diagram of an inter-view frame reference apparatus for encoding multi-view video data according to an embodiment of the present invention.

2 illustrates a coding structure of a multi-view GOP proposed for inter-view frame reference of multi-view video data according to an embodiment of the present invention.

3 illustrates an inter-view frame reference scheme for P picture encoding in a camera type of first order parallel and arc structure.

4 illustrates an inter-view frame reference scheme for B picture encoding in a camera type of first order parallel and arc structure.

5 illustrates an inter-view frame reference method for encoding of a P picture and a B picture in a camera type having a quadratic cross structure.

FIG. 6 illustrates another inter-view frame reference scheme for encoding a P picture and a B picture in a camera type having a quadratic cross structure.

7 is a diagram illustrating an inter-view frame reference method for encoding a P picture and a B picture in a camera type of a quadratic parallel structure (3 × 5).

8 illustrates a flowchart of an inter-view frame reference method for encoding multi-view video data, according to an embodiment of the present invention.

9A and 9B illustrate a test environment for experimenting with the effect of the inter-view frame reference method of the present invention.

10 (a) to 10 (g) show rate-distortion curves based on the experimental environment of FIG.

The present invention relates to a structure and an apparatus for referencing an inter-view frame in order to encode multi-view video data photographed by a plurality of cameras having different viewpoints.

Currently, according to the H.264 standard used for encoding video data, it refers to a plurality of frames that exist before and after in time only within the same viewpoint.

Until now, the technology has only been referred to a frame within the same view, but in the present invention, in encoding the multi-view video data, the present invention provides a technical idea of referring to a frame at a different point in time or at a different point in time. We want to increase the compression rate by reducing the residual signal rather than the method of referencing only past or future frames in time.

In a preferred embodiment of the present invention, an inter-view frame reference apparatus for encoding multi-view video data includes a GOP corresponding to at least one or more viewpoints, and each GOP includes an Instantaneous Decoder at an initial time at which coding starts. (Refresh) a multiview GOP including a picture and each view between each GOP having the same coding structure; and the base GOP, the IDR picture, and the last view, which perform only temporal frame references within their own view of each GOP. And a mutual viewpoint reference unit for referring to pictures within a spatially close viewpoint among pictures of other viewpoints already encoded.

In a preferred embodiment, the inter-view frame reference apparatus for encoding multi-view video data further includes a same-view reference unit that refers to the same-view picture that is already encoded.

In a preferred embodiment, the basic GOP is characterized in that the H.264 / AVC GOP.

In a preferred embodiment, in the inter-view frame reference apparatus for encoding multi-view video data, the GOP of each view is encoded in the order of an I picture, a P picture, an rB picture, and a B picture.

In a preferred embodiment, in the inter-view frame reference apparatus for encoding multi-view video data, the P picture of the cross-view reference unit may be referred to as b (i, t) for the P picture to be encoded at the current i-th time, t time. In the case of at least one of b (i-1, t), b (i-1, tn) and b (i + 1, tn), tn denotes an I picture or a P picture at the closest time before t hours. Let it be time coded.

According to a preferred embodiment, in the inter-view frame reference apparatus for encoding multi-view video data, an rB picture to be encoded at a current i-th point and t-th time of the cross-view reference unit is b (i, t). three of (i + 1, t + n), b (i-1, t + n), b (i-1, t), b (i-1, tn), and b (i + 1, tn) Tn is a time at which an I picture or a P picture is encoded at the closest time before t hours, and t + n is a time at which the I picture or P picture is encoded at the closest time after t hours.

As a preferred embodiment, as a preferred embodiment, in the inter-view frame reference apparatus for encoding multi-view video data, a B picture to be encoded at a current i-th point and t-th time of the cross-view reference unit is b (i, t) refers to three of b (i + 1, t + n), b (i-1, t + n), b (i-1, tn), and b (i + 1, tn) , tn is the time at which the I picture, P picture or rB picture was encoded at the closest time before t time, and t + n is the time at which the I picture P picture or rB picture was encoded at the closest time after t time.

In another preferred embodiment of the present invention, an inter-view frame reference method for encoding multi-view video data includes a GOP corresponding to at least one or more viewpoints, and each GOP includes an IDR picture at an initial time at which coding starts. A multi-view GOP array step wherein each view between each GOP has the same coding structure; and except for the base GOP, the IDR picture, and the last view, which perform only temporal frame references within their respective views among the GOPs. And a mutual view reference step of referring to pictures in a spatially close view among pictures of other encoded views.

Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. It should be noted that the same elements among the drawings are denoted by the same reference numerals and symbols as much as possible even though they are shown in different drawings. In the following description of the present invention, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted.

1 is a block diagram of an inter-view frame reference apparatus for encoding multi-view video data according to an embodiment of the present invention.

An inter-view frame reference apparatus for encoding multi-view video data includes a multi-view GOP arranging unit 110, a mutual view reference unit 120, and a same view reference unit 130.

The multi-view GOP arranging unit 110 includes GOPs corresponding to at least one or more viewpoints, respectively. Each GOP includes an Instantaneous Decoder Refresh (IDR) picture at the first time that coding starts and each time point between each GOP has the same coding structure (see Figure 2).

The multi-view GOP arranged in the multi-view GOP arranging unit 100 includes not only the H.264 / AVC GOP (Group of Picture) but also all other views as the view axis, and each GOP is the first in the time axis. Include one IDR picture in time. In the GOP at each viewpoint, encoding is performed in the order of an I picture, a P picture, an rB picture, and a B picture.

A coding structure of a multi-view GOP proposed for inter-view frame reference of multi-view video data in the present invention will be described in detail with reference to FIG. 2.

 The mutual viewpoint reference unit 120 increases the compression ratio by referring to the frames of the viewpoints closest to each other in space and time. More specifically, the cross-view reference unit 120 performs a frame or other time at another point in time at the same time except for a base GOP, an IDR picture, and a picture at the last point, which perform only a temporal frame reference within its viewpoint in the multi-view GOP. Reference the frame at different time points.

 In detail, the pictures referenced by the mutual view reference unit 120 to encode the P picture, the rB (reference B) picture, and the B picture are as follows.

When the P picture to be encoded at the current i-th time and t time is b (i, t), the b- (i-1, t), b (i- Reference is made to at least one of 1, tn) and b (i + 1, tn). In this case, t-n represents the time at which the I picture or the P picture was encoded at the closest time before t time. This will be described in detail with reference to FIG. 3.

If rB pictures to be encoded at the current i-th time and t-th time are b (i, t), b (i + 1, t + n), b in order to encode the P picture in the mutual view reference unit 120 Reference is made to three pictures of (i-1, t + n), b (i-1, t), b (i-1, tn), and b (i + 1, tn). In this case, t-n is a time at which an I picture or a P picture is encoded at the closest time before t hours, and t + n is a time at which an I picture or P picture is encoded at the closest time after t hours. This will be described in detail with reference to FIG. 4.

If the B picture to be encoded at the current i-th point and t-th time is b (i, t), b (i + 1, t + n), b (i-1, t + n), b (i- 1, tn) and b (i + 1, tn), where tn is the time when the I picture, P picture, or rB picture was encoded at the closest time before t hours, and t + n after t hours The time at which the I picture P picture or rB picture is encoded at the nearest time is shown. This will be described in detail with reference to FIG. 4.

In the mutual viewpoint reference unit 120, the structure of the frame reference method of the multi-view video data is differently applied according to the camera type. 3 and 4 illustrate a structure in which a video is photographed with a primary structure, FIGS. 5 and 6 as a secondary structure, and FIG. 7 as a secondary parallel structure.

The same view reference unit 130 refers to an already encoded same view picture. That is, the time picture is referred to before and after within the same time point.

2 illustrates a coding structure of a multi-view GOP proposed for inter-view frame reference of multi-view video data according to an embodiment of the present invention.

In FIG. 2, the x axis represents a viewpoint and the y axis represents time. The multi-view GOP 200 includes at least one H.264 / AVC GOP 210 and includes GOPs corresponding to each view. GOPs have the same coding structure.

The H.264 / AVC GOP only makes temporal frame references within its own viewpoint, not inter-view frame references. Any point in the multiview video data can be selected as an H.264 / AVC GOP.

The coding structure of the GOP at each time point may be I B rB B P B rB B P. As another example, the coding structure of the GOP may be I rB rB rB P rB rB rB P. In this case, rB is characterized in that the B picture available as a reference picture.

3 illustrates an inter-view frame reference scheme for P picture encoding in a camera type of first order parallel and arc structure.

인덱스는 인코딩되는 P 픽처의 순서를 나타낸 것이며, 화살표는 참조되는 시점 간 프레임을 나타낸다. The primary parallel and arc structure is a structure in which a video is photographed with a plurality of cameras positioned in a primary parallel and arc structure. 0 to 9

The index indicates the order of the P pictures to be encoded, and the arrow indicates the inter-view frame referenced.

As an example of an inter-view frame reference in a P picture, the P pictures of the H.264 / AVC GOP and the multiview GOP except for the last view are the same time P picture at the left view and an I picture encoded at the time immediately preceding the left view. Reference is made to the P picture and one I picture or P picture encoded at the time immediately preceding the right view. The last camera refers to the P picture at the same time of the left camera and the I picture or P picture encoded at the time immediately preceding the left camera.

4 illustrates an inter-view frame reference scheme for B picture encoding in a camera type of first order parallel and arc structure. First, the inter-view frame reference of the reference B picture rB located in the temporal center of the I picture and the P picture is as follows.

As an example of an inter-view frame reference for an rB picture, H.264 / AVC GOPs and multiview GOPs except for the last view refer to an rB picture encoded at the same time of the left camera and a P picture located at the next time of the left camera. Refer to the P picture located next time of the right camera. The last camera refers to the B picture encoded at the same time of the left camera and the P picture located at the next time of the left camera.

For more specific example, in the case of 1 index, any of (0, 20, 22, 15, 17) may refer to three indexes arbitrarily (0, 20, 22), (0, 15, 17) .. have.

As another example, the inter-view frame reference of a B picture located between an rB picture and a P picture or an I picture is as follows.

H.264 / AVC GOPs and multiview GOPs, except for the last view, refer to the rB picture encoded at the previous time of the left camera and the P picture or rB picture located at the next time of the left camera, and the P located at the next time of the right camera. See picture or rB picture. The last camera references the reference B picture encoded at the previous time of the left camera and the P picture or reference B picture located at the next time of the left camera.

For example, in the case of 6 indexes, three indexes may be referred to as (0, 2, 15), (0, 15, 17)... Among arbitrary (0, 2, 15, 17).

5 illustrates an inter-view frame reference method for encoding of a P picture and a B picture in a camera type having a quadratic cross structure.

The secondary cross structure is a structure in which a video is recorded with a plurality of cameras positioned in a secondary cross structure. The indices of 0 to 4 indicate the order of pictures to be encoded, and the arrows indicate the inter-view frames referred to.

Multi-view GOPs except H.264 / AVC GOP refer to frame-to-view frames as follows. The centrally located view does not have an already encoded view at the same time, so there is no frame reference between views. The viewpoint located on the left refers to the picture of the center viewpoint encoded at the same time. The viewpoint located above refers to the picture of the center viewpoint and the left viewpoint encoded at the same time. The view located on the right refers to the picture of the center view and the upper view encoded at the same time. The view located at the lower side refers to the picture of the center view and the left and right views encoded at the same time.

As another method of inter-view frame reference, a reference can be performed as follows. Multi-view GOPs except H.264 / AVC GOP refer to frame-to-view frames as follows. The centrally located view does not have an already encoded view at the same time, so there is no frame reference between views. The viewpoint located on the left refers to the picture of the center viewpoint encoded at the same time. The viewpoint located above refers to the picture of the center viewpoint and the left viewpoint encoded at the same time. The view located on the right refers to the picture of the center view and the upper view encoded at the same time. The view located at the lower side refers to the picture of the center view and the left and right views encoded at the same time.

FIG. 6 illustrates another inter-view frame reference scheme for encoding a P picture and a B picture in a camera type having a quadratic cross structure. Unlike the cross structure of FIG. 5, the cross structure may refer to an inter-frame frame reference as the structure of FIG. 6.

That is, multi-view GOPs except H.264 / AVC GOPs refer to inter-frame frames as follows. The centrally located view does not have an already encoded view at the same time, so there is no frame reference between views. The viewpoint located on the left refers to the picture of the center viewpoint encoded at the same time. The view located on the right refers to the picture of the center view and the left view encoded at the same time. The upper viewpoint includes the pictures of the center viewpoint and the left and right viewpoints encoded at the same time. The view located at the lower side refers to the picture of the center view and the left and right views encoded at the same time.

7 is a diagram illustrating an inter-view frame reference method for encoding a P picture and a B picture in a camera type of a quadratic parallel structure (3 × 5).

The second parallel structure is a structure in which a video is recorded with a plurality of cameras positioned in a second parallel structure. In FIG. 7, indices of 0 to 14 indicate the order of pictures to be encoded. The arrows indicate the frames between views that are referenced.

Multi-view GOPs except H.264 / AVC GOP refer to frame-to-view frames as follows. The centrally located view does not have an already encoded view at the same time, so there is no frame reference between views.

The first view refers to the picture of the center view encoded at the same time. The second view refers to pictures of the center view and the left view encoded at the same time. View point 3 refers to the picture of the center view and the upper and left views encoded at the same time. View point 4 refers to pictures of the center view and the left and right views encoded at the same time. Of the viewpoints located in the column located in the center, except for the viewpoints near the center and the center (1,3), the viewpoint 9 on the left side shows the pictures of the upper right, right, and lower sides based on its own viewpoint. See. The viewpoint 10 on the right side refers to the pictures on the upper left, left, and lower left sides of the viewpoint.

With respect to the center, the viewpoints 5 and 11 located on the upper left refer to the pictures on the right, lower right and lower sides with respect to their viewpoints encoded at the same time. The viewpoints 6 and 12 located on the upper right refer to the left, lower left and lower pictures based on their viewpoints encoded at the same time. The lower left viewpoints 7 and 13 refer to pictures on the right, upper right and upper sides with respect to their viewpoints encoded at the same time. The viewpoints 8 and 14 located at the lower right refer to the pictures on the left, upper left and upper sides with respect to their viewpoints encoded at the same time.

8 illustrates a flowchart of an inter-view frame reference method for encoding multi-view video data, according to an embodiment of the present invention.

In the inter-view frame reference method for encoding multi-view video data, a multi-view GOP is first arranged for the same view and cross-view reference, and then reference is made to a picture of a time that is close in time and space among pictures of an encoded view.

Multi-view GOPs used for cross-view reference each include a GOP corresponding to at least one or more viewpoints, wherein each GOP includes an IDR picture at the first time coding starts and each viewpoint between each GOP has the same coding structure. (S810).

In the cross-view reference step (S820), a picture within a spatially close view of the base GOP, the IDR picture, and the pictures of other views that are already encoded except for the last view, which performs only temporal frame references within its own view among each GOP. See in order.

Also, the simultaneous point is referred to for encoding the multiview video (S830). That is, the pixel is encoded before or after the same view. The same view reference and the mutual view reference may be performed simultaneously or in different orders.

9A and 9B illustrate a test environment for experimenting with the effect of the inter-view frame reference method of the present invention.

The spatial-distortion, temporal resolution, camera arrangement and rate-distortion curves for a given bitrate for the multi-view sequence in the test data set given in FIG. 9A are shown in FIGS. 9A and 9B.

10 (a) to 10 (g) show rate-distortion curves based on the experimental environment of FIG.

10A to 10G show rate distortion curves for a multi-view test sequence under the environment set in FIGS. 9A and 9B. 10 (a) to (g), it can be seen that a PSNR gain of 1.2 to 2 dB is obtained at every bitrate in all multi-view test sequences.

The invention can also be embodied as computer readable code on a computer readable recording medium. Computer-readable recording media include all kinds of recording devices that store data that can be read by a computer system.

Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage, and the like, which are also implemented in the form of carrier waves (for example, transmission over the Internet). It also includes. The computer readable recording medium can also be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion.

The best embodiments have been disclosed in the drawings and specification above. Although specific terms have been used herein, they are used only for the purpose of describing the present invention and are not used to limit the scope of the present invention as defined in the meaning or claims.

Therefore, those skilled in the art will understand that various modifications and equivalent other embodiments are possible from this. Therefore, the true technical protection scope of the present invention will be defined by the technical spirit of the appended claims.

According to the present invention, the compression rate is increased by referring to a frame at a point in time that is close in time and space in the multi-view video data. Accordingly, by referring to frames at different time points and at different time points in the same time, a higher compression ratio can be obtained by reducing the residual signal than a method of referring to only temporal or future frames at the same time point.

Claims (27)

A multi-view GOP array unit including GOPs corresponding to at least one viewpoint and arranging each viewpoint between each GOP to have the same coding structure; and And a cross-view reference unit for referring to pictures in a spatially close view among pictures of other viewpoints already encoded in the respective GOPs. The method of claim 1, An inter-view frame reference apparatus for encoding multi-view video data for encoding multi-view video data, further comprising: a same-view reference unit for referring to the same-view picture that is already encoded. The method of claim 1, wherein the base GOP is An inter-view frame reference apparatus for encoding multi-view video data, wherein the frame is H.264 / AVC GOP. The method of claim 1, An inter-view frame reference apparatus for encoding multi-view video data, characterized by encoding in the order of an I picture, a P picture, an rB picture, and a B picture in a GOP of each viewpoint. The method of claim 1, wherein If the P picture to be encoded at the current i th time and t time is b (i, t), among b (i-1, t), b (i-1, tn) and b (i + 1, tn) And at least one reference word, and tn is a time at which an I picture or a P picture is encoded at the closest time before t time. The method of claim 1, wherein the mutual view reference unit B (i + 1), b (i-1, t + n), b (i- 1, t), b (i-1, tn) and b (i + 1, tn) refer to three, and tn is the time when the I picture or P picture was encoded at the closest time before t time, and t + n is an inter-view frame reference apparatus for encoding multi-view video data, wherein the I picture or P picture is encoded at the closest time after time t. The method of claim 1, wherein the mutual view reference unit If the B picture to be encoded at the current i-th point and t-th time is b (i, t), b (i + 1, t + n), b (i-1, t + n), b (i- 1, tn) and b (i + 1, tn), where tn is the time when the I picture, P picture, or rB picture was encoded at the closest time before t hours, and t + n after t hours An inter-view frame reference apparatus for encoding multi-view video data, wherein the I-picture P picture or rB picture is encoded at the closest time. The method of claim 1, wherein the mutual view reference unit In the camera type of the quadratic cross-section and the camera type of the quadratic parallel structure, when the viewpoint and the time are the respective axes, the radius is determined based on the point closest to the center of gravity of the viewpoint and the time of the multi-view GOP on the coordinate axis. An inter-view frame reference apparatus for encoding multi-view video data, wherein the viewpoint located at a close point is first coded. The method of claim 1, wherein the mutual view reference unit An inter-view frame reference apparatus for encoding multi-view video data, characterized by referring to up to three pictures. The method of claim 1, wherein the coding structure of the GOP at each of the time points is I B rB B P B rB B P, in which case rB is a B picture usable as a reference picture. The method of claim 1, wherein the coding structure of the GOP at each of the time points is I rB rB rB P rB rB rB P, in which case rB is a B picture usable as a reference picture. The apparatus of claim 1, wherein each GOP includes an Instantaneous Decoder Refresh (IDR) picture at an initial time at which coding starts. The method of claim 1, The mutual view reference unit refers to a picture within a spatially close view among the base GOP, the IDR picture, and pictures of other views that are already encoded except for the last view, in which only the temporal frame reference is performed within the respective view among the GOPs. An inter-view frame reference apparatus for encoding multi-view video data, characterized in that the. A multiview GOP arrangement step of including a GOP corresponding to at least one viewpoint and arranging each viewpoint between each GOP to have the same coding structure; and And a mutual viewpoint reference step of referring to pictures in a spatially close viewpoint among pictures of other viewpoints already encoded in the respective GOPs. The method of claim 14, An inter-view frame reference method for encoding multi-view video data for encoding multi-view video data, further comprising: a same-view reference step of referring to an already-viewed same view picture. 15. The method of claim 14, wherein the base GOP is An inter-view frame reference method for encoding multi-view video data, characterized in that it is H.264 / AVC GOP. The method of claim 14, An inter-view frame reference method for encoding multi-view video data, characterized by encoding in the order of an I picture, a P picture, an rB picture, and a B picture in a GOP of each viewpoint. 15. The method of claim 14, wherein P picture is b (i-1, t), b (i-1, tn), b (i + 1, tn), wherein tn is a time of encoding an I picture or a P picture at the closest time before t time. 15. The method of claim 14, wherein B (i + 1), b (i-1, t + n), b (i- 1, t), b (i-1, tn) and b (i + 1, tn) refer to three, and tn is the time when the I picture or P picture was encoded at the closest time before t time, and t + n is an inter-view frame reference method for encoding multi-view video data, wherein the I picture or P picture is encoded at the closest time after time t. 15. The method of claim 14, wherein the cross-view reference step is If the B picture to be encoded at the current i-th point and t-th time is b (i, t), b (i + 1, t + n), b (i-1, t + n), b (i- 1, tn) and b (i + 1, tn), where tn is the time when the I picture, P picture, or rB picture was encoded at the closest time before t hours, and t + n after t hours An inter-view frame reference method for encoding multi-view video data, wherein the I-picture P picture or rB picture is encoded at the closest time. 15. The method of claim 14, wherein the cross-view reference step is In the camera type of the quadratic cross-section and the camera type of the quadratic parallel structure, when the viewpoint and the time are the respective axes, the radius is determined based on the point closest to the center of gravity of the viewpoint and the time of the multiview GOP on the coordinate axis. An inter-view frame reference method for encoding multi-view video data, characterized by first coding a viewpoint located at a close point. 15. The method of claim 14, wherein the mutual view reference step is An inter-view frame reference method for encoding multi-view video data, characterized by referring to up to three pictures. 15. The coding structure of claim 14, wherein the coding structure of the GOP at each time point is I B rB B P B rB B P, in which case rB is a B picture usable as a reference picture. 15. The coding structure of claim 14, wherein the coding structure of the GOP at each time point is I rB rB rB P rB rB rB P, in which case rB is a B picture usable as a reference picture. 15. The method of claim 14, wherein each GOP includes an IDR picture at an initial time at which coding starts. 15. The method of claim 14, wherein the cross-view reference step is Wherein, among the GOPs, the base GOP which performs only temporal frame reference within its own view, the IDR picture, and the pictures of other views that are already encoded except for the last view are referred to in the order of pictures within the spatially close view. An inter-view frame reference method for encoding point video data. A computer-readable recording medium having recorded thereon a program for executing the method according to any one of claims 14 to 26.

Images