KR100813064B1 - Method and Apparatus, Data format for decoding and coding of video sequence - Google Patents

Abstract

The present invention relates to a method and apparatus for efficiently decoding / coding a video signal, and to a data format thereof.

A method of decoding a video signal, the method comprising: extracting reference view identification information (view_dependency_flag) from a bitstream, extracting anchor picture identification information (anchor_pic_flag) according to the extracted reference view identification information, and extracting the extracted anchor picture identification It provides a video signal decoding method and apparatus comprising the step of decoding the corresponding bitstream based on the information. It also provides an encoding method through the data format and the inverse process. In the case of random access of a multi-view image through the present invention, it is possible to solve the time delay problem while being compatible with existing H.264 / AVC, thereby enabling efficient decoding / encoding.

Multiview, base views, anchor picture

Description

Method and apparatus for video image decoding / coding, data format {Method and Apparatus, Data format for decoding and coding of video sequence}

1 shows a configuration of a NAL (Network Abstraction Layer) unit of H.264 / AVC.

2 illustrates a multiview sequence encoding and decoding system to which the present invention is applied.

3 illustrates a prediction structure of pictures for explaining an entire encoding process of a multiview video signal to which the present invention is applied.

4 illustrates an example of applying the present invention to adding reference time identification information in a nal unit header on a syntax.

5A is a data format to which the present invention is applied and shows a NAL structure when a current picture corresponds to a reference time point.

5B is a data format to which the present invention is applied and shows a NAL structure when a current picture does not correspond to a reference time point.

6 is a flowchart illustrating a video signal decoding method to which the present invention is applied.

FIG. 7 illustrates an example in which anchor picture identification information is added to a slice layer on a syntax.

8 shows a part of a video signal decoding apparatus to which the present invention is applied.

<Description of the main parts in the drawing>

10: multi-view image generator 20: pre-processing

30: encoder 40: decoder

50: post-processing 60: display unit

61: two-dimensional display 63: stereo type display

65: Display that provides M viewpoints in 3D

501: First NAL header 502: First slice layer

503: second NAL header 504: second slice layer

510: NAL header 520: slice layer

810: First identification information extraction unit 820: Second identification information extraction unit

830: decryption unit

The present invention relates to a method and apparatus for decoding / encoding a video image and a data format thereof.

The mainstream video broadcasting image is a single view image acquired with one camera. Even if the video is taken by multiple cameras, it is edited and treated as a single video. Multi-view video, on the other hand, is a three-dimensional (3D) image processing method that geometrically corrects images taken by more than one camera and provides users with various viewpoints in various directions through spatial synthesis. It is a field. Multi-view video can increase the freedom of view for the user, and has a feature that includes a larger area than the image area that can be acquired using a single camera. Such multi-view video images may be acquired by moving cameras, arranging a plurality of cameras in various directions, or by using a special apparatus such as a reflector.

Recently, researches on a system for encoding, transmitting, decoding, and displaying a multiview image itself taken by multiple cameras have been actively conducted.

Moving Picture Experts Group (MPEG) and Video Coding Experts Group (VCEG) have developed new standards that promise better video image compression performance than the earlier MPEG-4 and H.263 standards. The new standard was named "Advanced Video Coding" and co-published in MPEG-4 Part 10 and ITU-T Recommendation H.264.

Looking at the configuration of the bit stream in H.264 / AVC, the NAL (Network) between the VCL (Video Coding Layer) that handles the video encoding process itself and the subsystem that transmits and stores the encoded information. It is defined as a separate hierarchical structure called Abstraction Layer. The output of the encoding process is VCL data and is mapped in units of NAL before transmission or storage. Each NAL unit includes raw video sequence payload (RBSP), which is data corresponding to compressed video data or header information.

1 shows a configuration of a NAL (Network Abstraction Layer) unit of H.264 / AVC. The NAL unit basically consists of two parts: the NAL header and the RBSP. The NAL header includes flag information (nal_ref_idc) indicating whether a slice serving as a reference picture of the NAL unit is included and an identifier (nal_unit_type) indicating the type of the NAL unit. The RBSP stores the compressed original data and adds an RBSP trailing bit to the end of the RBSP to express the length of the RBSP in multiples of 8 bits.

These NAL unit types include Instantaneous Decoding Refresh (IDR) pictures, Sequence Parameter Set (SPS), Picture Parameter Set (PPS), and Supplemental Enhancement Information (SEI). Etc.

When random access is performed according to the overall coding structure of MVC, a long time delay is a problem because the structure is complicated. Therefore, in order to solve this problem, it is necessary to reduce the minimum number of frames for random access, which may be possible by knowing the correlation between viewpoints in the decoder and defining a new picture type called an anchor picture.

An object of the present invention is to provide a method and apparatus for efficiently decoding and encoding multiview image data, and a data format thereof.

An object of the present invention is to provide a method and apparatus for efficiently encoding and decoding by adding anchor picture identification information or reference point identification information in a standardized manner, and a data format thereof.

An object of the present invention is to efficiently perform random access of a multiview video image by adding anchor picture identification information on a syntax.

An object of the present invention is to maintain compatibility with the existing H.264 / AVC by adding the reference time identification information on the syntax.

In order to achieve the above object, the present invention provides a method of decoding a video signal, extracting reference view identification information (view_dependency_flag) from the bitstream and anchor picture identification information (anchor_pic_flag) according to the extracted reference view identification information; And extracting a corresponding bitstream based on the extracted anchor picture identification information.

The present invention also provides a first identification information extraction unit for extracting reference view identification information from a bitstream, a second identification information extraction unit for extracting anchor picture identification information according to the extracted reference view identification information, and the extracted anchor picture. It provides a video signal decoding apparatus comprising a decoding unit for decoding the corresponding bitstream based on the identification information.

The present invention also provides a data format of a video signal, comprising: a first NAL header including attribute information of a current picture, a first slice layer including data information of the current picture, and a first slice layer sequentially added to the first slice layer; And a second slice layer including identification information on an anchor picture and a second NAL header including identification information on a reference time point.

The present invention also provides a data format of a video signal, comprising: a NAL header including attribute information of a current picture and reference view identification information, and a slice layer including data information and anchor picture identification information of the current picture. Provides a data format of a video signal.

In addition, the present invention provides a video signal encoding method comprising adding reference view identification information to a nal unit header.

In addition, the present invention provides a video signal encoding method characterized by adding anchor picture identification information to a slice layer.

The above objects and features of the construction will become more apparent from the following detailed description taken in conjunction with the accompanying drawings. Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

In addition, the terminology used in the present invention was selected as a general term that is widely used at present, but in certain cases, the term is arbitrarily selected by the applicant, in which case the meaning is described in detail in the corresponding part of the present invention, It is to be understood that the present invention is to be understood as the meaning of a term rather than a name of a term.

2 illustrates a multiview sequence encoding and decoding system to which the present invention is applied.

As shown in FIG. 2, the multi-view image encoding system to which the present invention is applied includes a multi-view image generation unit 10, a preprocessing unit 20, and an encoder 30. In addition, the decoding system includes a decoder 40, a post processing unit 50, and a display unit 60.

In this regard, the multi-view image generator 10 includes an image obtaining apparatus (for example, cameras # 1 to #N) corresponding to the number of multi-views to acquire independent images for each viewpoint. When the multiview image data is input, the preprocessor 20 performs a function of increasing the correlation between the multiview image data through a preprocessing process while solving noise removal and imbalancing problems. In addition, the encoder 30 includes a motion estimation / compensation and a disparity estimation / compensation between views, a bit rate control, a difference image encoding unit, and the like. The encoder 30 may apply a generally known scheme.

In addition, the decoder 40 receives the bitstream encoded by the above-described method, and decodes it in reverse. In addition, the post-processing unit 50 performs a function of increasing the reliability and resolution of the decoded data. Finally, the display unit 60 provides the decoded data to the user in various ways depending on the function of the display, in particular, the ability to process a multi-view image. For example, it is a 2D display 61 providing only planar two-dimensional images, or a stereo type display 63 providing two views as stereoscopic images, or stereoscopic images of M views (2 <M). It may be provided as a display (65).

3 illustrates a prediction structure of pictures for explaining an entire encoding process of a multiview video signal to which the present invention is applied.

As shown in FIG. 3, T0 to T100 on the horizontal axis represent frames according to time, and S0 to S100 on the vertical axis represent frames according to viewpoints, respectively. For example, pictures in T0 refer to images taken by different cameras in the same time zone (T0), and pictures in S0 refer to images in different time zones taken by one camera. In addition, the arrows in the drawings indicate the prediction direction and the order of each picture. For example, a P0 picture at S2 time point in the T0 time zone is a picture predicted from I0, which refers to a P0 picture at S4 time point in the TO time zone. It becomes a picture. It is also a reference picture of the B1 and B2 pictures in the T4 and T2 time zones at the time S2.

In the decoding process of a multiview image, random access between viewpoints is essential. Therefore, access to arbitrary time points should be made possible while minimizing decryption effort. In order to realize efficient random access, it is necessary to explain the concept of an anchor picture. An anchor picture refers to an encoded picture in which all slices refer only to slices in frames of the same time zone. For example, an encoded picture refers to only a slice at another viewpoint and no slice at the current viewpoint. 3, if the I0 picture at the time S0 of the time zone T0 is an anchor picture, all pictures in the same time zone, that is, at another time point in the time zone T0, are also anchor pictures. As another example, if an I0 picture at time S0 in the T8 time zone is an anchor picture, all pictures in the same time zone, that is, at other times in the T8 time zone, are also anchor pictures. Similarly, T16,... All pictures in, T96, T100 are examples of anchor pictures.

After the anchor picture is decoded, all pictures coded in turn are decoded without inter-prediction from the decoded picture prior to the anchor picture.

4 illustrates an example of applying the present invention to adding reference time identification information in a nal unit header on a syntax.

Before describing FIG. 4, it is necessary to explain the concept of base views. We need at least one view sequence to be compatible with the H.264 / AVC decoder. Therefore, it is necessary to define viewpoints that can be independently decoded for fast random access, which is called a base view. This base view serves as a reference for encoding among multi views, which corresponds to a reference view. In MVC (Multiview Video Coding), an image corresponding to a reference time point is encoded by a conventional general video encoding method (MPEG-2, MPEG-4, H.263, H.264, etc.) to form an independent bitstream.

The image corresponding to the reference time point may or may not be compatible with H.264 / AVC. However, an image of a time point compatible with H.264 / AVC is always a reference time point. Therefore, in the present invention, it is necessary to define "view_dependency_flag" in the nal unit header of the MVC NAL as a flag for identifying whether the current picture is included at the reference time point. For example, if view_dependency_flag = 0, it means that the current picture or the current slice is included in the reference view, and if view_dependency_flag ≠ 0, it means that the current picture or the current slice is not included in the reference view. In addition, for a new MVC slice, a new NAL unit type is defined for each slice type, type 22 for non-IDR slices and type 23 for IDR slices (see FIG. 5A for details). By adding the view_dependency_flag, it is possible to determine whether the current picture corresponds to the reference time point when decoding from the received bitstream. Therefore, in determining anchor picture identification information therefrom, it becomes compatible with H.264 / AVC. Hereinafter, the anchor picture identification information indicating whether the current picture corresponds to the anchor picture will be described.

5A is a data format to which the present invention is applied and shows a NAL structure when a current picture corresponds to a reference time point.

The data format to which the present invention is applied includes a first NAL header 501, a first slice layer 502, a second NAL header 503, and a second slice layer 504. The first NAL header 501 includes attribute information of the current picture. For example, there are nal_ref_idc and nal_unit_type, where nal_ref_idc indicates flag information indicating whether a slice serving as a reference picture of a NAL unit is included, and nal_unit_type indicates an identifier indicating a type of a NAL unit. The first slice layer 502 contains compressed result data. The NAL unit consisting of the first NAL header 501 and the first slice layer 502 has a nal_unit_type of 1 or 5. This indicates that the slice is for H.264 / AVC compatibility. For example, when nal_unit_type = 5, this means that the current slice is a slice of an IDR picture. When nal_unit_type = 1, the current slice is a picture other than an IDR picture. It means slice. The IDR (Instantaneous Decoding Refresh) picture is an instant decoded refresh picture and means a first picture of a video sequence. In the IDR picture, all the states necessary for decoding the picture bitstream are initialized.

The second NAL header 503 and the second slice layer 504 are added in sequence to the first slice layer. The second NAL header 503 includes reference time identification information, and this part includes view_dependency_flag for identifying whether the current picture is included in the reference time point. The second slice layer 504 includes only anchor picture identification information. Therefore, it is determined whether the current picture is an anchor picture from anchor_pic_flag. The added NAL unit consisting of the second NAL header 503 and the second slice layer 504 has a nal_unit_type of 22 or 23. This indicates that this is a slice for MVC. For example, when nal_unit_type = 22, this means that the current slice is a picture slice other than IDR picture in MVC, and when nal_unit_type = 23, the current slice is IDR picture in MVC It means slice. A NAL unit with nal_unit_type = 22 follows a NAL unit with nal_unit_type = 1, and a NAL unit with nal_unit_type = 23 follows a NAL unit with nal_unit_type = 5.

5B is a data format to which the present invention is applied and shows a NAL structure when a current picture does not correspond to a reference time point.

The NAL header 510 includes attribute information and reference viewpoint identification information of the current picture. For example, there are nal_ref_idc and nal_unit_type, where nal_ref_idc indicates flag information indicating whether a slice serving as a reference picture of a NAL unit is included, and nal_unit_type indicates an identifier indicating a type of a NAL unit. In addition, view_dependency_flag is included to identify whether the current picture is included in the reference time point. The slice layer 520 includes compressed result data and anchor picture identification information. The NAL unit consisting of the NAL header 510 and the slice layer 520 has a nal_unit_type of 22 or 23. This indicates that the slice is for MVC. For example, nal_unit_type = 22 means that the current slice is a slice of a non-IDR picture in MVC, and nal_unit_type = 23 means that the current slice is an IDR picture in MVC. It means slice. Since the NAL structure can be decoded only in the MVC decoder, that is, it is not necessary to be compatible with H.264 / AVC in case of other viewpoints that do not correspond to the reference viewpoint, so that anchor picture identification information can be included in the slice.

6 is a flowchart illustrating a video signal decoding method to which the present invention is applied.

Reference point identification information for determining whether the current picture or the current slice is included in the base view is extracted from the received bitstream (610), and it is determined whether view_dependency_flag = 0 from the extracted reference point identification information. (620). If view_dependency_flag = 0, this means that the current picture or the current slice is included at the reference time point. In this case, in the new NAL structure in which one NAL 503 and 504 is further added to the NAL 501 and 502 of the current picture, anchor picture identification information is extracted from the added NAL slice layer 504 (630). From the extracted anchor picture identification information, it is determined whether the picture type of the current picture is an anchor picture (650). For example, if anchor_pic_flag = 1, this indicates that the picture type of the current picture is an anchor picture, and if anchor_pic_flag ≠ 1, This indicates that the picture type of the current picture is not an anchor picture. After determining whether the anchor picture is present, the corresponding bitstream is decoded accordingly (660).

In addition, if view_dependency_flag ≠ 0 in the step 620 of determining whether view_dependency_flag = 0, this means that the current picture or the current slice is not included in the reference time point. In this case, the NAL header 510 of the current NAL structure includes attribute information and reference view identification information of the current picture, and the slice layer 520 includes compressed result data and anchor picture identification information. In the NAL structure of the current picture as described above, anchor picture identification information is extracted from the slice layer 520 (640). Since the NAL structure can be decoded only in the MVC decoder, that is, it is not necessary to be compatible with H.264 / AVC in case of other viewpoints that do not correspond to the reference viewpoint, so that anchor picture identification information can be included in the slice. From the extracted anchor picture identification information, it is determined whether the picture type of the current picture is an anchor picture (650). For example, if anchor_pic_flag = 1, this indicates that the picture type of the current picture is an anchor picture, and if anchor_pic_flag ≠ 1, This indicates that the picture type of the current picture is not an anchor picture. After determining whether the anchor picture is present, the corresponding bitstream is decoded accordingly (660).

FIG. 7 illustrates an example in which anchor picture identification information is added to a slice layer on a syntax.

When a multi-view video signal is input, the encoder generates the bitstream according to the syntax, and adds anchor picture identification information in the slice layer function among the syntaxes. In adding anchor picture identification information, compatibility with existing H.264 / AVC needs to be maintained. Therefore, it is necessary to first determine whether view_dependency_flag, which is reference view identification information, is 0 before adding anchor picture identification information. have. Therefore, it is determined whether the picture type of the current picture corresponds to the reference time point by inserting the "if (view_dependency_flag == 0)" portion on the syntax first, and if only the reference time, the anchor picture identification information anchor_pic_flag is transmitted. However, if the picture type of the current picture does not correspond to the reference time point, the slice_header_in_mvc_extension () function and the slice_data_in_mvc_extension () function are called, and the anchor_pic_flag is extracted in the slice_header_in_mvc_extension () function. That is, in case of other viewpoints that do not correspond to the reference viewpoint, the anchor picture identification information may be included in the slice since it is not necessary to be compatible with H.264 / AVC. In this way, it is compatible with H.264 / AVC by discriminating and adding anchor picture identification information after determining whether the reference point is applicable, and when random access is performed on a multiview video image, the image at any point of time can be accessed with minimal decoding. It becomes possible.

8 shows a part of a video signal decoding apparatus to which the present invention is applied.

The apparatus to which the present invention is applied includes a first identification information extracting unit, a second identification information extracting unit, and a decoding unit. The first identification information extracting unit 810 extracts reference view identification information (view_dependency_flag) from the received bitstream. The second identification information extracting unit 820 extracts anchor picture identification information according to the extracted reference view identification information. The decoder 830 decodes the corresponding bitstream based on the extracted anchor picture identification information.

As described above, in the present invention, in the random access of multi-view images obtained from several cameras, the present invention uses the concept of an anchor picture to decode without inter-prediction from a previously decoded picture. This can solve the time delay problem. In addition, by setting the base view (base views) and add the anchor picture identification information in a different way accordingly it can be made compatible with the existing H.264 / AVC. The present invention can utilize such features to more efficiently decode and encode a multiview video signal.

Claims (14)

Obtaining first identification information indicating a type of a current NAL from a video signal; Acquiring, according to the first identification information, second identification information indicating whether a coded picture of a current NAL is an anchor picture; And Decoding the video signal based on the second identification information Video signal decoding method comprising a. The method of claim 1, The second identification information is obtained from the header of the current NAL. The method of claim 1, Acquiring reference viewpoint identification information indicating whether the current picture corresponds to the reference viewpoint, And the second identification information is obtained based on the reference view identification information. A first identification information obtaining unit which obtains first identification information indicating the type of the current NAL from the video signal; A second identification information obtaining unit obtaining second identification information indicating whether a coded picture of a current NAL is an anchor picture according to the first identification information; And A decoder which decodes the video signal based on the second identification information Video signal decoding apparatus comprising a. The method of claim 4, wherein And the second identification information is obtained from a header of the current NAL. The method of claim 4, wherein Further comprising a third identification information acquisition unit for obtaining reference view identification information indicating whether the current picture corresponds to the reference view, And the second identification information is obtained based on the reference view identification information. delete delete delete delete delete delete Adding reference time identification information to the nal unit header Video signal coding method. A video signal encoding method comprising adding anchor picture identification information to a slice layer.

Images