JP6888695B2 - Receiver and receiving method - Google Patents

Description

The present technology relates to a receiving device and a receiving method.

When servicing compressed video over broadcasting, the Internet, etc., the upper limit of the reproducible frame frequency is limited by the decoding ability of the receiver. Therefore, it is necessary for the service side to limit the service to only the low frame frequency service or to provide the high and low frame frequency services at the same time in consideration of the reproduction capability of the widely used receiver.

Receivers are expensive to support high frame frequency services, which is a hindrance to their widespread use. Initially, only inexpensive receivers dedicated to low-frame frequency services were widespread, and if the service side started high-frame frequency services in the future, it would not be possible to watch at all without a new receiver, and the service would become widespread. It becomes an inhibitory factor.

For example, in HEVC (High Efficiency Video Coding), temporal scalability by hierarchically coding the image data of each picture constituting the moving image data has been proposed (see Non-Patent Document 1). On the receiving side, the hierarchy of each picture can be identified based on the temporal ID (temporal_id) inserted in the header of the NAL (Network Abstraction Layer) unit, and selective decoding up to the hierarchy corresponding to the decoding ability becomes possible. ..

Gary J. Sullivan, Jens-Rainer Ohm, Woo-Jin Han, Thomas Wiegand, “Overview of the High Efficiency Video Coding (HEVC) Standard” IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECNOROGY, VOL. 22, NO. 12, pp . 1649-1668, DECEMBER 2012

An object of the present technology is to enable good decoding processing according to the decoding ability on the receiving side.

The concept of this technology is
The image data of each picture constituting the moving image data is classified into a plurality of layers, the image data of each of the classified layers is encoded, and a video stream having the image data of the pictures of the encoded layers is generated. Image coding unit and
The image coding unit includes a transmission unit that transmits a container of a predetermined format including the generated video stream.
There is a transmission device that adds decoding timing information set so that the decoding time interval of the coded image data for each picture becomes smaller as the layer is higher, to the coded image data of the pictures in each layer.

In the present technology, the image coding unit encodes the image data of each picture constituting the moving image data to generate a video stream (encoded stream). In this case, the image data of each picture constituting the moving image data is classified into a plurality of layers and encoded, and a video stream having the image data of the pictures of each layer is generated. In this case, decoding timing information, for example, a decoding time stamp, which is set so that the decoding time interval of the encoded image data for each picture becomes smaller as the layer is higher, is added to the encoded image data of the pictures in each layer.

The transmitter transmits a container in a predetermined format containing the video stream described above. For example, the container may be a transport stream (MPEG-2 TS) adopted in the digital broadcasting standard. Further, for example, the container may be a container of MP4 or another format used for distribution on the Internet.

For example, the image coding unit generates a single video stream having encoded image data of the pictures of each layer, divides a plurality of layers into a predetermined number of layer sets of two or more, and of the pictures of each layer set. Identification information for identifying the affiliation hierarchical set may be added to the coded image data. In this case, for example, the identification information may be a bitstream level specification value, and the higher the hierarchy side, the higher the value.

Further, for example, the image coding unit divides a plurality of layers into a predetermined number of layer sets of two or more, and generates a predetermined number of video streams each having coded image data of a picture of each layer set. May be done. In this case, for example, the image coding unit may add identification information for identifying the belonging layer set to the coded image data of the picture of each layer set. Then, in this case, for example, the identification information may be a bitstream level designation value, and the higher the hierarchy side, the higher the value.

As described above, in the present technology, the decoding timing information set so that the decoding time interval of the coded image data for each picture becomes smaller as the layer is higher is added to the coded image data of the pictures in each layer. .. Therefore, good decoding processing according to the decoding performance is possible on the receiving side. For example, even when the decoding ability is low, it is possible to selectively decode the coded image data of a low-layer picture without causing a buffer failure.

In the present technology, for example, the image coding unit generates a single video stream having coded image data of pictures in each layer, or divides a plurality of layers into two or more predetermined number of layer sets. Then, a predetermined number of video streams having each layered picture coded image data are generated, and the layer of the container is further provided with an information insertion unit for inserting the configuration information of the video stream contained in the container. May be made. In this case, for example, the receiving side can easily grasp the configuration of the video stream based on the configuration information of the video stream included in the container, and can perform appropriate decoding processing.

Further, in the present technology, for example, the transmission unit divides a plurality of layers into a predetermined number of layer sets of two or more, and the priority of the packet for containerizing the coded image data of the pictures of the layer set on the lower layer side is higher. It may be set. In this case, for example, on the receiving side, based on the priority of this packet, only the coded image data of the hierarchical set of pictures according to its own decoding ability can be taken into the buffer.

In addition, other concepts of this technology
Receives a container in a predetermined format including a video stream containing encoded image data of the pictures of each layer obtained by classifying and encoding the image data of each picture constituting the moving image data into a plurality of layers. Equipped with a receiver
Decoding timing information set so that the decoding time interval of the coded image data for each picture becomes smaller as the layer is higher is added to the coded image data of the pictures in each layer.
The encoded image data of the picture in the layer below the predetermined layer selected from the video stream included in the received container is decoded at the decoding timing indicated by the decoding timing information, and the picture in the layer below the predetermined layer is decoded. The receiver further includes a processing unit for obtaining the image data of the above.

In the present technology, a container of a predetermined format is received by the receiving unit. This container contains a video stream having image data of the pictures of each layer obtained by classifying and encoding the image data of each picture constituting the moving image data into a plurality of layers. Decoding timing information, for example, a decoding time stamp, which is set so that the decoding time interval of the encoded image data for each picture becomes smaller as the layer is higher, is added to the encoded image data of the pictures in each layer.

The processing unit decodes the encoded image data of the pictures in the layers below the predetermined layer selected from the video stream included in the receiving container, and obtains the image data of each picture. In this case, the decoding of the coded image data of each picture is
At the latest, each is performed at the decoding timing indicated by the added decoding timing information.

For example, the received container contains a single video stream containing encoded image data of the pictures of each layer, and the plurality of layers are divided into a predetermined number of layer sets of two or more, and the lower layer side. The priority of the packet that containers the encoded image data of the picture of the hierarchical set of is set higher, and the processing unit sets the priority of the picture of the predetermined hierarchical set that is containerized with the packet of the priority selected according to the decoding ability. The encoded image data may be taken into a buffer and decoded.

Further, for example, the received container contains a predetermined number of video streams each having image data of two or more predetermined number of hierarchical sets of pictures obtained by dividing a plurality of layers, and is a processing unit. May be configured to take the encoded image data of a predetermined hierarchical set of pictures of the video stream selected according to the decoding ability into a buffer and decode it.

As described above, in the present technology, a decoding time stamp is added to the coded image data of the pictures in each layer so that the decoding time interval of the coded image data for each picture becomes smaller as the layer is higher. The encoded image data of the pictures in the selected layer below the predetermined layer is performed at the decoding timing indicated by the decoding timing information added to the encoded image data. Therefore, good decoding processing according to the decoding performance becomes possible. For example, even when the decoding ability is low, it is possible to selectively decode the coded image data of a low-layer picture without causing a buffer failure.

In the present technology, for example, a post processing unit that matches the frame rate of the image data of each picture obtained by the processing unit with the display capability may be further provided. In this case, even when the decoding ability is low, it is possible to obtain image data having a frame rate suitable for the high display ability.

In addition, other concepts of this technology
The image data of each picture constituting the moving image data is classified into a plurality of layers, the image data of the pictures of each classified layer is encoded, and a video stream having the image data of the encoded pictures of each layer is obtained. Equipped with an image coding unit to generate
The image coding unit divides the plurality of layers into a predetermined number of layer sets of two or more, inserts a bitstream level specification value into each of the substreams corresponding to each layer set, and inserts a bitstream level specification value.
The level specification value of the bitstream inserted in each of the substreams corresponding to each of the above layer sets is a level value including pictures of all layers included in the layer set below the own layer set. is there.

In the present technology, the image coding unit classifies the image data of each picture constituting the moving image data into a plurality of layers, and the image data of the pictures of each of the classified layers is encoded and encoded. A video stream containing the image data of the pictures in each layer is generated.

In this case, the plurality of layers are divided into a predetermined number of layer sets of two or more, and the level specified value of the bit stream is inserted into each of the substreams corresponding to each layer set. In this case, the level specification value of the bitstream inserted in each of the substreams corresponding to each layer set is a level value including pictures of all layers included in the layer set below the own layer set. ..

For example, the image coding unit may be configured to generate a predetermined number of video streams including each of the substreams corresponding to each layer set. Also, for example, the image coding unit may be configured to generate a single video stream that includes all of the substreams corresponding to each layer set.

As described above, in the present technology, the level specification value of the bitstream is inserted into each of the substreams corresponding to each layer set, and the value includes the pictures of all layers included in the layer set below the own layer set. It is a level value. Therefore, the receiving side of the video stream can easily determine whether or not each substream can be decoded based on the level specified value of the inserted bitstream.

In addition, other concepts of this technology
The image data of each picture constituting the moving image data is classified into a plurality of layers, the image data of the pictures of each classified layer is encoded, and a video stream having the image data of the encoded pictures of each layer is obtained. Equipped with an image coding unit to generate
The image coding unit divides the plurality of layers into a predetermined number of layer sets of two or more, inserts a bitstream level specification value into each of the substreams corresponding to each layer set, and inserts a bitstream level specification value.
The level specification value of the bitstream inserted in each of the substreams corresponding to each of the above layer sets is a level value including pictures of all layers included in the layer sets below the layer set.
A transmitter that sends a container of a predetermined format containing the generated video stream,
In the layer of the above container, flag information indicating that the level specification value of the bitstream inserted into the substream of each layer set is a level value including pictures of all layers included in the layer set below the layer set is added. The transmitter is further provided with an information insertion unit to be inserted.

In the present technology, the image coding unit classifies the image data of each picture constituting the moving image data into a plurality of layers, and the image data of the pictures of each of the classified layers is encoded and encoded. A video stream containing the image data of the pictures in each layer is generated.

In this case, the plurality of layers are divided into a predetermined number of layer sets of two or more, and the level specified value of the bit stream is inserted into each of the substreams corresponding to each layer set. In this case, the level designation value of the bitstream inserted into each of the substreams corresponding to each layer set is a level value including pictures of all layers included in the layer set below the layer set.

The transmitter transmits a container in a predetermined format containing the generated video stream. The information insertion unit tells the layer of the container that the level specification value of the bitstream inserted in the substream of each hierarchy is the level value including the pictures of all the layers included in the hierarchy below the hierarchy. The indicated flag information is inserted.

As described above, in the present technology, on the receiving side, the level specification value of the bitstream inserted into the substream of each layer set is included in the layer set below the layer set by the flag information inserted into the layer of the container. It can be seen that the level value includes the pictures of all the layers. Therefore, on the receiving side, it is not necessary to check the level value including the pictures of all the layers included in each substream below the predetermined layer set by using the level specified value for each layer, and the efficiency of the decoding process becomes unnecessary. It becomes possible to achieve the conversion.

According to this technology, good decoding processing according to the decoding ability becomes possible. The effects described here are not necessarily limited, and may be any of the effects described in the present disclosure.

It is a block diagram which shows the configuration example of the transmission / reception system as an embodiment. It is a block diagram which shows the configuration example of a transmission device. It is a figure which shows an example of the hierarchical coding performed by an encoder. It is a figure which shows the structure example (Syntax) of a NAL unit header, and the content (Semantics) of the main parameter in the structure example. It is a figure for demonstrating the structure of the coded image data of each picture by HEVC. It is a figure which shows an example of encoding, decoding, display order and delay at the time of hierarchical coding. It is a figure which shows the coded stream of the hierarchical coding, and the display expectation (display order) in a designated layer. It is a figure which shows the encoder input order and the display order of a decoder output in a designated layer. It is a figure which shows an example of the encoding timing (decoding timing at the time of decoding) of a picture at the time of hierarchical coding. It is a figure which shows the output example of a single video stream (encoded stream) of an encoder. It is a figure which shows the output example of two video streams (encoded stream) of the base stream (B-stream) and extended stream (E-stream) of an encoder. It is a block diagram which shows the structural example of an encoder. It is a figure which shows an example of the processing flow of an encoder. It is a figure which shows the structural example (Syntax) of a HEVC descriptor (HEVC_descriptor). It is a figure which shows the content (Semantics) of the main information in the structural example of a HEVC descriptor. It is a figure which shows the structural example (Syntax) of the scalability extension descriptor (scalability_extension_descriptor). It is a figure which shows the content (Semantics) of the main information in the structural example of a scalability extension descriptor. It is a figure which shows the structure example (Syntax) of a TS packet. It is a figure which shows the relationship between the level specification value (general_level_idc) of the bit rate included in VPS, and the setting value of "transport_priority" of a TS packet header. It is a block diagram which shows the configuration example of a multiplexer. It is a figure which shows an example of the processing flow of a multiplexer. It is a figure which shows the configuration example of the transport stream TS in the case of delivering by a single stream. It is a figure which shows the specific configuration example of the transport stream TS in the case of delivering by a single stream. It is a figure which shows the configuration example of the transport stream TS in the case of delivering by a plurality of streams (2 streams). It is a figure which shows the specific configuration example of the transport stream TS in the case of delivering by 2 streams. It is a figure which shows the other concrete configuration example of the transport stream TS in the case of delivering by 2 streams. It is a block diagram which shows the configuration example of a receiving device. It is a block diagram which shows the configuration example of a demultiplexer. It is a figure which shows the case which the transport stream TS contains a single video stream (encoded stream). It is a figure which shows the case where two video streams (encoded stream) of a base stream and an extended stream are included in a transport stream TS. It is a figure which shows an example of the processing flow (1 frame) of a demultiplexer. It is a figure which shows an example of the processing flow (2 frames) of a demultiplexer. It is a block diagram which shows the structural example of a decoder. It is a flowchart which shows an example of the decoding processing procedure for every video stream in consideration of the decoder processing capacity in a receiving device. It is a figure which shows the structural example of the post processing part. It is a figure which shows an example of the processing flow of a decoder and a post processing part.

Hereinafter, embodiments for carrying out the invention (hereinafter referred to as “embodiments”) will be described. The explanation will be given in the following order.
1. 1. Embodiment 2. Modification example

<1. Embodiment>
[Transmission / reception system]
FIG. 1 shows a configuration example of the transmission / reception system 10 as an embodiment. The transmission / reception system 10 has a transmission device 100 and a reception device 200.

The transmission device 100 transmits the transport stream TS as a container on a broadcast wave. The transport stream TS includes a video stream in which the image data of each picture constituting the moving image data is classified into a plurality of layers and has encoded data of the image data of the pictures in each layer. In this case, for example, H. 264 / AVC, H. Coding such as 265 / HEVC is applied so that the referenced picture belongs to a self-hierarchy and / or a lower hierarchy than the self-hierarchy.

Hierarchical identification information for identifying the affiliation hierarchy is added to the coded image data of the pictures of each layer for each picture. In this embodiment, hierarchical identification information (“nuh_temporal_id_plus1” meaning temporal_id) is arranged in the header portion of the NAL unit (nal_unit) of each picture. By adding the layer identification information in this way, the receiving side can selectively take out the coded image data of the layer below the predetermined layer and perform the decoding process.

In the transport stream TS, a single video stream having encoded image data of the pictures of each layer, or a plurality of layers are divided into a predetermined number of layer sets of two or more, and the coded images of the pictures of each layer set. Contains a predetermined number of video streams, each with data. Further, the hierarchical information of the hierarchical coding and the configuration information of the video stream are inserted into the transport stream TS. This information is inserted into the transport layer. With this information, the receiving side can easily grasp the hierarchical structure and the stream structure, and can perform appropriate decoding processing.

Further, as described above, a plurality of layers are divided into a predetermined number of layer sets, and the priority of the TS packet (transport stream packet) for containerizing the coded image data of the picture of the lower layer set is set high. To. With this priority, the receiving side can take in only the coded image data of the hierarchical set of pictures according to its own decoding ability into the buffer and process it.

Further, as described above, a plurality of layers are divided into a predetermined number of layer sets, and identification information for identifying the belonging layer set is added to the coded image data of the pictures of each layer set. As this identification information, for example, the level specified value (level_idc) of the bit stream is used, and the higher the hierarchy side, the higher the value.

The receiving device 200 receives the above-mentioned transport stream TS transmitted on the broadcast wave from the transmitting device 100. The receiving device 200 selectively extracts and decodes the coded image data of the layer below the predetermined layer from the video stream included in the transport stream TS according to its own decoding ability, and acquires the image data of each picture. Then, the image is reproduced.

For example, as described above, the transport stream TS may include a single video stream having coded image data of a plurality of layers of pictures. In that case, the encoded image data of a predetermined hierarchical set of pictures containerized with TS packets having a priority selected according to the decoding ability is taken into the buffer and decoded.

Further, for example, as described above, the transport stream TS includes a predetermined number of video streams each having coded image data of two or more predetermined number of hierarchical sets of pictures obtained by dividing a plurality of layers. It may be. In that case, the encoded image data of a predetermined hierarchical set of pictures of the video stream selected according to the decoding ability is taken into the buffer and decoded.

In addition, the receiving device 200 performs post processing to match the frame rate of the image data of each picture obtained by decoding as described above with the display capability. By this post processing, for example, even when the decoding ability is low, it is possible to obtain image data having a frame rate suitable for the high display ability.

"Configuration of transmitter"
FIG. 2 shows a configuration example of the transmission device 100. The transmission device 100 includes a CPU (Central Processing Unit) 101, an encoder 102, a compressed data buffer (cpb: coded picture buffer) 103, a multiplexer 104, and a transmission unit 105. The CPU 101 is a control unit and controls the operation of each unit of the transmission device 100.

The encoder 102 inputs uncompressed moving image data and performs hierarchical coding. The encoder 102 classifies the image data of each picture constituting the moving image data into a plurality of layers. Then, the encoder 102 encodes the image data of the pictures of each of the classified layers, and generates a video stream having the encoded image data of the pictures of each layer. The encoder 102 can be, for example, H.I. 264 / AVC, H. Coding such as 265 / HEVC is performed. At this time, the encoder 102 encodes the referenced picture (referenced picture) so that it belongs to a hierarchy lower than the self-hierarchy and / or the self-hierarchy.

FIG. 3 shows an example of hierarchical coding performed by the encoder 102. This example is an example in which the image data of the pictures in each layer is encoded by being classified into 5 layers from 0 to 4.

The vertical axis shows the hierarchy. 0 to 4 are set as temporary_id (layer identification information) arranged in the header portion of the NAL unit (nal_unit) constituting the coded image data of the pictures in layers 0 to 4, respectively. On the other hand, the horizontal axis indicates the display order (POC: picture order of composition), the left side shows the display time before, and the right side shows the display time after.

FIG. 4 (a) shows a structural example (Syntax) of the NAL unit header, and FIG. 4 (b) shows the contents (Semantics) of the main parameters in the structural example. 0 is required for the 1-bit field of "Forbidden_zero_bit". The 6-bit field of "Nal_unit_type" indicates the NAL unit type. The 6-bit field of "Nuh_layer_id" is assumed to be 0. The 3-bit field of "Nuh_temporal_id_plus1" indicates temporary_id and takes a value (1 to 7) to which 1 is added.

Returning to FIG. 3, each of the rectangular frames indicates a picture, and the numbers indicate the order of the encoded pictures, that is, the encoding order (decoding order on the receiving side). For example, a sub group of pictures is composed of 16 pictures from "2" to "17", and "2" is the first picture in the sub picture group. "1" is a picture of the previous sub picture group. Several of these sub-picture groups are grouped together to form a GOP (Group Of Pictures).

As shown in FIG. 5, the coded image data of the first picture of the GOP is composed of NAL units of AUD, VPS, SPS, PPS, PSEI, SLICE, SSEI, and EOS. On the other hand, the pictures other than the first picture of the GOP are composed of NAL units of AUD, PPS, PSEI, SLICE, SSEI and EOS. VPS, together with SPS, can be transmitted once in a sequence (GOP), and PPS can be transmitted in each picture.

Returning to FIG. 3, the solid arrow indicates the reference relationship of the picture in the coding. For example, the picture of "2" is a P picture and is encoded with reference to the picture of "1". Further, the picture of "3" is a B picture, and is encoded with reference to the pictures of "1" and "2". Similarly, the other pictures are encoded with reference to nearby pictures in display order. The picture in layer 4 is not referenced by other pictures.

The encoder 102 generates a video stream having coded image data of pictures in each layer. For example, the encoder 102 divides a plurality of layers into a predetermined number of layer sets of two or more, and generates a predetermined number of video streams including each of the substreams corresponding to each layer set, or each layer set. Generate a single video stream that contains all of the corresponding substreams.

For example, in the example of hierarchical coding in FIG. 3, when layers 0 to 3 are set as low-level hierarchical groups and layer 4 is set as high-level hierarchical groups and divided into two hierarchical groups, two substreams are used. Exists. That is, there are a sub-stream having coded image data of the pictures of layers 0 to 3 and a sub-stream having coded image data of the pictures of layer 4. In this case, the encoder 102 produces a single video stream containing two substreams, or two video streams containing each of the two substreams.

As described above, the encoder 102 divides a plurality of layers into a predetermined number of layer sets of two or more, and belongs to the coded image data of the pictures of each layer set, regardless of the number of video streams to be generated. Add identification information to identify. In this case, for example, SPS (Sequence Parameter Set) and "general_level_idc", which is a bitstream level specification value included in EPSS (Enhanced Sequence Parameter Set), are used as identification information.

The SPS is a conventionally known NAL unit, and is included in the lowest hierarchical substream, that is, the base substream, for each sequence (GOP). On the other hand, the ECSP is a newly defined NAL unit, and is included in each sequence (GOP) in the substream of the hierarchical group higher than the lowest level, that is, the enhanced substream. The value of "general_level_idc" included in SPS and ESPS is set to be higher in the higher hierarchy side.

Since "sub_layer_level_idc" can be sent by SPS and ESPS for each sublayer, it is also possible to use this "sub_layer_level_idc" as the identification information for identifying the hierarchical set. The above is supplied not only in SPS but also in VPS.

In this case, the value of "general_level_idc" inserted into the SPS and ESPS of the substream of each layer set is a level value including pictures of all layers included in the layer set below the own layer set. For example, in the example of hierarchical coding in FIG. 3, the value of "general_level_idc" inserted into the SPS of the substream of the hierarchical set of layers 0 to 3 is a level value including only the pictures of layers 0 to 3. For example, when the frame rate is 60P, it is set to "level 5.1". Further, for example, in the example of hierarchical coding in FIG. 3, the value of "general_level_idc" inserted into the EPSS of the substream of the hierarchical set of layer 4 is a level value including all the pictures of layers 0 to 4. .. For example, when the frame rate is 120P, it is set to "level 5.2".

FIG. 6 shows an example of encoding, decoding, display order and delay during hierarchical coding. This example corresponds to the hierarchical coding example of FIG. 3 described above. This example shows the case where all layers (all layers) are hierarchically coded at full time resolution. FIG. 6A shows the encoder input. As shown in FIG. 6B, each picture is encoded in the encoding order with a delay of 16 pictures to obtain a coded stream. FIG. 6B also shows a decoder input, and each picture is decoded in the order of decoding. Then, as shown in FIG. 6C, the image data decoded by each picture is obtained in the display order with a delay of 4 pictures.

FIG. 7A shows a coded stream similar to the coded stream shown in FIG. 6B described above, divided into three stages of layers 0 to 2, layer 3, and layer 4. Here, "Tid" indicates temporary_id. FIG. 7B shows display expectations (display order) when each picture in the sub-layers of layers 0 to 2, that is, Tid = 0 to 2, is selectively decoded. Further, FIG. 7C shows display expectations (display order) when each picture in the layers 0 to 3, that is, each picture in the partial layer of Tid = 0 to 3 is selectively decoded. Further, FIG. 7D shows display expectations (display order) in the case of selectively decoding each picture of layers 0 to 4, that is, all layers of Tid = 0 to 4.

In order to decode the coded stream of FIG. 7A according to the decoding ability, a decoding ability having a full time resolution is required. However, when decoding Tid = 0-2, a decoder with a decoding capability of 1/4 should be able to process the encoded full time resolution. Further, when decoding Tid = 0 to 3, a decoder having a decoding capability of 1/2 should be able to process the encoded full time resolution.

However, if the pictures belonging to the lower layers referred to in the layer coding are continuous and they are coded at full timing at the time resolution, the ability of the decoder to partially decode cannot catch up. The period A in FIG. 7 (a) corresponds to this. A decoder that decodes a partial hierarchy of Tid = 0 to 2 or Tid = 0 to 3 is for decoding and displaying with a time axis of 1/4 or 1/2 as shown in the display example. , A period-encoded time resolution is full and continuous pictures cannot be decoded. During that time, cpb becomes an unexpected buffer occupancy of the encoder.

Ta indicates the time required for the decoding process for each picture in the decoder that decodes Tid = 0 to 2. Tb indicates the time required for the decoding process for each picture in the decoder that decodes Tid = 0 to 3. Tc indicates the time required for the decoding process for each picture in the decoder that decodes Tid = 0 to 4 (all layers). The relationship between these times is Ta> Tb> Tc.

Therefore, in this embodiment, the pictures belonging to the lower layer of the layer coding have a large decoding interval for each picture, and the buffer control is performed so that the decoding interval becomes smaller as the layer goes to the higher layer. At that time, the minimum decoder capability for the number of layers is defined. For example, in the example of layer coding in FIG. 3, assuming that the minimum decoding ability is the ability to decode up to layer 2, encoding is performed so that pictures in layers 0 to 2 can be decoded at a time resolution of 1/4 of the five layers. When the time is spaced and the multiplexing is performed by the multiplexer 104 described later, the difference in decoding time is reflected in the value of the decoding time stamp (DTS).

As in the example of layer coding in FIG. 3, when the number of layers is 5, the number of layers is 0 to 4, the interval between pictures belonging to layers 0 to 2 is set to a time interval four times the full resolution, and the interval between pictures belonging to layer 3 is set. Is twice the time interval of the full resolution, and the interval of the pictures belonging to the layer 4 is the time interval of the full resolution.

On the other hand, the encoder 102 prevents the timings of picture encoding (= decoding) from overlapping between layers. That is, when the encoder 102 encodes each picture by the above method, if the encoding timings of the low-level picture and the high-level picture overlap, the encoder 102 refers to the low-level picture from more pictures. Priority is given to encoding, and high-level pictures are set to the same timing. However, since the picture belonging to the highest layer is a non-referenced B picture, it is possible to control the timing so that the picture is decoded and displayed as it is (that is, it is not stored in the dpb (decoded picture buffer)).

FIG. 8A shows the encoder input order (same as FIG. 6A). Further, FIGS. 8 (b) to 8 (d) show the display order (corresponding to PTS as a system layer) (same as FIGS. 7 (b) to (d)).

FIG. 9 shows an example of picture encoding timing (decoding timing at the time of decoding) at the time of hierarchical coding. This example corresponds to the hierarchical coding example of FIG. 3 described above. Then, in this example, the minimum decoding ability is set to the ability to decode up to layer 2. The part underlined by a solid line indicates a picture belonging to one SGP (Sub Group of Picture) (16 pictures of "2" to "17"). Further, the picture shown by the solid line rectangular frame belongs to the current SGP, but the picture shown by the broken line rectangular frame does not belong to the current SGP and does not affect the prediction by the picture belonging to the current SGP.

In this case, the intervals of the pictures belonging to layers 0 to 2, that is, the pictures "2", "3", "4", "11" ... Are set to Ta, which is a time interval four times the full resolution. Further, the intervals of the pictures belonging to the layer 3, that is, "5", "8", "12" ... Are basically Tb, which is twice the time interval of the full resolution.

However, the timing of the picture of "8" is set to the next time interval position in order to avoid overlapping with the timing of the picture of "11". Hereinafter, similarly, the timings of the pictures of "12" and "15" are also adjusted so as to avoid overlapping with the pictures belonging to layers 0 to 2. As a result, the timing of the pictures belonging to the layer 3 is set to be intermediate with the timing of the pictures belonging to the layers 0 to 2.

Further, the intervals of the pictures belonging to the layer 4, that is, "6", "7", "9" ... Are basically Tc, which is a full-resolution time interval. However, as a result of adjusting so as to avoid overlapping with the timing of each picture belonging to layers 0 to 3, the timing of the picture belonging to layer 4 is set to be intermediate with the timing of the picture belonging to layers 0 to 3.

As shown in the figure, the encoding process of 1 SGP worth of pictures (16 pictures of "2" to "17") is performed in 1 SGP period. This indicates that real-time processing is possible even when the encoding interval of the pictures belonging to the lower hierarchy is large as described above.

FIG. 10 shows an output example of the encoder 102. This example is an example in which the encoder 102 outputs a single video stream (encoded stream). This example corresponds to the hierarchical coding example of FIG. 3, and is an example in which each picture is encoded at the timing shown in FIG.

In the video stream, the coded image data of each picture belonging to layers 0 to 4 is arranged in the encoding order (coding order). When decoding this video stream on the receiving side, the referenced pictures (pictures of layers 0 to 3) belonging to the current SGP (thick line frame picture) are decoded and then uncompressed data buffer (dpb: decoded picture buffer). Stay in and prepare for reference from other pictures.

FIG. 11 shows an output example of the encoder 102. In this example, the encoder 102 outputs two video streams (encoded streams), a base strim (B_str) and an extension stream (E_str). This example corresponds to the hierarchical coding example of FIG. 3, and is an example in which each picture is encoded at the timing shown in FIG.

In the base trim (B-stream), the coded image data of each picture belonging to layers 0 to 3 is arranged in the encoding order (coding order). Further, in the extended stream (E-stream), the coded image data of each picture belonging to the layer 4 is arranged in the encoding order (coding order). When decoding this video stream on the receiving side, the referenced pictures (pictures of layers 0 to 3) belonging to the current SGP (thick line frame picture) are decoded and then uncompressed image data buffer (dpb: decoded picture buffer). ) To prepare for reference from other pictures.

FIG. 12 shows a configuration example of the encoder 102. The encoder 102 includes a temporal ID generation unit 121, a buffer delay control unit 122, an HRD (Hypothetical Reference Decoder) setting unit 123, a parameter set / SEI encoding unit 124, a slice encoding unit 125, and a NAL packetizing unit 126. have.

Information on the number of layers is supplied from the CPU 101 to the temporal ID generation unit 121. The temporary ID generation unit 121 generates temporary_id according to the number of layers based on the information of the number of layers. For example, in the hierarchical code example of FIG. 3, temporary_id = 0 to 4 is generated.

The buffer delay control unit 122 is supplied with information on the minimum decoding capability (minimum_target_decoder_level_idc) from the CPU 101, and is also supplied with the temporary_id generated by the temporal ID generation unit 121. The buffer delay control unit 122 calculates "cpb_removal_delay" and "dpb_output_delay" of each picture for each layer.

In this case, by specifying the minimum decoding ability of the target decoder that decodes the number of layers, the encoding timing of the referenced low-level picture and the encoding timing of the high-level picture that is displayed immediately after decoding are determined. (See FIG. 9). This encoding timing has the same meaning as the decoding timing read from the compressed data buffer (cpb: coded picture buffer) on the receiving side.

"Cbp_removal_delay" is determined by reflecting the hierarchy to which the picture belongs. For example, it is assumed that the number of layers is N and the temporary_id (Tid) takes a value in the range of 0 to N-1. Further, the minimum decoding ability is the ability to decode a picture in the hierarchy of temporary_id = K. The buffer delay control unit 122 obtains the picture encoding interval D in each layer by the following mathematical formula (1) and reflects it in "cpb_removal_delay" and "dpb_output_delay".

D = 2 ** (N-1 --K) (Tid <= K)
D = 2 ** (N-1 --Tid) (K <Tid <N --1)
D = Input sequence interval (Tid = N? 1)
... (1)

If the encoding timings overlap in time between layers, the lower layer side is preferentially encoded, and the higher layer side is encoded in the next time slot (timeslot) assigned by the above formula.

The HRD (Hypothetical Reference Decoder) setting unit 123 is supplied with "cpb_removal_delay" and "dpb_output_delay" of the pictures of each layer calculated by the buffer delay control unit 122, and information on the number of streams (Number of streams) from the CPU 101. Is supplied. The HRD setting unit 123 sets the HRD based on this information.

Temporal_id is supplied to the parameter set / SEI encoding unit 124 together with the HRD setting information. The parameter set / SEI encoding unit 124 generates a parameter set such as VPS, SPS (ESPS), and PPS of each layer of pictures and SEI according to the number of streams to be encoded.

For example, a picture timing SEI (Picture timing SEI) including "cpb_removal_delay" and "dpb_output_delay" is generated. Further, for example, a buffering period SEI (Buffereing Perifod SEI) including "initial_cpb_removal_time" is generated. The buffering period SEI is generated corresponding to the first picture (access unit) of the GOP.

The “initial cpb removal time” indicates a time (initial time) to be taken out when decoding the encoded image data of the first picture of the GOP (Group Of Picture) from the compressed data buffer (cpb). The "cpb_removal_delay" is the time to take out the encoded image data of each picture from the compressed data buffer (cpb), and the time is determined together with the "initial_cpb_removal_time". Further, "dpb_output_delay" indicates the time for decoding and entering the uncompressed data buffer (dpb) and then extracting the data.

The slice encoding unit 125 encodes the image data of the pictures of each layer to obtain slice data (slice segment header, slice segment data). The slice decoding unit 125 inserts "ref_idx_l0_active (ref_idx_l1_active)" indicating the index of the prediction destination picture of the "Prediction Unit" into the "slice segment header" as information indicating the state of prediction in the time direction by the frame buffer. As a result, at the time of decoding, the referenced picture is determined together with the hierarchy level indicated by temporary_id. Further, the slice decoding unit 125 inserts the index of the current slice (slice) into the "slice segment header" as "short_term_ref_pic_set_idx" or "it_idx_sps".

The NAL packetizing unit 126 generates coded image data of a picture of each layer based on the parameter set and SEI generated by the parameter set / SEI encoding unit 124 and the slice data generated by the slice encoding unit 125. Outputs a number of video streams (encoded streams) according to the number of streams.

At that time, a temporary_id indicating the hierarchy is added to the NAL unit header for each picture (see FIG. 4). In addition, pictures belonging to the hierarchy indicated by temporary_id are grouped as sublayers, and the bit rate level specification value "Level_idc" for each sublayer is set to "sublayer_level_idc" and inserted into VPS or SPS (ESPS). ..

FIG. 13 shows the processing flow of the encoder 102. The encoder 102 starts the process in step ST1, and then moves on to the process in step ST2. In this step ST2, the encoder 102 sets the number of layers N in the layer coding. Next, in step ST3, the encoder 102 sets the temporary_id of the picture in each layer to 0 to (N-1).

Next, in step ST4, the encoder 102 sets the layer level K that can be decoded by the decoder having the minimum capacity among the target decoders in the range of 0 to N-1. Then, in step ST5, the encoder 102 uses the buffer delay control unit 122 to obtain the picture encoding interval D in each layer by the above mathematical formula (1).

Next, in step ST6, the encoder 102 determines whether or not the encoding timings of the pictures overlap in time between the layers. When the encoding timings overlap, the encoder 102 preferentially encodes the lower layer picture in step ST7, and encodes the higher layer picture at the timing of the next encoding interval D. After that, the encoder 102 moves to the process of step ST8.

When the encoding timings do not overlap in step ST6, the encoder 102 immediately moves to the process of step ST8. In step ST8, the encoder 102 reflects the encoding interval D of the pictures of each layer obtained in step ST5 in "cpb_removal_delay" and "dpb_output_delay", performs HRD setting, parameter set / SEI encoding, and slice encoding, and NAL. Transfer to the multiplexing block as a unit. After that, the encoder 102 ends the process in step ST9.

Returning to FIG. 2, the compressed data buffer (cpb) 103 temporarily stores a video stream containing encoded data of the pictures of each layer generated by the encoder 102. The multiplexer 104 reads the video stream stored in the compressed data buffer 103, converts it into a PES packet, further converts it into a transport packet and multiplexes it, and obtains a transport stream TS as a multiplexed stream.

In this transport stream TS, a single video stream having encoded image data of the pictures of each layer, or a plurality of layers are divided into a predetermined number of layer sets of two or more, and the pictures of each layer set are encoded. Includes a predetermined number of video streams, each with image data. The multiplexer 104 inserts hierarchical information and stream configuration information into the transport stream TS.

The transport stream TS includes PMT (Program Map Table) as PSI (Program Specific Information). In this PMT, there is a video elemental loop (video ES1 loop) that has information related to each video stream. In this video elemental loop, information such as a stream type and a packet identifier (PID) is arranged corresponding to each video stream, and a descriptor describing information related to the video stream is also arranged.

The multiplexer 104 inserts a HEVC descriptor (HEVC_descriptor) as one of the descriptors, and further inserts a newly defined scalability extension descriptor (scalability_extension_descriptor).

FIG. 14 shows a structural example (Syntax) of a HEVC descriptor (HEVC_descriptor). Further, FIG. 15 shows the contents (Semantics) of the main information in the structural example.

The 8-bit field of "descriptor_tag" indicates the descriptor type, and here, it indicates that it is a HEVC descriptor. The 8-bit field of "descriptor_length" indicates the length (size) of the descriptor, and indicates the number of bytes thereafter as the length of the descriptor.

The 8-bit field of "level_idc" indicates the level specification value of the bit rate. Further, when "temporal_layer_subset_flag = 1", there are a 5-bit field of "temporal_id_min" and a 5-bit field of "temporal_id_max". "Temporal_id_min" indicates the value of temporary_id of the lowest hierarchy of the hierarchically coded data contained in the corresponding video stream. "Temporal_id_max" indicates the value of temporary_id of the highest hierarchy of the hierarchically coded data of the corresponding video stream.

The 1-bit field of "level_constrained_flag" is newly defined, and SPS or EPSS exists in the corresponding substream, and the element "general_level_idc" is the temporary_id (hierarchical identification information) included in the substream. Indicates that it has a Level value that includes the following Picture. “1” indicates that SPS or ESPS exists in the corresponding substream, and “general_level_idc” of the element has a level value including pictures below the temporal_id included in the substream. “0” means that one SPS exists in the substream group constituting the target service, and the “general_level_idc” includes not only the substream but also other substreams under the same service. Indicates the level value.

The 3-bit field of "scalability_id" is newly defined and is an ID indicating the scalability assigned to each stream when a plurality of video streams provide a scalable service. “0” indicates a base stream, and “1” to “7” are IDs that increase depending on the degree of scalability from the base stream.

FIG. 16 shows a structural example (Syntax) of a scalability extension descriptor (scalability_extension_descriptor). In addition, FIG. 17 shows the contents (Semantics) of the main information in the structural example.

The 8-bit field of "scalability_extension_descriptor_tag" indicates the descriptor type, and here, it indicates that it is a scalability extension descriptor. The 8-bit field of "scalability_extension_descriptor_length" indicates the length (size) of the descriptor, and indicates the number of bytes thereafter as the descriptor length. The 1-bit field of "extension_stream_existing_flag" is a flag indicating that there is an extension service by another stream. “1” indicates that there is an extended stream, and “0” indicates that there is no extended stream.

The 3-bit field of "extension_type" indicates the type of extension. “001” indicates that the extension is time-wise scalable. “010” indicates that the extension is spatially scalable. “011” indicates that the extension is bit rate scalable.

The 4-bit field of "number_of_streams" indicates the total number of streams involved in the distribution service. The 3-bit field of "scalability_id" is an ID indicating the scalability attached to each stream when a plurality of video streams provide a scalable service. “0” indicates a base stream, and “1” to “7” are IDs that increase depending on the degree of scalability from the base stream. The 8-bit field of "minimum_target_decoder_level_idc" indicates the ability of the decoder targeted by the corresponding stream. This information is used in the receiver to determine whether the assumed decoding timing of the coded picture does not exceed the range of the decoder's picture decoding processing capacity before the decoder decodes the stream.

As described above, in this embodiment, the bit rate level specified value (general_level_idc) and the like included in SPS and ESPS are the affiliation layer sets when a plurality of layers are divided into a predetermined number of layer sets of two or more. It is used as identification information. The value of the level designation value of each layer set is a value corresponding to the frame rate including the picture of this layer group and the picture of all the layer groups on the lower layer side of this layer group.

The multiplexer 104 is set as high as the priority of the TS packet that containers the encoded image data of the pictures of the lower layer set. The multiplexer 104 uses the 1-bit field of "transport_priority" of the TS packet header, for example, when dividing a plurality of layers into a low-layer group and a high-layer group.

FIG. 18 shows a structural example (Syntax) of the TS packet. The 1-bit field of "transport_priority" is set to "1" in the case of a TS packet that containers the encoded image data of the picture of the base layer, that is, the lower layer, and the non-base layer, that is, the higher layer. In the case of a TS packet that containers encoded image data of a layered picture, it is set to "0".

FIG. 19 shows the relationship between the bit rate level specified value (general_level_idc) included in the NAL unit of SPS and ECSP and the set value of “transport_priority” in the TS packet header. On the receiving side, one or both of these pieces of information can be used to separate the coded image data of the lower layer set of pictures and the coded image data of the higher layer set of pictures. It will be possible.

FIG. 20 shows a configuration example of the multiplexer 104. It has a TS priority generation unit 141, a section coding unit 142, a PES packetization unit 143-1 to 143-N, a switch unit 144, and a transport packetization unit 145.

The PES packetizing units 143-1 to 143-N read the video streams 1 to N stored in the compressed data buffer 103, respectively, and generate a PES packet. At this time, the PES packetizing units 143-1 to 143-N add DTS (Decoding Time Stamp) and PTS (Presentation Time Stamp) time stamps to the PES header based on the HRD information of the video streams 1 to N. In this case, "cpu_removal_delay" and "dpb_output_delay" of each picture are referred to, converted into DTS and PTS, respectively, with an accuracy synchronized with the STC (System Time Clock) time, and placed at a predetermined position in the PES header.

The switch unit 144 selectively takes out the PES packet generated by the PES packetizing units 143-1 to 143-N based on the packet identifier (PID) and sends it to the transport packetizing unit 145. The transport packetization unit 145 generates a TS packet containing a PES packet in the payload and obtains a transport stream.

Information on the number of layers and the number of streams is supplied from the CPU 101 to the TS priority generation unit 141. The TS priority generation unit 141 generates the priority of each layer set when a plurality of layers indicated by the number of layers are divided into two or more layer sets. For example, when it is divided into two, a value to be inserted in the 1-bit field of "transport_priority" of the TS packet header is generated (see FIG. 19).

The TS priority generation unit 141 sends the priority information of each layer set to the transport packetization unit 145. The transport packetization unit 145 sets the priority of each TS packet based on this information. In this case, as described above, the priority of the TS packet that containers the encoded image data of the pictures of the lower layer set is set higher.

Information on the number of layers, the number of streams, and the minimum target decoder level (Minimum_target_decoder_level_idc) is supplied to the section coding unit 142 from the CPU 101. Based on this information, the section coding unit 142 generates various section data to be inserted into the transport stream TS, for example, the above-mentioned HEVC descriptor (HEVC_descriptor), scalability extension descriptor (scalability_extension_descriptor), and the like.

The section coding unit 142 sends various section data to the transport packetization unit 145. The transport packetization unit 145 generates a TS packet containing this section data and inserts it into the transport stream TS.

FIG. 21 shows the processing flow of the multiplexer 104. This example is an example of dividing a plurality of layers into two groups, a low layer group and a high layer group. The multiplexer 104 starts processing in step ST11, and then moves on to processing in step ST12. In this step ST12, the multiplexer 104 sets the temporary_id_ of each picture of the video stream (video elementary stream) and the number of coded streams to be composed.

Next, in step ST13, the multiplexer 104 sets the “transport_priority” when multiplexing the low-layer set of pictures or the video stream including the low-layer set of pictures to “1”. Further, in step ST14, the multiplexer 104 determines the DTS and PTS with reference to the HRD information (cpu_removal_delay, dpb_output_delay) and inserts them into the PES header.

Next, the multiplexer 104 determines in step ST15 whether or not it is a single stream (single video stream). When it is a single stream, the multiplexer 104 decides to proceed with the multiplexing process with one PID (packet identifier) in step ST16, and then moves to the process of step ST17. On the other hand, when it is not a single stream, the multiplexer 104 decides to proceed with the multiplexing process with a plurality of packet PIDs (packet identifiers) in step ST18, and then proceeds to the process of step ST17.

In this step ST17, the multiplexer 104 codes a HEVC descriptor, a scalability extension descriptor, and the like. Then, in step ST19, the multiplexer 104 inserts the video stream into the PES payload to form a PES packet, and then in step ST20, transport packetizes the video stream to obtain a transport stream TS. After that, the multiplexer 104 ends the process in step ST21.

FIG. 22 shows a configuration example of the transport stream TS in the case of distribution by a single video stream. This transport stream TS includes a single video stream. That is, in this configuration example, a video stream PES packet "video PES1" having, for example, HEVC-encoded image data of a plurality of layers of pictures exists, and an audio stream PES packet "audio PES1" exists.

This single video stream includes a predetermined number of substreams obtained by dividing a plurality of layers of hierarchical coding into a predetermined number of hierarchical sets of two or more. Here, the substream (base substream) of the lowest hierarchy group includes SPS, and the substream of the hierarchy group higher than the lowest level (enhanced substream) includes EPSS. Then, the value of "general_level_idc" of the elements of SPS and EPSS is set to the level value including the pictures of all the layers included in the hierarchy group below the own hierarchy set.

The coded image data of each picture includes NAL units such as VPS, SPS, EPSS, and SEI. As described above, a temporary_id indicating the hierarchy of the picture is inserted in the header of the NAL unit of each picture. Further, for example, SPS and EPSS include a bit rate level specified value (general_level_idc). Further, for example, the picture timing SEI (Picture timing SEI) includes "cpb_removal_delay" and "dpb_output_delay".

In the header of the TS packet that containers the encoded image data of each picture, there is a field indicating the priority of 1 bit of "transport_priority". By this "transport_priority", it is possible to identify whether the coded image data to be containerized is a picture of a low-layer group or a picture of a high-layer group.

Further, the transport stream TS includes PMT (Program Map Table) as PSI (Program Specific Information). This PSI is information describing which program each elementary stream included in the transport stream belongs to.

In PMT, there is a program loop that describes information related to the entire program. In addition, the PMT has an elemental loop having information related to each elemental stream. In this configuration example, there is a video elemental loop (video ES1 loop) and an audio elemental loop (audio ES1 loop).

In the video elemental loop, information such as the stream type and packet identifier (PID) is arranged corresponding to the video stream (video PES1), and a descriptor that describes the information related to the video stream is also arranged. To. As one of the descriptors, the above-mentioned HEVC descriptor (HEVC_descriptor) and the scalability extension descriptor (scalability_extension_descriptor) are inserted.

FIG. 23 shows a case where the base substream (B stream) is generated in the pictures of the layers 0 to 3 and the enhanced substream (E stream) is generated in the pictures of the layer 4 in the example of the hierarchical coding of FIG. Shown. In this case, each picture included in the base substream constitutes 60P, and each picture included in the enhanced substream (E stream) is added to each picture included in the base substream to form 120P in the entire PES. ..

The base substream picture is composed of NAL units such as "AUD", "VPS", "SPS", "PPS", "PSEI", "SLICE", "SSEI", and "EOS". The "VPS" and "SPS" are inserted into, for example, the first picture of the GOP. The value of "general_level_idc" of the SPS element is "level5.1". It should be noted that "EOS" may not be present.

On the other hand, the enhanced substream picture is composed of NAL units such as "AUD", "ESPS", "PPS", "PSEI", "SLICE", "SSEI", and "EOS". In addition, "ESPS" is inserted in the first picture of GOP, for example. The value of "general_level_idc" of the element of ECSP is set to "level5.2". It should be noted that "PSEI", "SSEI", and "EOS" may be omitted.

In the "video ES1 loop", information such as the stream type and packet identifier (PID) is arranged corresponding to the video stream (video PES1), and a descriptor that describes the information related to the video stream is also arranged. To. This stream type is set to "0x24" indicating the base stream. Further, as one of the descriptors, the above-mentioned HEVC descriptor is inserted.

The 1-bit field of "level_constrained_flag" is set to "1". This indicates that "SPS or ESPS exists in the relevant substream, and the" general_level_idc "of the element has a level value including the pictures below the temporary_id included in the substream." The value of "level_idc" is "level5.2", which indicates the overall level value of the video stream (video PES1). Further, "temporal_id_min" is set to 0, "temporal_id_max" is set to 4, and it is shown that the video stream (video PES1) contains pictures of layers 0 to 4.

When distribution by such a single video stream is performed, on the receiving side, each substream is within the range of its own decoder processing capacity based on "level_constrained_flag", "general_level_idc" of SPS, EPSS elements, and the like. Whether or not it is determined, and the substream within the range is decoded.

FIG. 24 shows a configuration example of the transport stream TS in the case of distribution by a plurality of streams, here, two streams. This transport stream TS contains two video streams. That is, in this configuration example, a plurality of layers are divided into two layers, a low layer group and a high layer group, and a PES packet "video" of a video stream having pictures of the two layers, for example, encoded image data by HEVC. Along with the existence of "PES1" and "video PES2", the PES packet "audio PES1" of the audio stream exists.

The two video streams include each of two substreams obtained by dividing a plurality of layers of hierarchical coding into two hierarchical sets. Here, the substream (base substream) of the lower hierarchical group includes SPS, and the substream of the upper hierarchical group (enhanced substream) includes EPSS. Then, the value of "general_level_idc" of the elements of SPS and EPSS is set to the level value including the pictures of all the layers included in the hierarchy group below the own hierarchy set.

NAL units such as SPS and ESPS are present in the coded image data of each picture. As described above, a temporary_id indicating the hierarchy of the picture is inserted in the header of the NAL unit of each picture. Further, for example, SPS and EPSS include a bit rate level specified value (general_level_idc). Further, for example, the picture timing SEI (Picture timing SEI) includes "cpb_removal_delay" and "dpb_output_delay".

In addition, there is a field indicating the priority of 1 bit of "transport_priority" in the header of the TS packet that containers the encoded image data of each picture. By this "transport_priority", it is possible to identify whether the coded image data to be containerized is a picture of a low-layer group or a picture of a high-layer group.

Further, the transport stream TS includes PMT (Program Map Table) as PSI (Program Specific Information). This PSI is information describing which program each elementary stream included in the transport stream belongs to.

In PMT, there is a program loop that describes information related to the entire program. In addition, the PMT has an elemental loop having information related to each elemental stream. In this configuration example, there are two video elemental loops (video ES1 loop, video ES2 loop) and an audio elemental loop (audio ES1 loop).

In each video elemental loop, information such as stream type and packet identifier (PID) is arranged corresponding to the video stream (video PES1, video PES2), and information related to the video stream is described. Descriptors are also placed. As one of the descriptors, the above-mentioned HEVC descriptor (HEVC_descriptor) and the scalability extension descriptor (scalability_extension_descriptor) are inserted.

FIG. 25 shows a case where the base substream (B stream) is generated in the pictures of the layers 0 to 3 and the enhanced substream (E stream) is generated in the pictures of the layer 4 in the example of the hierarchical coding of FIG. Shown. In this case, each picture included in the base substream constitutes 60P, and each picture included in the enhanced substream (E stream) is added to each picture included in the base substream to form 120P in the entire PES. ..

The base substream picture is composed of NAL units such as "AUD", "VPS", "SPS", "PPS", "PSEI", "SLICE", "SSEI", and "EOS". Note that "VPS" and "SPS" are inserted into, for example, the first picture of the GOP. The value of "general_level_idc" of the SPS element is "level5.1". It should be noted that "EOS" may not be present.

On the other hand, the enhanced substream picture is composed of NAL units such as "AUD", "ESPS", "PPS", "PSEI", "SLICE", "SSEI", and "EOS". In addition, "ESPS" is inserted in the first picture of GOP, for example. The value of "general_level_idc" of the element of ECSP is set to "level5.2". It should be noted that "PSEI", "SSEI", and "EOS" may be omitted.

In the "video ES1 loop", information such as the stream type and packet identifier (PID) is arranged corresponding to the video stream (video PES1), and a descriptor that describes the information related to the video stream is also arranged. To. This stream type is set to "0x24" indicating the base stream. Further, as one of the descriptors, the above-mentioned HEVC descriptor is inserted.

The 1-bit field of "level_constrained_flag" is set to "1". This indicates that "SPS or ESPS exists in the relevant substream, and the" general_level_idc "of the element has a level value including the pictures below the temporary_id included in the substream." The value of "level_idc" is "level5.1" indicating the level value of the base substream (B stream). Further, "temporal_id_min" is set to 0, "temporal_id_max" is set to 3, and it is shown that the base substream (B stream) contains pictures of layers 0 to 3.

In the "video ES2 loop", information such as the stream type and packet identifier (PID) is arranged corresponding to the video stream (video PES2), and a descriptor that describes the information related to the video stream is also arranged. To. This stream type is set to "0x25" indicating an enhanced stream. Further, as one of the descriptors, the above-mentioned HEVC descriptor is inserted.

The 1-bit field of "level_constrained_flag" is set to "1". This indicates that "SPS or ESPS exists in the relevant substream, and the" general_level_idc "of the element has a level value including the pictures below the temporary_id included in the substream." The value of "level_idc" is set to "level5.2" indicating the level values of the base substream (B stream) and the enhancement stream (E stream). Further, "temporal_id_min" is set to 4, "temporal_id_max" is set to 4, and it is shown that the enhanced stream (E stream) contains the picture of the layer 4.

When distribution by such multiple video streams is performed, on the receiving side, is each substream within the range of its own decoder processing capacity based on "level_constrained_flag", "general_level_idc" of SPS, EPSS elements, and the like? Whether or not it is determined, and the substream within the range is decoded.

FIG. 26 shows a case where the base substream (B stream) is generated in the pictures of the layers 0 to 3 and the enhanced substream (E stream) is generated in the pictures of the layer 4 in the example of the hierarchical coding of FIG. Another configuration example of the transport stream TS is shown. In this case, each picture included in the base substream constitutes 60P, and each picture included in the enhanced substream (E stream) is added to each picture included in the base substream to form 120P in the entire PES. ..

The base substream picture is composed of NAL units such as "AUD", "VPS", "SPS", "PPS", "PSEI", "SLICE", "SSEI", and "EOS". Note that "VPS" and "SPS" are inserted into, for example, the first picture of the GOP. The value of "general_level_idc" of the SPS element is "level5.2". In this case, the "sub_layer_level_present_flag" of the SPS element is set to "1", and the level value "level5.1" of the base substream is indicated by "sublayer_level_idc [3]". It should be noted that "EOS" may not be present.

The enhanced substream picture is composed of NAL units such as "AUD", "PPS", and "SLICE". However, there is no "ESPS" NAL unit as shown in FIG.

In the "video ES1 loop", information such as the stream type and packet identifier (PID) is arranged corresponding to the video stream (video PES1), and a descriptor that describes the information related to the video stream is also arranged. To. This stream type is set to "0x24" indicating the base stream. Further, as one of the descriptors, the above-mentioned HEVC descriptor is inserted.

There is no "level_constrained_flag" as in FIG. 25. The value of "level_idc" is set to "level5.1" indicating the level value of the base substream (B stream). Further, "temporal_id_min" is set to 0, "temporal_id_max" is set to 3, and it is shown that the base substream (B stream) contains pictures of layers 0 to 3.

In the "video ES2 loop", information such as the stream type and packet identifier (PID) is arranged corresponding to the video stream (video PES2), and a descriptor that describes the information related to the video stream is also arranged. To. This stream type is set to "0x25" indicating an enhanced stream. Further, as one of the descriptors, the above-mentioned HEVC descriptor is inserted.

There is no "level_constrained_flag" as in FIG. 25. The value of "level_idc" is set to "level5.2" indicating the level value of the base substream (B stream) and the enhancement stream (E stream). Further, "temporal_id_min" is set to 4, "temporal_id_max" is set to 4, and it is shown that the enhanced stream (E stream) contains the picture of the layer 4.

When distribution by such multiple video streams is performed, on the receiving side, whether or not each substream is within the range of its own decoder processing capacity based on the SPS elements "general_level_idc", "sublayer_level_idc", etc. Is determined, and the substream within the range is decoded.

Returning to FIG. 2, the transmission unit 105 modulates the transport stream TS by a modulation method suitable for broadcasting such as QPSK / OFDM, and transmits an RF modulated signal from the transmission antenna.

The operation of the transmission device 100 shown in FIG. 2 will be briefly described. Uncompressed moving image data is input to the encoder 102. In the encoder 102, hierarchical coding is performed on the moving image data. That is, in the encoder 102, the image data of each picture constituting the moving image data is classified into a plurality of layers and encoded, and a video stream having the encoded image data of the pictures of each layer is generated. At this time, the referenced picture is encoded so that it belongs to the self-hierarchy and / or the hierarchy lower than the self-hierarchy.

The encoder 102 generates a video stream having coded image data of pictures in each layer. For example, in the encoder 102, a plurality of layers are divided into a predetermined number of layer sets of two or more, and a predetermined number of video streams including each of the substreams corresponding to each layer set are generated, or each layer set is generated. A single video stream containing all of the corresponding substreams is generated.

In the encoder 102, identification information for identifying the belonging layer set is added to the coded image data of the picture of each layer set. In this case, for example, "general_level_idc", which is an element of SPS and EPSS, is used as the identification information. The SPS is included in the substream (base substream) of the lowest hierarchical set for each sequence (GOP). On the other hand, ECSP is included in each sequence (GOP) in the substream (enhanced substream) of the hierarchical set higher than the lowest. The value of "general_level_idc" included in SPS and ESPS is set to be higher in the higher hierarchy side. For example, the value of "general_level_idc" inserted into the SPS and ESPS of the substream of each layer set is a level value including pictures of all layers included in the layer set below the own layer set.

The video stream including the coded data of the pictures of each layer generated by the encoder 102 is supplied to the compressed data buffer (cpb) 103 and temporarily stored. In the multiplexer 104, the video stream stored in the compressed data buffer 103 is read out, converted into a PES packet, further converted into a transport packet and multiplexed, and the transport stream TS as a multiplexed stream is obtained.

The transport stream TS includes a single video stream having encoded image data of pictures in each layer, or two or more predetermined number of video streams. In the multiplexer 104, hierarchical information and stream configuration information are inserted into the transport stream TS. That is, in the multiplexer 104, the HEVC descriptor (HEVC_descriptor) and the scalability extension descriptor (scalability_extension_descriptor) are inserted into the video elemental loop corresponding to each video stream.

Further, in the multiplexer 104, the priority of the TS packet that containers the encoded image data of the pictures of the lower layer set is set higher. In the multiplexer 104, for example, when a plurality of layers are divided into a low-layer group and a high-layer group, the 1-bit field of "transport_priority" of the TS packet header is used to set the priority.

The transport stream TS generated by the multiplexer 104 is sent to the transmission unit 105. In the transmission unit 105, the transport stream TS is modulated by a modulation method suitable for broadcasting such as QPSK / OFDM, and an RF modulation signal is transmitted from the transmission antenna.

"Receiver configuration"
FIG. 27 shows a configuration example of the receiving device 200. The receiving device 200 includes a CPU (Central Processing Unit) 201, a receiving unit 202, a demultiplexer 203, and a compressed data buffer (cpb: coded picture buffer) 204. Further, the receiving device 200 has a decoder 205, an uncompressed data buffer (dpb: decoded picture buffer) 206, and a post processing unit 207. The CPU 201 constitutes a control unit and controls the operation of each unit of the receiving device 200.

The receiving unit 202 demodulates the RF modulated signal received by the receiving antenna and acquires the transport stream TS. The demultiplexer 203 selectively extracts the encoded image data of the layered picture according to the decoding capability (Decoder temporal layer capability) from the transport stream TS and sends it to the compressed data buffer (cpb: coded picture buffer) 204. ..

FIG. 28 shows a configuration example of the demultiplexer 203. The demultiplexer 203 includes a PCR extraction unit 231, a time stamp extraction unit 232, a section extraction unit 233, a TS priority extraction unit 234, a PES payload extraction unit 235, and a picture selection unit 236.

The PCR extraction unit 231 extracts PCR from a TS packet containing PCR (Program Clock Reference) and sends it to CPU 201. The time stamp extraction unit 232 extracts the time stamps (DTS, PTS) inserted in the PES header for each picture and sends them to the CPU 201. The section extraction unit 233 extracts section data from the transport stream TS and sends it to the CPU 201. This section data includes the above-mentioned HEVC descriptor (HEVC_descriptor), scalability extension descriptor (scalability_extension_descriptor), and the like.

The TS priority extraction unit 234 extracts the priority information set in each TS packet. As described above, this priority is the priority of each layer group when a plurality of layers are divided into a predetermined number of layer groups of two or more, and is set higher as the layer group on the lower layer side. For example, when it is divided into a low-layer group and a high-level group, the value of the 1-bit field of "transport_priority" of the TS packet header is extracted. This value is set to "1" in the low-level group and "0" in the high-level group.

The PES payload extraction unit 235 extracts the PES payload, that is, the encoded image data of the pictures of each layer from the transport stream TS. The picture selection unit 236 selectively extracts the coded image data of the layered pictures according to the decoding capability (Decoder temporal layer capability) from the coded image data of the pictures of each layer taken out by the PES payload extraction unit 235. , Send to compressed data buffer (cpb: coded picture buffer) 204. In this case, the picture selection unit 236 refers to the hierarchical information, the stream configuration information, and the priority information extracted by the TS priority extraction unit 234 obtained by the section extraction unit 233.

For example, consider a case where the frame rate of the video stream (encoded stream) included in the transport stream TS is 120 fps. For example, it is assumed that a plurality of layers are divided into a lower layer side layer group and a higher layer side layer group, and the frame rate of the picture of each layer group is 60 fps. For example, in the hierarchical coding example shown in FIG. 3 described above, layers 0 to 3 are set on the lower layer side, and layer 4 is set as the layer set on the higher layer side.

The 1-bit field of "transport_priority" included in the header of the TS packet is set to "1" in the case of the TS packet that containers the encoded image data of the picture of the base layer, that is, the lower layer set, and is non- It is set to "0" in the case of a TS packet that containers the encoded image data of the base layer, that is, the picture of the higher layer set.

In this case, the transport stream TS may include a single video stream (encoded stream) (see FIG. 10) having encoded data of the pictures of each layer. Further, in this case, the transport stream TS has a base stream (B-stream) having encoded image data of the lower layer set of pictures and an enhancement having the encoded image data of the higher layer side layer set of pictures. There is a case (see FIG. 11) in which two video streams (encoded streams) of a stream (E-stream) are included.

For example, when the decoding ability corresponds to 120P (120 fps), the picture selection unit 236 takes out the encoded image data of the pictures of all layers and sends them to the compressed data buffer (cpb) 204. On the other hand, when the decoding ability does not correspond to 120P but corresponds to 60P (60fps), for example, the picture selection unit 236 extracts and compresses only the encoded image data of the pictures in the lower layer set. It is sent to the data buffer (cpb) 204.

FIG. 29 shows a case where the transport stream TS includes a single video stream (encoded stream). Here, "High" indicates a picture of a hierarchical group on the high-level side, and "Low" indicates a picture of a hierarchical group on the low-level side. Further, "P" indicates "transport_priority".

When the decoding ability corresponds to 120P, the picture selection unit 236 takes out the coded image data of the pictures of all layers, sends them to the compressed data buffer (cpb) 204, and stores them in the area 1 (cpb_1). On the other hand, when the decoding ability does not correspond to 120P but corresponds to 60P, filtering based on "transport_priority" is performed to extract only the pictures of the lower layer set with P = 1 and compressed data. It is sent to the buffer (cpb) 204 and accumulated in the area 1 (cpb_1).

FIG. 30 shows a case where the transport stream TS includes two video streams (encoded streams), a base stream and an extended stream. Here, "High" indicates a picture of a hierarchical group on the high-level side, and "Low" indicates a picture of a hierarchical group on the low-level side. Further, "P" indicates "transport_priority". Further, it is assumed that the packet identifier (PID) of the base stream is PID1 and the packet identifier (PID) of the extended stream is PID2.

When the decoding ability corresponds to 120P, the picture selection unit 236 takes out the coded image data of the pictures of all layers and sends them to the compressed data buffer (cpb) 204. Then, the coded image data of the pictures of the lower layer set is stored in the area 1 (cpb_1), and the coded image data of the pictures of the lower layer side is stored in the area 2 (cpb_2).

On the other hand, when the decoding ability does not correspond to 120P but corresponds to 120P, filtering is performed based on the packet identifier (PID), and only the pictures of the lower layer set which is PID1 are extracted and compressed data. It is sent to the buffer (cpb) 204 and accumulated in the area 1 (cpb_1). In this case as well, filtering based on "transport_priority" may be performed.

FIG. 31 shows an example of the processing flow of the demultiplexer 203. This processing flow shows the case where the transport stream TS contains a single video stream (encoded stream).

The demultiplexer 203 starts the process in step ST31, and then moves to the process in step ST32. In this step ST32, the decoding capability (Decoder temporal layer capability) is set from the CPU 201. Next, the demultiplexer 203 determines in step ST33 whether or not it has the ability to decode all layers.

When the demultiplexer 203 has the ability to decode all layers, in step ST34, the demultiplexer 203 demultiplexes all TS packets passing through the corresponding PID filter and performs section parsing. After that, the demultiplexer 203 moves to the process of step ST35.

When the demultiplexer 203 does not have the ability to decode all layers in step ST33, the demultiplexer 203 demultiplexes the TS packet whose “transport_priority” is “1” in step ST36 and performs section parsing. After that, the demultiplexer 203 moves to the process of step ST35.

In step ST35, the demultiplexer 203 reads the HEVC descriptor (HEVC_descriptor), the scalability_extension_descriptor (scalability_extension_descriptor) in the section of the target PID, and determines the presence / absence of the extension stream, the scalable type, the number and ID of the streams, and so on. Get the maximum, minimum, and minimum target decoder levels for temporal_id.

Next, in step ST37, the demultiplexer 203 transfers the coded stream to be PID to the compressed data buffer (cpb) 204, and notifies the CPU 201 of the DTS and PTS. The demultiplexer 203 ends the process in step ST38 after the process in step ST37.

FIG. 32 shows an example of the processing flow of the demultiplexer 203. This processing flow shows the case where the transport stream TS includes two video streams (encoded streams), a base stream and an extended stream.

The demultiplexer 203 starts the process in step ST41, and then moves to the process in step ST42. In this step ST42, the decoding capability (Decoder temporal layer capability) is set from the CPU 201. Next, the demultiplexer 203 determines in step ST43 whether or not it has the ability to decode all layers.

When the demultiplexer 203 has the ability to decode all layers, in step ST44, the demultiplexer 203 demultiplexes all TS packets passing through the corresponding PID filter and performs section parsing. After that, the demultiplexer 203 moves to the process of step ST45.

When the demultiplexer 203 does not have the ability to decode all layers in step ST43, the demultiplexer 203 demultiplexes the TS packet with PID = PID1 and performs section parsing in step ST46. After that, the demultiplexer 203 moves to the process of step ST45.

In step ST45, the demultiplexer 203 reads the HEVC descriptor (HEVC_descriptor), the scalability_extension_descriptor (scalability_extension_descriptor) in the section of the target PID, and determines the presence / absence of the extension stream, the scalable type, the number and ID of the streams, and so on. Get the maximum, minimum, and minimum target decoder levels for temporal_id.

Next, in step ST47, the demultiplexer 203 transfers the coded stream to be PID to the compressed data buffer (cpb) 204, and notifies the CPU 201 of the DTS and PTS. The demultiplexer 203 ends the process in step ST48 after the process in step ST47.

Returning to FIG. 27, the compressed data buffer (cpb) 204 temporarily stores the video stream (encoded stream) taken out by the demultiplexer 203. The decoder 205 extracts the encoded image data of the picture of the layer designated as the layer to be decoded from the video stream stored in the compressed data buffer 204. Then, the decoder 205 decodes the encoded image data of each of the extracted pictures at the decoding timing of the picture, and sends the encoded image data to the uncompressed data buffer (dpb) 206.

Here, in the decoder 205, the hierarchy to be decoded from the CPU 201 is specified by temporary_id. This designated layer is all layers included in the video stream (encoded stream) taken out by the demultiplexer 203, or a part of the lower layer side, and is set automatically by the CPU 201 or according to the user operation. To. Further, the decoder 205 is given a decoding timing from the CPU 201 based on the DTS (Decoding Time stamp). When decoding the coded image data of each picture, the decoder 205 reads out the image data of the referenced picture from the uncompressed data buffer 206 and uses it as necessary.

FIG. 33 shows a configuration example of the decoder 205. The decoder 205 has a temporal ID analysis unit 251, a target layer selection unit 252, a stream integration unit 253, and a decoding unit 254. The temporary ID analysis unit 251 reads the video stream (encoded stream) stored in the compressed data buffer 204, and analyzes the temporary_id inserted in the NAL unit header of the encoded image data of each picture.

The target layer selection unit 252 extracts the encoded image data of the picture of the layer designated as the layer to be decoded based on the analysis result of the temporal ID analysis unit 251 from the video stream read from the compressed data buffer 204. .. In this case, the target layer selection unit 252 outputs a single or a plurality of video streams (encoded streams) according to the number of video streams read from the compressed data buffer 204 and the designated layer.

The stream integration unit 253 integrates a predetermined number of video streams (encoded streams) output from the target layer selection unit 252 into one. The decoding unit 254 decodes the encoded image data of each picture of the video stream (encoded stream) integrated by the stream integration unit 253 sequentially at the decoding timing, and sends the encoded image data to the uncompressed data buffer (dpb) 206.

In this case, the decoding unit 254 analyzes SPS and ESPS by the Level_constrained_flag obtained from the demultiplexer 203, grasps "general_level_idc", "sublayer_level_idc", etc., and the stream or substream is within its own decoder processing capacity range. Check if it can be decoded. Further, in this case, the decoding unit 254 analyzes the SEI, grasps, for example, "initial_cpb_removal_time" and "cpb_removal_delay", and confirms whether the decoding timing from the CPU 201 is appropriate.

Further, when decoding the slice, the decoding unit 254 acquires "ref_idx_l0_active (ref_idx_l1_active)" as information indicating the prediction destination in the time direction from the slice header (Slice header), and predicts in the time direction. The decoded picture is processed as a reference by another picture using "short_term_ref_pic_set_idx" or "it_idx_sps" obtained from the slice header as an index.

The flowchart of FIG. 34 shows an example of the decoding processing procedure for each video stream in consideration of the decoder processing capacity of the receiving device 200. The receiving device 200 starts processing in step ST61, and reads the HEVC descriptor (HEVC_descriptor) in step ST62.

Next, in step ST63, the receiving device 200 determines whether or not “level_constrained_flag” exists in the HEVC descriptor. When present, the receiving device 200 determines in step ST64 whether the “level_constrained_flag” is “1”. When it is "1", the receiving device 200 proceeds to the process of step ST65.

In this step ST65, the receiving device 200 refers to the time stamp of the PES packet of the corresponding PID and reads the SPS or EPSS of the video stream of the payload portion. Then, the receiving device 200 reads "general_level_idc" which is an element of SPS or EPSS in step ST66.

Next, in step ST67, the receiving device 200 determines whether "general_level_idc" is within the decoder processing capacity range. When it is within the decoder processing capacity range, the receiving device 200 decodes the corresponding stream or substream in step ST68. After that, the receiving device 200 ends the process in step ST69. On the other hand, when it is not within the decoder processing capacity range in step ST67, the receiving device 200 immediately proceeds to step ST69 and ends the processing.

Further, when "level_constrained_flag" does not exist in step ST63, or when "level_constrained_flag" is "0" in step ST64, the receiving device 200 moves to the process of step ST70. In this step ST70, the receiving device 200 refers to the time stamp of the PES packet of the corresponding PID and reads the SPS of the video stream of the payload portion. On the other hand, if SPS does not exist in the corresponding video stream, temporal_layer refers to the SPS of the substream containing the lower picture (Picture).

Next, the receiving device 200 reads "general_level_idc" which is an element of SPS in step ST71. Then, in step ST72, the receiving device 200 determines whether "general_level_idc" is within the decoder processing capacity range. When it is within the decoder processing capacity range, the receiving device 200 moves to the processing of step ST73.

On the other hand, when it is not within the decoder processing capacity range, the receiving device 200 checks the "Sublayer_level_idc" of the SPS element in step ST74. Then, in step ST75, the receiving device 200 determines whether or not there is a "Sublayer" whose "Sublayer_level_idc" is within the decoder processing capacity range. When not present, the receiving device 200 immediately proceeds to step ST69 and ends the process. On the other hand, when present, the receiving device 200 moves to the process of step ST73.

In this step ST73, the receiving device 200 decodes the entire stream or the corresponding Sublayer portion with reference to the temporary_id value. After that, the receiving device 200 ends the process in step ST69.

Returning to FIG. 27, the uncompressed data buffer (dpb) 206 temporarily stores the image data of each picture decoded by the decoder 205. The post processing unit 207 performs a process of adjusting the frame rate of the image data of each picture sequentially read from the uncompressed data buffer (dpb) 206 at the display timing according to the display capability. In this case, the display timing is given by the CPU 201 based on the PTS (Presentation Time stamp).

For example, when the frame rate of the image data of each decoded picture is 120 fps and the display capacity is 120 fps, the post processing unit 207 sends the image data of each decoded picture to the display as it is. Further, for example, when the frame rate of the image data of each picture after decoding is 120 fps and the display capacity is 60 fps, the post processing unit 207 has a temporal resolution with respect to the image data of each picture after decoding. Subsample processing is performed so as to be 1/2 times, and the image data is sent to the display as 60 fps image data.

Further, for example, when the frame rate of the image data of each picture after decoding is 60 fps and the display capacity is 120 fps, the post processing unit 207 has a time-direction resolution with respect to the image data of each picture after decoding. Interpolation processing is performed so as to double the image data, and the image data is sent to the display at 120 fps. Further, for example, when the frame rate of the image data of each decoded picture is 60 fps and the display capability is 60 fps, the post processing unit 207 sends the image data of each decoded picture to the display as it is.

FIG. 35 shows a configuration example of the post processing unit 207. This example is an example in which it is possible to deal with a case where the frame rate of the image data of each picture after decoding is 120 fps or 60 fps and the display capability is 120 fps or 60 fps as described above.

The post processing unit 207 has an interpolation unit 271, a subsample unit 272, and a switch unit 273. The image data of each picture after decoding from the uncompressed data buffer 206 is directly input to the switch unit 273, or is input to the switch unit 273 after being doubled in the frame rate by the interpolation unit 271, or is input to the subsample unit. It is input to the switch unit 273 after the frame rate is set to 1/2 times that of 272.

Selection information is supplied to the switch unit 273 from the CPU 201. This selection information is generated automatically by the CPU 201 with reference to the display capability or in response to a user operation. The switch unit 273 selectively outputs any of the inputs based on the selection information. As a result, the frame rate of the image data of each picture sequentially read from the uncompressed data buffer (dpb) 206 at the display timing is set to match the display capability.

FIG. 36 shows an example of the processing flow of the decoder 205 and the post processing unit 207. The decoder 205 and the post processing unit 207 start processing in step ST51, and then move to the processing in step ST52. In this step ST52, the decoder 205 reads the video stream to be decoded stored in the compressed data buffer (cpb) 204, and selects a picture in the hierarchy designated as the decoding target from the CPU 201 based on the temporal_id.

Next, in step ST53, the decoder 205 sequentially decodes the encoded image data of each selected picture at the decoding timing, and transfers the image data of each decoded picture to the uncompressed data buffer (dpb) 206. , Temporarily accumulate. Next, in step ST54, the post processing unit 207 reads out the image data of each picture from the uncompressed data buffer (dpb) 206 at the display timing.

Next, the post processing unit 207 determines whether or not the frame rate of the image data of each read picture matches the display capability. When the frame rate does not match the display capability, the post processing unit 207 sends the frame rate to the display in accordance with the display capability in step ST56, and then ends the process in step ST57. On the other hand, when the frame rate matches the display capability, the post processing unit 207 sends the frame rate to the display as it is in step ST58, and then ends the processing in step ST57.

The operation of the receiving device 200 shown in FIG. 27 will be briefly described. In the receiving unit 202, the RF modulated signal received by the receiving antenna is demodulated, and the transport stream TS is acquired. This transport stream TS is sent to the demultiplexer 203. In the demultiplexer 203, the encoded image data of the layered picture according to the decoding capability (Decoder temporal layer capability) is selectively extracted from the transport stream TS, sent to the compressed data buffer (cpb) 204, and temporarily. Accumulates.

In the decoder 205, the encoded image data of the picture of the layer designated as the layer to be decoded is taken out from the video stream stored in the compressed data buffer 204. Then, in the decoder 205, the encoded image data of each of the extracted pictures is decoded at the decoding timing of the picture, sent to the uncompressed data buffer (dpb) 206, and temporarily stored. In this case, when the coded image data of each picture is decoded, the image data of the referenced picture is read out from the uncompressed data buffer 206 and used as needed.

The image data of each picture sequentially read from the uncompressed data buffer (dpb) 206 at the display timing is sent to the post processing unit 207. The post processing unit 207 performs interpolation or subsamples for adjusting the frame rate of the image data of each picture to match the display capability. The image data of each picture processed by the post processing unit 207 is supplied to the display, and the moving image is displayed by the image data of each picture.

As described above, in the transmission / reception system 10 shown in FIG. 1, the encoding interval is calculated for each layer on the transmitting side, and the higher the layer, the smaller the decoding time interval of the coded image data for each picture is set. The decoded time stamp is added to the coded image data of the picture in each layer. Therefore, for example, good decoding processing can be performed on the receiving side according to the decoding ability. For example, even when the decoding ability is low, it is possible to selectively decode the coded image data of a low-layer picture without causing the buffer of the compressed data buffer 204 to collapse.

Further, in the transmission / reception system 10 shown in FIG. 1, a scalability extension descriptor (scalability_extension_descriptor) or the like is inserted into the layer of the transport stream TS on the transmission side. Therefore, for example, the receiving side can easily grasp the hierarchical information in the hierarchical coding, the configuration information of the video stream included in the transport stream TS, and the like, and can perform appropriate decoding processing.

Further, in the transmission / reception system 10 shown in FIG. 1, in the transmission unit, a TS packet that divides a plurality of layers into a predetermined number of layer sets of two or more and containers the encoded image data of the pictures of the layer set on the lower layer side. The higher the priority, the higher the setting. For example, in the case of two divisions, the 1-bit field of "transport_priority" is set to "1" in the case of a TS packet that containers the encoded image data of the picture of the base layer, that is, the lower layer set, and is non-base. It is set to "0" in the case of a TS packet that containers the encoded image data of a layer, that is, a picture of a layered set on the higher layer side. Therefore, for example, on the receiving side, based on the priority of the TS packet, only the coded image data of the hierarchical set of pictures according to its own decoding ability can be taken into the compressed data buffer (cpb) 204. It becomes easier to avoid buffer rupture.

Further, in the transmission / reception system 10 shown in FIG. 1, on the transmitting side, a bitstream level designation value is inserted into each of the substreams corresponding to each layer set, and the value is included in the layer set below the own layer set. It is a level value that includes pictures of all layers. Therefore, the receiving side of the video stream can easily determine whether or not each substream can be decoded based on the level specified value of the inserted bitstream.

Further, in the transmission / reception system 10 shown in FIG. 1, on the transmitting side, the level specified value of the bit stream inserted into the substream of each layer set is set in the layer (container layer) of the transport stream TS. Flag information (level_constrained_flag) indicating that the level value includes pictures of all layers included in the following layer set is inserted. Therefore, on the receiving side, according to this flag information, the level specification value of the bitstream inserted into the substream of each layer set is a level value including pictures of all layers included in the layer set below the layer set. It is possible to improve the efficiency of decoding processing by eliminating the need for confirmation processing using sublayer_level_idc.

<2. Modification example>
In the above-described embodiment, the transmission / reception system 10 including the transmission device 100 and the reception device 200 is shown, but the configuration of the transmission / reception system to which the present technology can be applied is not limited to this. For example, the portion of the receiving device 200 may be, for example, a configuration of a set-top box and a monitor connected by a digital interface such as (HDMI (High-Definition Multimedia Interface)).

Further, in the above-described embodiment, an example in which the container is a transport stream (MPEG-2 TS) is shown. However, this technology can be similarly applied to a system configured to be delivered to a receiving terminal using a network such as the Internet. In Internet distribution, it is often distributed in containers of MP4 or other formats. That is, the container corresponds to a container of various formats such as transport stream (MPEG-2 TS) adopted in the digital broadcasting standard and MP4 used in Internet distribution.

In addition, the present technology can also have the following configurations.
(1) The image data of each picture constituting the moving image data is classified into a plurality of layers, the image data of the pictures of each classified layer is encoded, and the image data of the encoded pictures of each layer is obtained. An image encoding unit that generates a video stream to have
It is equipped with a transmitter that transmits a container of a predetermined format including the generated video stream.
The image coding unit
A transmission device that adds decoding timing information set so that the decoding time interval of the coded image data for each picture becomes smaller as the layer is higher, to the coded image data of the pictures in each layer.
(2) The image coding unit is
Generate a single video stream with the coded image data of the pictures in each of the above layers.
The transmission according to (1) above, wherein the plurality of layers are divided into two or more predetermined number of layer groups, and identification information for identifying the layer group to which the layer belongs is added to the coded image data of the picture of each layer group. apparatus.
(3) The transmission device according to (2) above, wherein the identification information is a bit stream level specified value, and the higher the hierarchy side, the higher the value.
(4) The image coding unit is
The plurality of layers are divided into a predetermined number of layer sets of two or more, and the layers are divided into two or more layers.
The transmission device according to (1) above, which generates the predetermined number of video streams having the encoded image data of the pictures of each layer set.
(5) The image coding unit is
The transmission device according to (4) above, wherein identification information for identifying the belonging layer set is added to the coded image data of the picture of each layer set.
(6) The transmission device according to (5) above, wherein the identification information is a bit stream level specified value, and the higher the hierarchy side, the higher the value.
(7) The image coding unit is
A single video stream having the coded image data of the pictures of each layer is generated, or the plurality of layers are divided into two or more predetermined number of layer sets, and the coded image data of the pictures of each layer set is generated. Generate the above predetermined number of video streams, each with
The transmission device according to any one of (1) to (6) above, further comprising an information insertion unit for inserting configuration information of a video stream included in the container into the layer of the container.
(8) The transmitter is
The plurality of layers are divided into a predetermined number of layer sets of two or more, and the priority of the packet for containerizing the coded image data of the picture of the layer set on the lower layer side is set higher. The transmitter according to any.
(9) The image data of each picture constituting the moving image data is classified into a plurality of layers, the image data of the pictures of each classified layer is encoded, and the image data of the encoded pictures of each layer is obtained. Image coding steps to generate a video stream with
It has a transmission step of transmitting a container of a predetermined format including the generated video stream by the transmission unit.
In the above image coding step,
A transmission method in which decoding timing information set so that the decoding time interval of the coded image data for each picture becomes smaller as the layer is higher is added to the coded image data of the pictures in each layer.
(10) A container having a predetermined format including a video stream having encoded image data of the pictures of each layer obtained by classifying and encoding the image data of each picture constituting the moving image data into a plurality of layers. Equipped with a receiver to receive
Decoding timing information set so that the decoding time interval of the coded image data for each picture becomes smaller as the layer is higher is added to the coded image data of the pictures in each layer.
The encoded image data of the picture in the layer below the predetermined layer selected from the video stream included in the received container is decoded at the decoding timing indicated by the decoding timing information, and the picture in the layer below the predetermined layer is decoded. A receiving device further provided with a processing unit for obtaining image data of the above.
(11) The received container contains a single video stream having coded image data of the pictures in each layer.
The plurality of layers are divided into a predetermined number of layer sets of two or more, and are set as high as the priority of the packet that containers the coded image data of the picture of the layer set on the lower layer side.
The receiving device according to (10) above, wherein the processing unit takes in the encoded image data of a predetermined hierarchical set of pictures containerized with packets having a priority selected according to the decoding ability into a buffer and decodes the data.
(12) The received container includes the predetermined number of video streams each having encoded image data of two or more predetermined number of hierarchical sets of pictures obtained by dividing the plurality of layers. Ori,
The receiving device according to (10) above, wherein the processing unit takes in the encoded image data of a predetermined layer set of pictures of a video stream selected according to the decoding ability into a buffer and decodes the data.
(13) The receiving device according to any one of (10) to (12) above, further comprising a post processing unit that matches the frame rate of the image data of each picture obtained by the processing unit with the display capability.
(14) A predetermined video stream including the coded image data of the pictures of each layer obtained by classifying the image data of each picture constituting the moving image data by the receiving unit into a plurality of layers and encoding the images. Has a receive step to receive the format container and
Decoding timing information set so that the decoding time interval of the coded image data for each picture becomes smaller as the layer is higher is added to the coded image data of the pictures in each layer.
The encoded image data of the picture in the layer below the predetermined layer selected from the video stream included in the received container is decoded at the decoding timing indicated by the decoding timing information, and the picture in the layer below the predetermined layer is decoded. A receiving method further comprising a processing step of obtaining image data of.
(15) The image data of each picture constituting the moving image data is classified into a plurality of layers, the image data of the pictures of each classified layer is encoded, and the image data of the encoded pictures of each layer is obtained. An image encoding unit that generates a video stream to have
It is equipped with a transmitter that transmits a container of a predetermined format including the generated video stream.
The image coding unit
A single video stream having the coded image data of the pictures of each layer is generated, or the plurality of layers are divided into two or more predetermined number of layer sets, and the coded image data of the pictures of each layer set is generated. Generate the above predetermined number of video streams, each with
A transmission device further comprising an information insertion unit for inserting configuration information of a video stream included in the container into the layer of the container.
(16) The image data of each picture constituting the moving image data is classified into a plurality of layers, the image data of the pictures of each classified layer is encoded, and the image data of the encoded pictures of each layer is obtained. An image encoding unit that generates a video stream to have
It is equipped with a transmitter that transmits a container of a predetermined format including the generated video stream.
The above transmitter
A transmission device that divides the plurality of layers into a predetermined number of layer sets of two or more, and sets the priority of the packet that containers the coded image data of the pictures of the layer set on the lower layer side higher.
(17) The image data of each picture constituting the moving image data is classified into a plurality of layers, the image data of the pictures of each classified layer is encoded, and the image data of the encoded pictures of each layer is obtained. Equipped with an image encoding unit that generates a video stream to have
The image coding unit divides the plurality of layers into a predetermined number of layer sets of two or more, inserts a bitstream level specification value into each of the substreams corresponding to each layer set, and inserts a bitstream level specification value.
The level specification value of the bitstream inserted into each of the substreams corresponding to each of the above-mentioned layer sets is a level value including pictures of all layers included in the layer sets below the own layer set.
(18) The image coding unit is
To generate the predetermined number of video streams including each of the substreams corresponding to each layer set, or to generate a single video stream including all of the substreams corresponding to each layer set in (17). The coding device described.
(19) The image coding unit classifies the image data of each picture constituting the moving image data into a plurality of layers, encodes the image data of the pictures of each of the classified layers, and encodes each layer. Has an image coding step that produces a video stream with image data of the picture of
In the image coding step, the plurality of layers are divided into a predetermined number of layer sets of two or more, and a bitstream level specification value is inserted into each of the substreams corresponding to each layer set.
A coding method in which the level specification value of the bitstream inserted into each of the substreams corresponding to each of the above layer sets is a level value including pictures of all layers included in the layer sets below the own layer set.
(20) The image data of each picture constituting the moving image data is classified into a plurality of layers, the image data of the pictures of each classified layer is encoded, and the image data of the encoded pictures of each layer is obtained. Equipped with an image encoding unit that generates a video stream to have
The image coding unit divides the plurality of layers into a predetermined number of layer sets of two or more, inserts a bitstream level specification value into each of the substreams corresponding to each layer set, and inserts a bitstream level specification value.
The level specification value of the bitstream inserted in each of the substreams corresponding to each of the above layer sets is a level value including pictures of all layers included in the layer sets below the layer set.
A transmitter that sends a container of a predetermined format containing the generated video stream,
In the layer of the above container, flag information indicating that the level specification value of the bitstream inserted into the substream of each layer set is a level value including pictures of all layers included in the layer set below the layer set is added. A transmitter further including an information insertion unit to be inserted.

The main feature of this technology is that the encoding interval is calculated for each layer, and the decoding time stamp set so that the decoding time interval of the encoded image data for each picture becomes smaller as the layer is higher is the coding of the pictures in each layer. By adding it to the image data, it is possible to perform good decoding processing according to the decoding performance on the receiving side (see FIG. 9). Further, the main feature of the present technology is that a plurality of layers are divided into a predetermined number of layer sets of two or more, and the priority of the packet for containerizing the coded image data of the picture of the layer set on the lower layer side is set higher. As a result, on the receiving side, only the encoded image data of the hierarchical set of pictures according to its own decoding ability is taken into the buffer based on the priority, and the buffer failure can be avoided (see FIG. 19).

10 ... Transmission / reception system 100 ... Transmission device 101 ... CPU
102 ... Encoder 103 ... Compressed data buffer (cpb)
104 ... multiplexer 105 ... transmitter 121 ... temporal ID generator 122 ... buffer delay control unit 123 ... HRD setting unit 124 ... parameter set / SEI encoding unit 125 ... slice encoding Part 126 ・ ・ ・ NAL packetization part 141 ・ ・ ・ TS priority generation part 142 ・ ・ ・ Section coding part 143-1 to 143-N ・ ・ ・ PES packetization part 144 ・ ・ ・ Transport packetization part 200 ・ ・・ Receiver 201 ・ ・ ・ CPU
202 ... Receiver 203 ... Demultiplexer 204 ... Compressed data buffer (cpb)
205 ... Decoder 206 ... Uncompressed data buffer (dpb)
207 ... Post processing unit 231 ... PCR extraction unit 232 ... Time stamp extraction unit 233 ... Section extraction unit 234 ... TS priority extraction unit 235 ... PES payload extraction unit 236 ... Picture Selection unit 251 ... Temporal ID analysis unit 252 ... Target hierarchy selection unit 253 ... Stream integration unit 254 ... Decoding unit 271 ... Interpolation unit 272 ... Subsample unit 273 ... Switch unit

Claims (3)

When classifying the image data of the pictures constituting the moving image data into a plurality of layers and decoding the image data of each picture from the lowest layer to the higher layer among the image data of the classified pictures, each picture The first encoded image data including the picture on the lower layer side and the second encoded image data including the picture on the higher layer side, which are encoded so that the decoding order and the display order of the image data of the above are different. At the same time, corresponding to the first level specified value based on the frame rate of 60 Hz consisting of only the pictures on the lower layer side corresponding to the first coded image data, and corresponding to the second coded image data. A receiver for receiving a second level specified value based on a frame rate of 120 Hz including the picture on the lower layer side and the picture on the higher layer side is provided.
The decoding time of each picture included in the first coded image data is specified as a time at 60 Hz intervals in the additional information of the access unit of each picture, and the decoding of each picture included in the second coded image data is performed. The time is specified as a time at intervals of 60 Hz in the additional information of the access unit of each picture, and the decoding timing of each picture included in the second coded image data is the decoding timing of the picture included in the first coded image data. It is encoded so that it is an intermediate timing of the decoding timing.
A process of calculating the specified decoding time of each picture included in the received first coded image data and the second coded image data based on the additional information of the access unit of each picture. A receiving device further provided with a unit. Based on the first level designated value and the second level designated value, the processing unit receives the first coded image data, the first coded image data, and the second coded image data. The receiving device according to claim 1, wherein the coded image data of the above is subjected to a decoding process to obtain moving image data. When classifying the image data of the pictures constituting the moving image data into a plurality of layers and decoding the image data of each picture from the lowest layer to the higher layer among the image data of the classified pictures, each picture The first encoded image data including the picture on the lower layer side and the second encoded image data including the picture on the higher layer side, which are encoded so that the decoding order and the display order of the image data of the above are different. At the same time, corresponding to the first level specified value based on the frame rate of 60 Hz consisting of only the pictures on the lower layer side corresponding to the first coded image data, and corresponding to the second coded image data. It has a reception step of receiving a second level specified value based on a frame rate of 120 Hz including the picture on the lower layer side and the picture on the higher layer side.
The decoding time of each picture included in the first coded image data is specified as a time at 60 Hz intervals in the additional information of the access unit of each picture, and the decoding of each picture included in the second coded image data is performed. The time is specified as a time at intervals of 60 Hz in the additional information of the access unit of each picture, and the decoding timing of each picture included in the second coded image data is the decoding timing of the picture included in the first coded image data. It is encoded so that it is an intermediate timing of the decoding timing.
A processing step of calculating the above-specified decoding time of each picture included in the received first coded image data and the second coded image data based on the additional information of the access unit of each picture. A receiving method that further has.

Images