JP7230981B2 - Receiving device and receiving method - Google Patents

Description

The present technology relates to a receiving device and a receiving method .

When a compressed moving image is serviced through broadcasting, the Internet, etc., the upper limit of the reproducible frame frequency is limited by the decoding capability of the receiver. Therefore, it is necessary for the service side to consider the reproduction capability of popular receivers, limit the service to only low frame frequencies, or simultaneously provide services of a plurality of high and low frame frequencies.

The cost of receivers to support high frame frequency services is high, which is an obstacle to early popularization. In the early days, only inexpensive receivers dedicated to low frame frequency services were popular, but in the future when the service side starts high frame frequency services, it will be impossible to view at all without new receivers, and new services will spread. is a hindrance to

For example, HEVC (High Efficiency Video Coding) proposes temporal scalability by hierarchically encoding image data of each picture constituting moving image data (see Non-Patent Document 1). On the receiving side, the layer of each picture can be identified based on the temporal ID (temporal_id) inserted in the header of the NAL (Network Abstraction Layer) unit, enabling selective decoding up to the layer corresponding to the decoding capability. .

Gary J. Sullivan, Jens-Rainer Ohm, Woo-Jin Han, Thomas Wiegand, “Overview of the High Efficiency Video Coding (HEVC) Standard” IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECNOROGY, VOL. 22, NO. 12, pp. 1649-1668, DECEMBER 2012

An object of the present technology is to enable good decoding processing according to the decoding capability on the receiving side.

The concept of this technology is
Classifying image data of each picture constituting moving image data into a plurality of hierarchies, encoding the image data of the classified picture of each hierarchy, and video data having the image data of the encoded picture of each hierarchy an image encoding unit that generates
a transmission unit for transmitting a container of a predetermined format containing the generated video data;
dividing the plurality of hierarchies into a predetermined number of hierarchies of two or more, and coding a picture belonging to which hierarchy set the encoded image data of each picture contained in the video data is stored in a packet containing the video data; A transmission device comprising an identification information inserting unit for inserting identification information for identifying image data.

In the present technology, the image encoding unit encodes image data of each picture forming moving image data to generate video data. In this case, the image data of each picture that constitutes the moving image data is classified into a plurality of layers and encoded, and video data having encoded image data of the picture of each layer is generated.

The transmitting unit transmits a container of a predetermined format containing the video data described above. For example, the container may be a transport stream (MPEG-2 TS) adopted by digital broadcasting standards. Also, for example, the container may be MP4 used for Internet distribution or a container of other formats.

The identification information inserting unit divides the plurality of layers into a predetermined number of layer groups of two or more, and the encoded image data of each picture included in the video data belongs to which layer group in the packet that contains the video data. Identification information is inserted to identify whether it is encoded image data of a picture. For example, the identification information may be priority information that is set higher for a hierarchy set on the lower hierarchy side.

For example, the identification information may be inserted into the header of a PES packet containing encoded image data for each picture in the payload. In this case, for example, the identification information may be inserted using the PES priority field of the header. Also, for example, the identification information may be inserted into this adaptation field of a TS packet having an adaptation field. In this case, for example, the identification information may be inserted using the ES priority indicator field of the adaptation field. Also, for example, the identification information may be inserted in a header box associated with the track of the picture in question.

As described above, in the present technology, identification information identifying to which layer set the encoded image data of each picture included in the video data belongs to the packet that contains the video data. is inserted. Therefore, on the receiving side, by using this identification information, it becomes possible to selectively decode encoded image data of pictures in a hierarchy below a predetermined hierarchy according to the decoding capability.

In addition, in the present technology, for example, the image encoding unit generates a single video stream having encoded image data of pictures in each layer, or divides a plurality of layers into a predetermined number of layer sets of two or more. a configuration information inserting unit that generates a predetermined number of video streams each having encoded image data of pictures in each layer set, and inserts configuration information of the video streams included in the container into the layer of the container; may be made In this case, for example, the receiving side can easily grasp the structure of the video stream based on the structure information of the video stream included in the container.

Another concept of this technology is
A container of a predetermined format containing video data having encoded image data of pictures in each layer obtained by classifying and encoding image data of each picture constituting moving image data into a plurality of layers is received. a receiver;
Selectively fetching into a buffer coded image data of pictures in a hierarchy below a predetermined hierarchy according to decoding capability from the video stream contained in the received container, and coded image data of each picture fetched into the buffer , and obtains image data of a picture in a hierarchy below the predetermined hierarchy.

In the present technology, the receiving unit receives a container in a predetermined format. This container contains video data having picture image data of each layer obtained by classifying the image data of each picture constituting moving image data into a plurality of layers and encoding them.

An image decoding unit selectively fetches encoded image data of pictures in a hierarchy lower than a predetermined hierarchy according to the decoding capability from the video data contained in the received container into a buffer, and each picture fetched into the buffer. is decoded to obtain image data of pictures in a hierarchy below a predetermined hierarchy.

For example, a plurality of layers are divided into a predetermined number of layer sets of two or more, and encoded image data of each picture included in the video data is encoded in a packet that contains the video data. Identification information for identifying whether it is image data is inserted. Based on the identification information, the image decoding unit takes in the encoded image data of the picture of the predetermined layer set corresponding to the decoding ability into the buffer and decodes it. , may be so.

In this case, for example, the identification information may be inserted into the header of the PES packet containing the encoded image data for each picture in the payload. Also in this case, for example, the identification information may be inserted in this adaptation field of a TS packet having an adaptation field. Also in this case, for example, the identification information may be inserted in a box in the header associated with the track of the picture in question.

Also, for example, the plurality of layers are divided into a predetermined number of layer sets equal to or greater than two, and the received container includes a predetermined number of video streams each having encoded image data of pictures of the predetermined number of layer sets. Based on the stream identification information, the image coding unit may load the coded image data of the picture of the predetermined layer set according to the decoding ability into the buffer and decode the data. At this time, for example, when the encoded image data of the pictures of the predetermined layer group are included in a plurality of video streams, the image decoding unit converts the encoded image data of each picture into one stream based on the decoding timing information. , and then capture it in the buffer.

As described above, according to the present technology, encoded image data of pictures in a hierarchy below a predetermined hierarchy according to decoding capability is selectively loaded into a buffer and decoded from received video data. Therefore, appropriate decoding processing can be performed according to the decoding ability.

Note that in the present technology, for example, the image decoding unit has a function of rewriting the decoding time stamp of the encoded image data of each picture selectively captured in the buffer to adjust the decoding interval of the low-layer pictures. may be In this case, even a decoder with low decoding capability can perform a reasonable decoding process.

Further, in the present technology, for example, a post processing unit that matches the frame rate of image data of each picture obtained by the image decoding unit to the display capability may be provided. In this case, even if the decoding ability is low, it is possible to obtain image data at a frame rate suitable for high display ability.

According to the present technology, it is possible to perform good decoding processing according to the decoding capability on the receiving side. Note that the effects described here are not necessarily limited, and may be any of the effects described in the present disclosure.

1 is a block diagram showing a configuration example of a transmission/reception system as an embodiment; FIG. It is a block diagram which shows the structural example of a transmitter. FIG. 4 is a diagram showing an example of hierarchical encoding performed by an encoder; FIG. 2 is a diagram showing an example structure (Syntax) of a NAL unit header and the contents (Semantics) of main parameters in the example structure. FIG. 2 is a diagram for explaining the configuration of encoded image data of each picture by HEVC; FIG. 4 is a diagram showing an example of encoding, decoding, display order and delay in hierarchical coding; FIG. 10 is a diagram showing an encoded stream of hierarchical encoding and display expectations (display order) in a designated layer; FIG. 4 is a diagram showing a structural example (Syntax) of an HEVC descriptor (HEVC_descriptor); FIG. 2 is a diagram showing the content (semantics) of main information in an example structure of an HEVC descriptor; FIG. 10 is a diagram showing a structural example (Syntax) of a scalability extension descriptor (scalability_extension_descriptor); FIG. 10 is a diagram showing the content (Semantics) of main information in a structural example of a scalability extension descriptor; 4 is a block diagram showing a configuration example of a multiplexer; FIG. FIG. 10 is a diagram illustrating an example of a processing flow of a multiplexer; FIG. 10 is a diagram showing a configuration example of a transport stream TS for distribution by a single stream; It is a block diagram which shows the structural example of a receiver. It is a block diagram which shows the structural example of a demultiplexer. FIG. 3 is a diagram showing a case where a transport stream TS includes a single video stream (encoded stream); FIG. 3 is a diagram showing a case where a transport stream TS includes two video streams (encoded streams), a base stream and an extension stream. FIG. 10 is a diagram for explaining a function of rewriting the decoding time stamp of the encoded image data of each picture to adjust the decoding interval of the low-hierarchy pictures; FIG. 4 is a diagram showing an example of a processing flow (one frame) of a demultiplexer; FIG. 4 is a diagram showing an example of a processing flow (2 frames) of a demultiplexer; FIG. 4 is a block diagram showing a configuration example of a decoder; FIG. 4 is a diagram illustrating a configuration example of a post-processing unit; FIG. 10 is a diagram illustrating an example of a processing flow of a decoder and a post-processing unit; FIG. 10 is a diagram showing an example of arrangement of adaptation fields; FIG. 10 is a block diagram showing a configuration example of a multiplexer when inserting identification information of a hierarchy group into an adaptation field; FIG. 10 is a diagram showing a configuration example of a transport stream TS in the case of inserting identification information of a layer set into an adaptation field; FIG. 11 is a block diagram showing a configuration example of a demultiplexer when inserting identification information of a hierarchy group into an adaptation field; FIG. 4 is a diagram showing a configuration example of an MP4 stream; FIG. 10 is a diagram showing a structural example of “SampleDependencyTypeBox”; FIG. 10 is a diagram showing the contents of main information in the structural example of "SampleDependencyTypeBox"; It is a figure which shows the structural example of "SampleScalablePriorityBox." FIG. 10 is a diagram showing the contents of main information in a structural example of "SampleScalablePriorityBox";

DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, modes for carrying out the invention (hereinafter referred to as "embodiments") will be described. The description will be given in the following order.
1. Embodiment 2. Modification

<1. Embodiment>
[Transmitting/receiving system]
FIG. 1 shows a configuration example of a transmission/reception system 10 as an embodiment. This transmission/reception system 10 has a configuration including a transmission device 100 and a reception device 200 .

The transmitting device 100 transmits the transport stream TS as a container on broadcast waves. The transport stream TS includes a video stream in which the image data of each picture constituting the moving image data is classified into a plurality of hierarchies, and has encoded data of the image data of the pictures of each hierarchy. In this case, for example, H. H.264/AVC, HEVC, or the like is applied, and the referenced picture is coded so as to belong to the self-layer and/or a layer lower than the self-layer.

Layer identification information for identifying the layer to which each picture belongs is added to the encoded image data of the picture in each layer. In this embodiment, "nuh_temporal_id_plus1" meaning layer identification information (temporal_id) is arranged in the header portion of the NAL unit (nal_unit) of each picture. By adding the layer identification information in this way, it becomes possible for the receiving side to identify the layer of each picture in the layer of the NAL unit. It can be carried out.

In this embodiment, a plurality of hierarchies are divided into a predetermined number of hierarchies that are equal to or greater than two. Identification information is inserted to identify whether the data is converted image data.

In this embodiment, this identification information is priority information that is set higher for a hierarchy set on the lower hierarchy side, and is inserted into the header of a PES packet containing encoded image data for each picture in the payload. This identification information enables the receiving side to store and process only the coded image data of the picture of the hierarchy set according to its own decoding capability.

The transport stream TS includes a single video stream having encoded image data of pictures in each layer, or a predetermined number of video streams each having encoded image data of pictures in each layer set. Hierarchical coding layer information and video stream configuration information are inserted into this transport stream TS. With this information, the receiving side can easily grasp the hierarchical structure and the stream structure, and can perform appropriate decoding processing.

The receiving device 200 receives the transport stream TS transmitted from the transmitting device 100 over a broadcast wave. The receiving apparatus 200 selectively stores encoded image data of pictures in a hierarchy below a predetermined hierarchy selected according to the decoding capability from the video stream included in the transport stream TS, decodes the encoded image data, and decodes the encoded image data of each picture. Acquire image data and perform image reproduction.

For example, as described above, the transport stream TS may include a single video stream having encoded image data of pictures in multiple layers. In this case, based on the identification information described above, the coded image data of the picture of the predetermined hierarchy set corresponding to the decoding ability is loaded into the buffer and processed.

Further, for example, as described above, the transport stream TS includes a predetermined number of video streams each having encoded image data of pictures in a predetermined number of two or more layer sets obtained by dividing a plurality of layers. may be In this case, based on the stream identification information, the encoded image data of the picture of the predetermined hierarchy set corresponding to the decoding capability is loaded into the buffer and processed.

In addition, the receiving device 200 rewrites the decoding time stamp of the encoded image data of each picture selectively stored in the buffer to adjust the decoding interval of the low-hierarchy pictures. Due to this adjustment process, even a decoder with a low decoding capability can perform a reasonable decoding process.

Further, the receiving device 200 performs post-processing for matching the frame rate of the image data of each picture obtained by decoding as described above to the display capability. By this post-processing, for example, even if the decoding capability is low, it is possible to obtain image data with a frame rate suitable for high display capability.

"Transmitter configuration"
FIG. 2 shows a configuration example of the transmission device 100. As shown in FIG. This transmission device 100 has a CPU (Central Processing Unit) 101 , an encoder 102 , a compressed data buffer (cpb: coded picture buffer) 103 , a multiplexer 104 and a transmission section 105 . The CPU 101 is a control unit and controls the operation of each unit of the transmission device 100 .

The encoder 102 inputs uncompressed moving image data and performs hierarchical encoding. The encoder 102 classifies the image data of each picture forming this moving image data into a plurality of hierarchies. Then, the encoder 102 encodes the classified image data of the pictures in each layer to generate a video stream having the encoded image data of the pictures in each layer. Encoder 102 may be, for example, H.264. H.264/AVC, HEVC, etc. are encoded. At this time, the encoder 102 encodes a picture to be referred to (referenced picture) so as to belong to the self layer and/or a layer lower than the self layer.

FIG. 3 shows an example of hierarchical encoding performed by the encoder 102. As shown in FIG. In this example, the images are classified into five hierarchies from 0 to 4, and the image data of the pictures in each hierarchy are coded by, for example, HEVC.

The vertical axis indicates the hierarchy. 0 to 4 are set as temporal_id (hierarchy identification information) arranged in the header portion of the NAL unit (nal_unit) constituting the encoded image data of the pictures of hierarchies 0 to 4, respectively. On the other hand, the horizontal axis indicates the display order (POC: picture order of composition), with the display time on the left side being earlier and the display time on the right side being later.

FIG. 4(a) shows an example structure (Syntax) of the NAL unit header, and FIG. 4(b) shows the contents (Semantics) of main parameters in the example structure. The 1-bit field of "Forbidden_zero_bit" must be 0. A 6-bit field of "Nal_unit_type" indicates the NAL unit type. The 6-bit field of "Nuh_layer_id" is assumed to be 0. A 3-bit field of "Nuh_temporal_id_plus1" indicates temporal_id and takes a value (1 to 7) with 1 added.

Returning to FIG. 3, each rectangular frame indicates a picture, and the numbers indicate the order of encoded pictures, that is, the encoding order (decoding order on the receiving side). 16 pictures "1" to "17" (excluding "2") constitute a sub group of pictures, and "1" is the leading picture of the sub picture group. . "2" is the first picture of the next sub-picture group. Alternatively, 16 pictures from "2" to "17" excluding "1" constitute a sub-picture group, and "2" is the top picture of the sub-picture group.

A picture of "1" can be the leading picture of a GOP (Group Of Pictures). As shown in FIG. 5, the encoded image data of the top picture of the GOP is composed of NAL units of AUD, VPS, SPS, PPS, PSEI, SLICE, SSEI, and EOS. On the other hand, pictures other than the first picture of the GOP are composed of NAL units of AUD, PPS, PSEI, SLICE, SSEI, and EOS. VPS can be transmitted together with SPS once in a sequence (GOP), and PPS can be transmitted in My Picture.

Returning to FIG. 3, the solid line arrows indicate the reference relationship of pictures in encoding. For example, a "1" picture is an I picture and does not refer to other pictures. The "2" picture is a P picture and is encoded with reference to the "1" picture. Also, the picture of "3" is a B picture and is encoded with reference to the pictures of "1" and "3". Other pictures are coded with reference to pictures nearby in display order. A picture of layer 4 is not referred to by other pictures.

The encoder 102 generates a single video stream (single stream) having encoded image data of pictures in each layer, or divides a plurality of layers into a predetermined number of two or more layer groups, and divides each layer group into A predetermined number of video streams (multi-streams) each having encoded image data of pictures are generated. For example, in the example of hierarchical encoding in FIG. 2 video streams (encoded streams) each having encoded image data of pictures of .

Regardless of the number of video streams to be generated, the encoder 102 divides a plurality of layers into a predetermined number of layer groups equal to or greater than two, as described above, and encodes the encoded image data of the picture in each layer group into the belonging layer group. Add identification information to identify the In this case, for example, "general_level_idc", which is the level designation value of the bitstream included in the SPS, is used as the identification information, and the higher the layer set, the higher the value. Since "sub_layer_level_idc" can be sent by SPS for each sublayer, this "sub_layer_level_idc" may be used as identification information. The above is supplied in VPS as well as in SPS.

In this case, the value of the level designation value of each layer group is a value corresponding to the frame rate of the picture of this layer group and the pictures of all layer groups below this layer group. For example, in the example of hierarchical coding shown in FIG. The specified value is a value corresponding to the frame rate of pictures of all layers 0 to 4. FIG.

FIG. 6 shows an example of encoding, decoding, display order and delay in hierarchical coding. This example corresponds to the hierarchical coding example of FIG. 3 described above. This example shows a case where all hierarchies (all layers) are hierarchically encoded with full temporal resolution. FIG. 6(a) shows the encoder input. As shown in FIG. 6B, each picture is encoded in encoding order with a delay of 16 pictures to obtain an encoded stream. Also, FIG. 6(b) shows the decoder input, and each picture is decoded in decoding order. Then, as shown in FIG. 6(c), the image data of each picture is obtained in display order with a delay of 4 pictures.

FIG. 7A shows an encoded stream similar to the encoded stream shown in FIG. Here, "Tid" indicates temporal_id. FIG. 7(b) shows display expectations (display order) when selectively decoding pictures of layers 0 to 2, that is, partial layers of Tid=0 to 2. FIG. FIG. 7(c) shows display expectations (display order) when selectively decoding pictures of layers 0 to 3, that is, partial layers of Tid=0 to 3. FIG. Furthermore, FIG. 7(d) shows the display expectation (display order) when selectively decoding each picture of layers 0 to 4, that is, all layers of Tid=0 to 4. FIG.

In order to decode the encoded stream of FIG. 7(a) by decoding capability, decoding capability with full-rate temporal resolution is required. However, when decoding Tid = 0 to 2, a decoder with 1/4 decoding capability should be able to handle the full temporal resolution of the encoded data. Also, when decoding Tid=0 to 3, a decoder with half the decoding capability of the encoded full temporal resolution should be able to process.

However, if the pictures belonging to the lower layers that are referred to in the hierarchical coding continue and are coded at full timing with temporal resolution, the ability of the decoder to perform partial decoding cannot catch up. The period A in FIG. 7A corresponds to this. Decoders that decode partial hierarchies with Tid=0 to 2 or Tid=0 to 3 decode and display with the capability of 1/4 or 1/2 of the time axis as shown in the display example. , A, continuous pictures with full temporal resolution encoded cannot be decoded.

Ta indicates the time required for the decoding process for each picture in the decoder that decodes Tid=0-2. Tb indicates the time required for the decoding process for each picture in the decoder that decodes Tid=0-3. Tc indicates the time required for decoding processing for each picture in a decoder that decodes Tid=0 to 4 (all hierarchies). The relationship between these times is Ta>Tb>Tc.

In this embodiment, as will be described later, when receiving apparatus 200 has a decoder with low decoding capability and selectively decodes low-layer pictures, decoding time stamp (DTS) is rewritten. It has a function to adjust the decoding interval of low-hierarchical pictures. As a result, even a decoder with a low decoding capability can perform a reasonable decoding process.

Returning to FIG. 2, the compressed data buffer (cpb) 103 temporarily accumulates a video stream containing encoded data of pictures in each layer generated by the encoder 102 . The multiplexer 104 reads out the video stream stored in the compressed data buffer 103, converts it into PES packets, further converts it into transport packets, and multiplexes them to obtain a transport stream TS as a multiplexed stream.

In this embodiment, as described above, a plurality of hierarchies are divided into a predetermined number of hierarchy sets equal to or greater than two. The multiplexer 104 inserts, into the header of the PES packet (PES header), identification information identifying to which layer set the encoded image data of each picture in the video stream belongs. This identification information enables the receiving side to store and process only the coded image data of the picture of the hierarchy set according to its own decoding capability.

The multiplexer 104 uses a well-known PES priority (PES_priority) 1-bit field present in the PES header when, for example, dividing a plurality of hierarchies into a low-hierarchy group and a high-hierarchy group. This 1-bit field is set to "1" when the PES payload contains the encoded image data of the picture of the layer set on the lower layer side, that is, the priority is set high. On the other hand, this 1-bit field is set to "0" when the PES payload contains the encoded image data of the picture of the layer group on the higher layer side, that is, the priority is set low.

As described above, the transport stream TS includes a single video stream having encoded image data of pictures in each layer, or a predetermined number of video streams each having encoded image data of pictures in each layer group. is included. The multiplexer 104 inserts layer information and stream configuration information into the transport stream TS.

The transport stream TS includes PMT (Program Map Table) as one of PSI (Program Specific Information). In this PMT there is a video elementary loop (video ES1 loop) with information related to each video stream. In this video elementary loop, information such as a stream type, packet identifier (PID), etc. is arranged corresponding to each video stream, and a descriptor describing information related to the video stream is also arranged.

The multiplexer 104 inserts an HEVC descriptor (HEVC_descriptor) as one of these descriptors, and further inserts a newly defined scalability extension descriptor (scalability_extension_descriptor).

FIG. 8 shows an example structure (Syntax) of an HEVC descriptor (HEVC_descriptor). Also, FIG. 9 shows the main information contents (Semantics) in the structural example.

An 8-bit field of "descriptor_tag" indicates a descriptor type, here indicating that it is an HEVC descriptor. An 8-bit field of "descriptor_length" indicates the length (size) of the descriptor, and indicates the number of subsequent bytes as the length of the descriptor.

An 8-bit field of "level_idc" indicates a bit rate level designation value. Also, when 'temporal_layer_subset_flag=1', there are a 5-bit field of 'temporal_id_min' and a 5-bit field of 'temporal_id_max'. "temporal_id_min" indicates the value of temporal_id of the lowest layer of hierarchically-encoded data included in the corresponding video stream. "temporal_id_max" indicates the value of temporal_id of the highest layer of hierarchically encoded data of the corresponding video stream.

A 1-bit field of “level_constrained_flag” is newly defined, and indicates that the level specification value (general_level_idc) of the bitstream included in the VPS NAL unit can change for each picture. "1" indicates that it can change, and "0" indicates that it does not change.

As described above, for example, "general_level_idc" is used as identification information of a hierarchy set to which a plurality of hierarchies belong when a plurality of hierarchies are divided into a predetermined number of hierarchy sets equal to or greater than two. Therefore, in the case of a video stream having encoded image data of pictures of multiple layer sets, "general_level_idc" can change for each picture. On the other hand, in the case of a video stream having encoded image data of pictures of a single layer set, "general_level_idc" does not change for each picture. Alternatively, "sublayer_level_idc" is attached to each sublayer, and the decoder processes the data of the corresponding layer by reading the temporal_id packets within the decodable range.

A 3-bit field of "scalability_id" is newly defined, and is an ID indicating scalability given to each stream when multiple video streams provide scalable services. "0" indicates the base stream, and "1" to "7" are IDs that increase according to the degree of scalability from the base stream.

FIG. 10 shows a structural example (Syntax) of a scalability extension descriptor (scalability_extension_descriptor). Also, FIG. 11 shows the content (semantics) of main information in the structural example.

An 8-bit field of "scalability_extension_descriptor_tag" indicates a descriptor type, here indicating that it is a scalability extension descriptor. An 8-bit field of “scalability_extension_descriptor_length” indicates the length (size) of the descriptor, and indicates the number of subsequent bytes as the length of the descriptor. A 1-bit field of "extension_stream_existing_flag" is a flag indicating that there is an extension service by another stream. A "1" indicates that there is an extension stream, and a "0" indicates that there is no extension stream.

A 3-bit field of “extension_type” indicates the extension type. “001” indicates that the extension is temporally scalable. “010” indicates that the extension is spatially scalable. "011" indicates that the extension is bitrate scalable.

A 4-bit field of "number_of_streams" indicates the total number of streams involved in the delivery service. A 3-bit field of "scalability_id" is an ID indicating scalability given to each stream when multiple video streams provide scalable services. "0" indicates the base stream, and "1" to "7" are IDs that increase according to the degree of scalability from the base stream.

A 3-bit field of "number_of_layers" indicates the total number of layers of the stream. ``The 8-bit field of sublayer_level_idc indicates the value of level_idc that the decoder corresponds to, including the lower layers of the corresponding sublayer indicated by temporal_id. This includes all values of "Nuh_temporal_id_plus1", and the demuxer detects this to determine to what layer the decoder corresponding to the given level_idc can decode in advance by "sublayer_level_idc" It becomes possible to recognize

As described above, in this embodiment, the bit rate level specification value (general_level_idc) included in the SPS is the identification information of the group of layers to which it belongs when dividing a plurality of layers into a predetermined number of layer groups of two or more. used as The value of the level designation value of each layer group is a value corresponding to the frame rate of the picture of this layer group and the pictures of all layer groups lower than this layer group.

FIG. 12 shows an example configuration of the multiplexer 104 . It has a PES priority generation unit 141 , a section coding unit 142 , PES packetization units 143 - 1 to 143 -N, a switch unit 144 and a transport packetization unit 145 .

The PES packetization units 143-1 to 143-N respectively read the video streams 1 to N accumulated in the compressed data buffer 103 and generate PES packets. At this time, the PES packetization units 143-1 to 143-N attach time stamps of DTS (Decoding Time Stamp) and PTS (Presentation Time Stamp) to the PES header based on the HRD information of the video streams 1 to N. In this case, "cpu_removal_delay" and "dpb_output_delay" of each picture are referred to, converted into DTS and PTS, respectively, with accuracy synchronized with the STC (System Time Clock) time, and arranged at a predetermined position of the PES header.

Information on the number of layers and the number of streams is supplied from the CPU 101 to the PES priority generation unit 141 . The PES priority generation unit 141 generates priority information for each hierarchy set when a plurality of hierarchies indicated by the number of hierarchies is divided into a predetermined number of hierarchy sets equal to or greater than two. For example, when dividing into two, a value ("1" for a low-layer group and "0" for a high-layer group) to be inserted into the 1-bit field of "PES_priority" of the PES packet header is generated.

The priority information of each layer set generated by the PES priority generator 141 is supplied to the PES packetizers 143-1 to 143-N. The PES packetization units 143-1 to 143-N insert the priority of each layer group into the header of the PES packet containing the encoded image data of the picture of the layer group as identification information.

Note that the process of inserting the priority of the layer group to which the picture belongs to the header of the PES packet for each picture as header information is limited to the case where the encoder 102 generates a single video stream (single stream). may In this case, processing is performed only by the PES packetization unit 143-1.

The switch section 144 selectively extracts the PES packets generated by the PES packetization sections 143 - 1 to 143 -N based on the packet identifier (PID), and sends them to the transport packetization section 145 . The transport packetization unit 145 generates TS packets containing PES packets in their payloads, and obtains a transport stream TS.

The section coding unit 142 generates various section data to be inserted into the transport stream TS. Information on the number of layers (Number of layers) and the number of streams (Number of streams) is supplied from the CPU 101 to the section coding unit 142 . The section coding unit 142 generates the above-described HEVC descriptor (HEVC_descriptor) and scalability extension descriptor (scalability_extension_descriptor) based on this information.

Section coding section 142 sends various section data to transport packetization section 145 . The transport packetization unit 145 generates TS packets containing this section data and inserts them into the transport stream TS.

FIG. 13 shows the processing flow of multiplexer 104 . This example is an example of dividing a plurality of hierarchies into two groups, a low-hierarchy group and a high-hierarchy group. The multiplexer 104 starts processing in step ST1, and then proceeds to processing in step ST2. In this step ST2, the multiplexer 104 sets temporal_id_ of each picture of the video stream (video elementary stream) and the number of coded streams to constitute.

Next, in step ST3, the multiplexer 104 refers to the HRD information (cpu_removal_delay, dpb_output_delay) to determine the DTS and PTS, and inserts them at predetermined positions in the PES header.

Next, in step ST4, multiplexer 104 determines whether it is a single stream (single video stream). If it is a single stream, the multiplexer 104 advances the multiplexing process with one PID (packet identifier) in step ST5, and then moves to the process of step ST7.

In this step ST7, the multiplexer 104 determines whether each picture is a picture (slice) of the low layer set. If it is a picture of the low hierarchy group, the multiplexer 104 sets "PES_priority" of the header of the PES packet containing the encoded image data of the picture in the payload to "1" in step ST8. On the other hand, if the picture is a high layer set (non-low layer set), the multiplexer 104 sets "PES_priority" of the header of the PES packet containing the encoded image data of the picture in the payload to "0" in step ST9. do. After the processing of steps ST8 and ST9, the multiplexer 104 proceeds to the processing of step ST10.

Here, the association between pictures and slices will be described. A picture is conceptually the same as a slice in structural definition. One picture is divided into a plurality of slices, and the fact that these slices are the same as access units can be understood from a parameter set.

When it is determined in step ST4 that the stream is not a single stream, the multiplexer 104 proceeds with multiplexing processing using a plurality of packet PIDs (packet identifiers) in step ST6, and then proceeds to processing in step ST10. In this step ST10, the multiplexer 104 inserts the encoded stream (video elementary stream) into the PES payload and PES packetizes it.

Next, multiplexer 104 codes HEVC descriptors, scalability extension descriptors, etc. in step ST11. Then, the multiplexer 104 transport packetizes in step ST12 to obtain a transport stream TS. After that, the multiplexer 104 terminates the processing in step ST13.

FIG. 14 shows a configuration example of a transport stream TS for single-stream distribution. This transport stream TS contains one video stream. That is, in this configuration example, there is a PES packet "video PES1" of a video stream having, for example, HEVC-encoded image data of pictures in a plurality of layers, and a PES packet "audio PES1" of an audio stream.

Encoded image data of each picture includes NAL units such as VPS, SPS, and SEI. As described above, temporal_id indicating the hierarchy of the picture is inserted in the header of the NAL unit of each picture. Also, for example, the VPS includes a bit rate level designation value (general_level_idc). Also, for example, picture timing SEI includes "cpb_removal_delay" and "dpb_output_delay".

In addition, there is a field indicating a 1-bit priority of "PES_priority" in the header of the PES packet (PES header). With this "PES_priority", it is possible to identify whether the encoded image data of the picture included in the PES payload is the picture of the low layer group or the picture of the high layer group.

The transport stream TS also includes a PMT (Program Map Table) as one of PSI (Program Specific Information). This PSI is information describing to which program each elementary stream included in the transport stream belongs.

A PMT has a program loop that describes information related to the entire program. Also in the PMT there is an elementary loop with information related to each elementary stream. In this configuration example, there is a video elementary loop (video ES1 loop) and an audio elementary loop (audio ES1 loop).

In the video elementary loop, information such as stream type and packet identifier (PID) is arranged corresponding to the video stream (video PES1), and descriptors describing information related to the video stream are also arranged. be. As one of these descriptors, the above-described HEVC descriptor (HEVC_descriptor) and scalability extension descriptor (scalability_extension_descriptor) are inserted.

Returning to FIG. 2, the transmission unit 105 modulates the transport stream TS with a modulation scheme suitable for broadcasting, such as QPSK/OFDM, and transmits an RF modulated signal from a transmission antenna.

The operation of the transmission device 100 shown in FIG. 2 will be briefly described. Uncompressed moving image data is input to the encoder 102 . The encoder 102 performs hierarchical coding on this moving image data. That is, the encoder 102 classifies the image data of each picture forming the moving image data into a plurality of layers and encodes them, thereby generating a video stream having encoded image data of the pictures of each layer. At this time, the picture to be referred to is coded so as to belong to the self-layer and/or a layer lower than the self-layer.

In the encoder 102, a single video stream having encoded image data of pictures in each layer is generated, or a plurality of layers is divided into a predetermined number of layer sets of two or more, and the pictures in each layer set are divided into two or more layer sets. A predetermined number of video streams each having encoded image data are generated.

A video stream containing encoded data of pictures in each layer generated by the encoder 102 is supplied to a compressed data buffer (cpb) 103 and temporarily stored. The multiplexer 104 reads the video stream stored in the compressed data buffer 103, PES-packetizes it, further transport-packets it, and multiplexes it to obtain a transport stream TS as a multiplexed stream.

In the multiplexer 104, for example, in the case of a single video stream (single stream), the encoded image data of each picture of the video stream is encoded in the header of the PES packet (PES header) to which layer group each picture belongs. Identification information for identifying image data is inserted. For example, when dividing a plurality of hierarchies into a low-hierarchy group and a high-hierarchy group, a 1-bit field of PES priority (PES_priority) of the PES header is used.

Also, the multiplexer 104 inserts layer information and stream configuration information into the transport stream TS. That is, the multiplexer 104 inserts an HEVC descriptor (HEVC_descriptor) and a scalability extension descriptor (scalability_extension_descriptor) into the video elementary loop corresponding to each video stream.

A transport stream TS generated by the multiplexer 104 is sent to the transmission section 105 . The transmission unit 105 modulates the transport stream TS with a modulation scheme suitable for broadcasting, such as QPSK/OFDM, and transmits an RF modulated signal from a transmission antenna.

"Receiving Device Configuration"
FIG. 15 shows a configuration example of the receiving device 200. As shown in FIG. This receiver 200 has a CPU (Central Processing Unit) 201 , a receiver 202 , a demultiplexer 203 , and a compressed data buffer (cpb: coded picture buffer) 204 . The receiver 200 also has a decoder 205 , a non-compressed data buffer (dpb: decoded picture buffer) 206 , and a post-processing section 207 . The CPU 201 constitutes a control unit and controls the operation of each unit of the receiving device 200 .

The receiving section 202 demodulates the RF modulated signal received by the receiving antenna and acquires the transport stream TS. The demultiplexer 203 selectively extracts the coded image data of the picture of the layer set according to the decoding capability (Decoder temporal layer capability) from the transport stream TS, and sends it to the compressed data buffer (cpb: coded picture buffer) 204 . .

FIG. 16 shows a configuration example of the demultiplexer 203. As shown in FIG. The demultiplexer 203 has a TS adaptation field extractor 231, a clock information extractor 232, a TS payload extractor 233, a section extractor 234, a PSI table/descriptor extractor 235, and a PES packet extractor 236. ing. The demultiplexer 203 also has a PES header extractor 237 , a time stamp extractor 238 , an identification information extractor 239 , a PES payload extractor 240 and a stream composer 241 .

The TS adaptation field extraction unit 231 extracts the adaptation field from the TS packet having the adaptation field of the transport stream TS. The clock information extraction unit 232 extracts the PCR (Program Clock Reference) from the adaptation field containing the PCR, and sends the PCR to the CPU 201 .

The TS payload extraction unit 233 extracts the TS payload from the TS packets having the TS payload of the transport stream TS. The section extractor 234 extracts the section data from the TS payload containing the section data. A PSI table/descriptor extraction unit 235 analyzes the section data extracted by the section extraction unit 234 and extracts a PSI table and descriptors. The PSI table/descriptor extraction unit 235 then sends the minimum value (min) and maximum value (max) of temporal_id to the CPU 201 and to the stream construction unit 241 .

The PES packet extractor 236 extracts the PES packet from the TS payload containing the PES packet. The PES header extractor 237 extracts the PES header from the PES packets extracted by the PES packet extractor 236 . The time stamp extraction unit 238 extracts the time stamp (DTS, PTS) inserted in the PES header for each picture and sends it to the CPU 201 and the stream construction unit 241 .

The identification information extracting unit 239 extracts identification information for identifying the layer set to which the picture belongs, which is inserted in the PES header for each picture, and sends it to the stream forming unit 241 . For example, when a plurality of hierarchies are divided into a low hierarchy set and a high hierarchy set, the priority information of the 1-bit field of “PES_priority” of the PES header is extracted and sent to the stream construction section 241 . This identification information is always inserted by the transmitting side when the transport stream TS contains a single video stream, but when the transport stream TS contains a plurality of video streams, the transmitting side may not be inserted with

The PES payload extractor 240 extracts the PES payload, that is, the encoded image data of the picture of each layer from the PES packets extracted by the PES packet extractor 236 . The stream configuration unit 241 selectively extracts the encoded image data of the picture of the layer set according to the decoding capability (Decoder temporal layer capability) from the encoded image data of the picture of each layer extracted by the PES payload extraction unit 240. , to a compressed data buffer (cpb: coded picture buffer) 204 . In this case, stream configuration section 241 refers to layer information and stream configuration information obtained by PSI table/descriptor extraction section 235, identification information (priority information) extracted by identification information extraction section 239, and the like.

For example, consider a case where the frame rate of the video stream (encoded stream) included in the transport stream TS is 120 fps. For example, it is assumed that a plurality of hierarchies are divided into a hierarchy group on the low hierarchy side and a hierarchy group on the high hierarchy side, and the frame rate of each picture in each hierarchy group is 60 fps. For example, in the hierarchical encoding example shown in FIG. 3 described above, hierarchies 0 to 3 are set as a hierarchy set on the low hierarchy side, and can be decoded by a decoder that supports level_idc at 60 fps. Hierarchy 4 is a hierarchy group on the higher hierarchy side, and can be decoded by a decoder that supports level_idc at 120 fps.

In this case, the transport stream TS includes a single video stream (coded stream) having coded data of pictures in each layer, or coded image data of pictures in a layer set on the lower layer side. 2 video streams (encoded streams), a base stream (B_str) with the video stream (B_str) and an extended stream (E_str) with encoded image data of the picture of the layer set on the higher layer side.

If the decoding capability is compatible with 120 fps, the stream structuring unit 241 extracts the encoded image data of the pictures of all layers and sends them to the compressed data buffer (cpb) 204 . On the other hand, if the decoding capability is not compatible with 120 fps but is compatible with 60 fps, the stream configuration unit 241 extracts only the encoded image data of the picture of the layer set on the lower layer side, and stores the encoded image data in the compressed data buffer (cpb). 204.

FIG. 17 shows an example of picture (slice) selection by the stream configuration unit 241 when the transport stream TS contains a single video stream (encoded stream). Here, "High" indicates a picture in a layer group on the high layer side, and "Low" indicates a picture in a layer group on the low layer side. Also, "P" indicates "PES_priority".

If the decoding capability is compatible with 120 fps, the stream construction unit 241 extracts encoded image data of pictures in all layers and sends them to the compressed data buffer (cpb) 204 . On the other hand, if the decoding capability is not compatible with 120 fps but is compatible with 60 fps, the stream configuration unit 241 performs filtering based on "PES_priority" to obtain pictures in the lower layer side layer set with P=1. and send it to compressed data buffer (cpb) 204 .

FIG. 18 shows an example of picture (slice) selection by the stream configuration unit 241 when two video streams (encoded streams), a base stream and an extension stream, are included in the transport stream TS. Here, "High" indicates a picture in a layer group on the high layer side, and "Low" indicates a picture in a layer group on the low layer side. Also assume that the packet identifier (PID) of the base stream is PID A and the packet identifier (PID) of the extended stream is PID B.

If the decoding capability is compatible with 120 fps, the stream construction unit 241 extracts encoded image data of pictures in all layers and sends them to the compressed data buffer (cpb) 204 . In this case, the stream construction unit 241 converts the encoded image data of each picture into one stream based on the decode timing information and sends the stream to the compressed data buffer (cpb) 204 .

In that case, the value of DTS is considered as the decoding timing, and the stream is put together so that it monotonously increases between pictures. This picture grouping process itself is performed on a plurality of streams read out from a plurality of compressed data buffers (cpb) 204 corresponding to the number of streams, as one stream. Decoding processing may be performed.

On the other hand, if the decoding capability is not compatible with 120 fps but is compatible with 60 fps, the stream configuration unit 241 performs filtering based on the packet identifier (PID), and selects a layer set on the lower layer side, which is PID A. Only pictures are retrieved and sent to compressed data buffer (cpb) 204 .

Note that the stream structuring unit 241 has a function of selectively rewriting the decoding time stamp of the encoded image data of each picture sent to the compressed data buffer (cpb) 204 to adjust the decoding interval of the low layer pictures. As a result, even if the decoding capability of the decoder 205 is low, a reasonable decoding process is possible.

FIG. 19 is an example of hierarchical encoding shown in FIG. is selectively extracted and sent to the compressed data buffer (cpb) 204 .

FIG. 19(a) shows the decode timing before the decode interval adjustment. In this case, the decoding interval between pictures varies, and the shortest decoding interval is equal to the decoding interval of 120 fps full resolution. On the other hand, FIG. 19(b) shows the decode timing after adjusting the decode interval. In this case, the decoding interval between pictures is made equal, and the decoding interval is 1/2 of the full resolution decoding interval. In this way, the decoding interval is adjusted in each layer according to the capability of the target decoder.

FIG. 20 shows an example of the processing flow of the demultiplexer 203. As shown in FIG. This processing flow shows a case where the transport stream TS contains a single video stream (encoded stream).

The demultiplexer 203 starts processing in step ST31, and then proceeds to processing in step ST32. In this step ST32, the CPU 201 sets the decoding capability (decoder temporal layer capability). Next, in step ST33, demultiplexer 203 determines whether or not it has the ability to decode all layers.

When it is capable of decoding all layers, the demultiplexer 203 demultiplexes all TS packets passing through the corresponding PID filter and performs section parsing in step ST34. After that, the demultiplexer 203 moves to the process of step ST35.

When it is not possible to decode all layers in step ST33, the demultiplexer 203 demultiplexes TS packets with "PES_priority" of "1" and performs section parsing in step ST36. After that, the demultiplexer 203 moves to the process of step ST35.

In step ST35, the demultiplexer 203 reads the HEVC descriptor (HEVC_descriptor) and the scalability extension descriptor (scalability_extension_descriptor) in the section of the target PID, and determines the presence or absence of extension streams, the scalability type, the number and ID of streams, Get the maximum and minimum values of temporal_id and the decoder corresponding Level for each layer.

Next, in step ST37, the demultiplexer 203 transfers the PID-targeted encoded stream to the compressed data buffer (cpb) 204 and notifies the CPU 201 of the DTS and PTS. After the process of step ST37, the demultiplexer 203 ends the process in step ST38.

FIG. 21 shows an example of the processing flow of the demultiplexer 203. As shown in FIG. This processing flow shows a case where the transport stream TS contains two video streams (encoded streams), a base stream and an extension stream.

The demultiplexer 203 starts processing in step ST41, and then proceeds to processing in step ST42. In this step ST42, the CPU 201 sets the decoding capability (decoder temporal layer capability). Next, in step ST43, demultiplexer 203 determines whether or not it has the ability to decode all layers.

When it has the ability to decode all layers, the demultiplexer 203 demultiplexes a plurality of streams forming all layers using a PID filter and performs section parsing in step ST44. After that, the demultiplexer 203 moves to the process of step ST45.

When it is not possible to decode all layers in step ST43, the demultiplexer 203 demultiplexes the stream of PID=PID A and performs section parsing in step ST46. After that, the demultiplexer 203 moves to the process of step ST45.

In step ST45, the demultiplexer 203 reads the HEVC descriptor (HEVC_descriptor) and the scalability extension descriptor (scalability_extension_descriptor) in the section of the target PID, and determines the presence or absence of extension streams, the scalability type, the number and ID of streams, Get the maximum and minimum values of temporal_id and the decoder corresponding Level for each layer.

Next, in step ST47, the demultiplexer 203 combines the encoded streams to be PID into one stream based on the DTS (PTS if not present) information, and transfers the stream to the compressed data buffer (cpb) 204. , DTS, and PTS are notified to the CPU 201 . After the process of step ST47, the demultiplexer 203 ends the process in step ST48.

Returning to FIG. 15, the compressed data buffer (cpb) 204 temporarily stores the video stream (encoded stream) taken out by the demultiplexer 203 . The decoder 205 extracts the encoded image data of the picture of the layer designated as the layer to be decoded from the video stream accumulated in the compressed data buffer 204 . Then, the decoder 205 decodes the extracted encoded image data of each picture at the decoding timing of the picture, and sends the data to the uncompressed data buffer (dpb) 206 .

Here, the hierarchy to be decoded from the CPU 201 is designated by temporal_id in the decoder 205 . The designated hierarchy is all the hierarchies included in the video stream (encoded stream) taken out by the demultiplexer 203, or a part of the hierarchies on the lower hierarchy side, and is automatically set by the CPU 201 or according to a user's operation. be. Further, the decoder 205 is given decoding timing from the CPU 201 based on a DTS (Decoding Time Stamp). When decoding the encoded image data of each picture, the decoder 205 reads and uses the image data of the referenced picture from the uncompressed data buffer 206 as necessary.

FIG. 22 shows a configuration example of the decoder 205. As shown in FIG. This decoder 205 has a temporal ID analysis section 251 , a target hierarchy selection section 252 and a decoding section 253 . The temporal ID analysis unit 251 reads the video stream (encoded stream) accumulated in the compressed data buffer 204 and analyzes the temporal_id inserted in the NAL unit header of the encoded image data of each picture.

The target layer selection unit 252 extracts the encoded image data of the picture of the layer designated as the layer to be decoded from the video stream read from the compressed data buffer 204, based on the analysis result of the temporal ID analysis unit 251. . The decoding unit 253 sequentially decodes the encoded image data of each picture extracted by the target layer selection unit 252 at decoding timing, and sends the data to the non-compressed data buffer (dpb) 206 .

In this case, the decoding unit 253 analyzes the VPS and SPS, for example, grasps the bit rate level designation value “sublayer_level_idc” for each sublayer, and confirms whether decoding is possible within the decoding capability. Also, in this case, the decoding unit 253 analyzes the SEI, grasps, for example, “initial_cpb_removal_time” and “cpb_removal_delay”, and confirms whether the decoding timing from the CPU 201 is appropriate.

When decoding a slice, the decoding unit 253 acquires “ref_idx_l0_active (ref_idx_l1_active)” from the slice header as information indicating the prediction destination in the time direction, and performs prediction in the time direction. Note that the picture after decoding is processed as being referenced by other pictures, using "short_term_ref_pic_set_idx" or "it_idx_sps" obtained from the slice header as an index.

Returning to FIG. 15, the uncompressed data buffer (dpb) 206 temporarily accumulates image data of each picture decoded by the decoder 205 . The post-processing unit 207 performs processing for adjusting the frame rate of the image data of each picture sequentially read from the uncompressed data buffer (dpb) 206 at the display timing to match the display capability. In this case, the display timing is given from the CPU 201 based on a PTS (Presentation Time Stamp).

For example, when the frame rate of the image data of each picture after decoding is 120 fps and the display capability is 120 fps, the post-processing unit 207 sends the image data of each picture after decoding as it is to the display. Further, for example, when the frame rate of the image data of each picture after decoding is 120 fps and the display capability is 60 fps, the post-processing unit 207 sets the temporal resolution of the image data of each picture after decoding to 120 fps. A sub-sampling process is performed so that it becomes 1/2 times, and it is sent to a display as image data of 60 fps.

Further, for example, when the frame rate of the image data of each picture after decoding is 60 fps and the display capability is 120 fps, the post-processing unit 207 sets the temporal resolution of the image data of each picture after decoding. Interpolation processing is performed so as to double the data, and the data is sent to the display as 120 fps image data. Further, for example, when the frame rate of the image data of each picture after decoding is 60 fps and the display capability is 60 fps, the post-processing unit 207 sends the image data of each picture after decoding as it is to the display.

FIG. 23 shows a configuration example of the post-processing unit 207. As shown in FIG. In this example, as described above, the frame rate of the image data of each picture after decoding is 120 fps or 60 fps and the display capability is 120 fps or 60 fps.

The post-processing section 207 has an interpolation section 271 , a sub-sampling section 272 and a switch section 273 . The image data of each picture after decoding from the uncompressed data buffer 206 is input directly to the switch section 273, or input to the switch section 273 after being doubled in the frame rate by the interpolation section 271, or input to the sub-sampling section. After the frame rate is reduced to 1/2 by 272 , it is input to the switch section 273 .

Selection information is supplied to the switch unit 273 from the CPU 201 . This selection information is generated automatically by the CPU 201 with reference to the display capability or according to user operation. The switch unit 273 selectively outputs one of the inputs based on the selection information. As a result, the frame rate of the image data of each picture sequentially read from the uncompressed data buffer (dpb) 206 at the display timing matches the display capability.

FIG. 24 shows an example of the processing flow of the decoder 205 and post-processing section 207. As shown in FIG. The decoder 205 and post-processing section 207 start processing in step ST51, and then proceed to processing in step ST52. In this step ST52, the decoder 205 reads the video stream to be decoded stored in the compressed data buffer (cpb) 204, and selects a picture of the hierarchy specified by the CPU 201 as a decoding target based on temporal_id.

Next, in step ST53, the decoder 205 sequentially decodes the encoded image data of each selected picture at the decoding timing, and transfers the decoded image data of each picture to the uncompressed data buffer (dpb) 206. , temporarily accumulates. Next, in step ST54, the post-processing section 207 reads the image data of each picture from the uncompressed data buffer (dpb) 206 at the display timing.

Next, the post-processing unit 207 determines whether the frame rate of the read image data of each picture matches the display capability. When the frame rate does not match the display capability, the post-processing section 207 sends the frame rate to the display in step ST56, and then terminates the process in step ST57. On the other hand, when the frame rate matches the display capability, the post-processing section 207 sends the image to the display without changing the frame rate in step ST58, and then ends the processing in step ST57.

The operation of the receiver 200 shown in FIG. 15 will be briefly described. The receiving section 202 demodulates the RF modulated signal received by the receiving antenna and acquires the transport stream TS. This transport stream TS is sent to the demultiplexer 203 . In the demultiplexer 203, the encoded image data of the picture of the layer group corresponding to the decoding capability (Decoder temporal layer capability) is selectively extracted from the transport stream TS, sent to the compressed data buffer (cpb) 204, and temporarily accumulated.

The decoder 205 extracts the encoded image data of the picture of the layer designated as the layer to be decoded from the video stream accumulated in the compressed data buffer 204 . Then, the decoder 205 decodes the extracted encoded image data of each picture at the decoding timing of the picture, sends the data to the uncompressed data buffer (dpb) 206, and temporarily stores it. In this case, when the encoded image data of each picture is decoded, the image data of the referenced picture is read from the uncompressed data buffer 206 and used as needed.

The image data of each picture sequentially read from the uncompressed data buffer (dpb) 206 at the display timing is sent to the post-processing unit 207 . The post-processing unit 207 performs interpolation or sub-sampling on the image data of each picture to match the frame rate with the display capability. The image data of each picture processed by the post-processing unit 207 is supplied to a display, and a moving image is displayed using the image data of each picture.

As described above, in the transmitting/receiving system 10 shown in FIG. 1, the layer of the video stream (the header of the PES packet) contains the coded image data of each picture included in the video stream on the transmitting side. ID information is inserted to identify whether the encoded image data of a picture belonging to . Therefore, for example, on the receiving side, by using this identification information, it becomes possible to selectively decode encoded image data of pictures in a hierarchy below a predetermined hierarchy according to the decoding capability.

In the transmitting/receiving system 10 shown in FIG. 1, the transmitting side inserts a scalability extension descriptor (scalability_extension_descriptor) into the layer of the transport stream TS. Therefore, for example, the receiving side can easily grasp the hierarchical information in hierarchical coding, the configuration information of the video stream included in the transport stream TS, and the like, and can perform appropriate decoding processing.

Further, in the transmitting/receiving system 10 shown in FIG. 1, the receiving side selectively compresses encoded image data of pictures in a hierarchy below a predetermined hierarchy according to the decoder temporal layer capability from the received video stream. It is taken into the data buffer 204 and decoded. Therefore, for example, appropriate decoding processing can be performed according to the decoding ability.

Further, in the transmitting/receiving system 10 shown in FIG. 1, the receiving side selectively rewrites the decoding time stamp of the encoded image data of each picture taken into the compressed data buffer 204, and adjusts the decoding interval of the low-hierarchical picture. It has Therefore, for example, even if the decoding capability of the decoder 205 is low, a reasonable decoding process is possible.

In the transmitting/receiving system 10 shown in FIG. 1, the post-processing unit 207 adjusts the frame rate of the image data of each picture after decoding to match the display capability on the receiving side. Therefore, for example, even if the decoding capability is low, it is possible to obtain image data at a frame rate suitable for high display capability.

<2. Variation>
In the above-described embodiment, the identification information that identifies to which layer set among a predetermined number of layer sets the encoded image data of each picture included in the video stream belongs is An example of insertion into the PES packet header (PES header) has been shown. However, the insertion position of this identification information is not limited to this.

For example, multiplexer 104 (see FIG. 2) may insert this identification information into the adaptation field of a TS packet that has an adaptation field. The multiplexer 104 utilizes, for example, a 1-bit field of the well-known elementary stream priority indicator (elementary_stream_priority_indicator) present in the adaptation field when dividing multiple hierarchies into a low-hierarchy set and a high-hierarchy set.

This 1-bit field is set to "1" when the payload of the subsequent TS packet contains the PES packet having the encoded image data of the picture of the layer set on the lower layer side in the payload, that is, the priority is set high. On the other hand, this 1-bit field is set to "0" when the payload of the subsequent TS packet contains the PES packet having the encoded image data of the picture of the layer set on the lower layer side in the payload, that is, the priority is set low. .

FIG. 25 shows an example of arrangement of adaptation fields. This example is a case where multiple hierarchies are divided into a low hierarchy group and a high hierarchy group, and a 1-bit field of an elementary stream priority indicator (elementary_stream_priority_indicator) is used.

In the illustrated example, a TS packet having an adaptation field is arranged immediately before each predetermined number of TS packet groups each having a divided PES packet having one picture of encoded image data as a payload. In this case, the 1-bit field of the elementary stream priority indicator is set to "1" when the one picture is a picture of the layer set on the lower layer side. On the other hand, when the one picture is a picture of a layer set on the higher layer side, the 1-bit field of the elementary stream priority indicator is set to "0".

As shown in FIG. 25, by arranging the TS packets having the adaptation field, the receiving side can recognize the coded data of the picture belonging to any layer set for each coded image data of the picture included in the video stream. can be easily identified. In the arrangement example of FIG. 25, a TS packet having an adaptation field is arranged for each picture, but every time the hierarchical group to which the picture belongs is switched, a TS packet having an adaptation field is arranged just before that. may be made to do so.

FIG. 26 shows a configuration example of the multiplexer 104A of the transmission device 100 when inserting the identification information of the hierarchy set into the adaptation field as described above. In FIG. 26, parts corresponding to those in FIG. 12 are denoted by the same reference numerals, and detailed description thereof will be omitted. This multiplexer 104A has an adaptation field priority indicator 146 instead of the PES priority generator 141 in the multiplexer 104 of FIG.

Information on the number of layers (Number of layers) and the number of streams (Number of streams) is supplied from the CPU 101 to the priority instruction unit 146 . The priority instruction unit 146 generates priority information for each hierarchy set when a plurality of hierarchies indicated by the number of hierarchies is divided into a predetermined number of hierarchy sets equal to or greater than two. For example, in the case of division into two, a value ("1" for the low layer group and "0" for the high layer group) to be inserted into the 1-bit field of the elementary stream priority indicator is generated.

The priority information of each layer set generated by the priority instruction section 146 is supplied to the transport packetization section 145 . The transport packetization unit 145 arranges a TS packet having an adaptation field immediately before each predetermined number of TS packet groups having a divided PES packet having one picture of encoded image data as a payload. In that case, the transport packetization unit 145 inserts priority information corresponding to the layer set to which the picture belongs to the adaptation field as identification information.

FIG. 27 shows a configuration example of the transport stream TS when the identification information of the layer set is inserted into the adaptation field as described above. This configuration example has substantially the same configuration as the configuration example shown in FIG. In this configuration example, there is a TS packet having an adaptation field, and identification information for identifying the layer group to which each picture belongs is inserted in this adaptation field. For example, when multiple hierarchies are divided into a low hierarchy group and a high hierarchy group, a 1-bit field of elementary stream priority indicator (elementary_stream_priority_indicator) is used.

FIG. 28 shows a configuration example of the demultiplexer 203A of the receiving device 200 when inserting the identification information of the hierarchy set into the adaptation field as described above. 28, parts corresponding to those in FIG. 16 are denoted by the same reference numerals, and detailed description thereof will be omitted. This demultiplexer 203A has an identification information extraction section 242 instead of the identification information extraction section 239 in the demultiplexer 203 of FIG.

This identification information extraction unit 242 extracts identification information from the adaptation field and sends it to the stream construction unit 241 . For example, when a plurality of hierarchies are divided into a low hierarchy group and a high hierarchy group, the priority information of the 1-bit field “elementary_stream_priority_indicator” of the adaptation field is extracted and sent to the stream configuration unit 241 .

The stream configuration unit 241 selectively extracts the encoded image data of the picture of the layer set according to the decoding capability (Decoder temporal layer capability) from the encoded image data of the picture of each layer extracted by the PES payload extraction unit 240. , to the compressed data buffer (cpb) 204 . In this case, the stream configuration unit 241 refers to layer information and stream configuration information obtained by the PSI table/descriptor extraction unit 235, identification information (priority information) extracted by the identification information extraction unit 242, and the like.

Further, although the transmission/reception system 10 including the transmission device 100 and the reception device 200 is shown in the above embodiment, the configuration of the transmission/reception system to which the present technology can be applied is not limited to this. For example, the part of the receiving device 200 may be a configuration of a set-top box and a monitor connected by a digital interface such as HDMI (High-Definition Multimedia Interface). is a trademark.

Also, in the above-described embodiment, an example in which the container is a transport stream (MPEG-2 TS) has been shown. However, the present technology can be similarly applied to a system configured to distribute to a receiving terminal using a network such as the Internet. Internet distribution is often distributed in MP4 or other format containers. In other words, containers of various formats such as a transport stream (MPEG-2 TS) adopted by the digital broadcasting standard and MP4 used for Internet distribution correspond to containers.

For example, FIG. 29 shows a configuration example of an MP4 stream. This MP4 stream contains "moov
, ``moof'', and ``mdat''. A video elementary stream "track1:video ES1", which is an encoded video stream, and an audio elementary stream "track1:audio ES1", which is an encoded audio stream, exist as tracks in the "mdat" box. exist

Also, in the "moof" box, "mfhd (movie fragment header" exists as a header part, and "track fragment" corresponding to each track exists as its data part. Video elementary stream "track1: "track1 fragment(video)" corresponding to "video ES1" has "Independent and disposal samples", in which a box called "SampleDependencyTypeBox" corresponding to each picture is inserted.

In this box, it is possible to insert identification information for identifying to which layer group the encoded image data of each picture belongs. For example, when dividing a plurality of hierarchies into two hierarchies, the highest layer and the other lower layers, using the 2-bit field of "sample_depends_on" and the 2-bit field of "sample_is_dependent_on", the identification information Insertion is possible.

FIG. 30 shows a structural example (Syntax) of "SampleDependencyTypeBox". Also, FIG. 31 shows the content (Semantics) of main information in the structural example. In this case, "sample_depends_on" is set to "1" to indicate that it refers to other pictures and is not an I picture, and "sample_is_dependent_on" is set to "2" to indicate that it is not referred to by other pictures. can be identified as a picture belonging to the set of . In other states, the picture can be identified as a picture belonging to a hierarchy group of hierarchy layers.

It is also possible to use a newly defined box called "SampleScalablePriorityBox" instead of using the "SampleDependencyTypeBox" box. FIG. 32 shows a structural example (Syntax) of "SampleScalablePriorityBox". Also, FIG. 33 shows the content (Semantics) of main information in the structural example.

In this case, when dividing a plurality of hierarchies into two hierarchy sets, a lowest hierarchy set and a high hierarchy set, the identification information is inserted using a 2-bit field of “base_and_priority”. That is, by setting "base_and_priority" to, for example, "1", it is possible to identify a picture that has a low priority and belongs to a high hierarchy group. On the other hand, by setting "base_and_priority" to, for example, "2", it is possible to identify the picture as having a high priority and belonging to the low hierarchy set.

Moreover, this technique can also take the following structures.
(1) classifying image data of each picture constituting moving image data into a plurality of hierarchies, encoding the image data of the classified pictures of each hierarchy, and converting the encoded image data of the pictures of each hierarchy; an image encoder that generates video data having
a transmission unit for transmitting a container of a predetermined format containing the generated video data;
dividing the plurality of hierarchies into a predetermined number of hierarchies of two or more, and coding a picture belonging to which hierarchy set the encoded image data of each picture contained in the video data is stored in a packet containing the video data; A transmission device comprising an identification information inserting unit for inserting identification information for identifying image data.
(2) The transmitting device according to (1), wherein the identification information is priority information that is set higher for a layer set on the lower layer side.
(3) The transmission device according to (1), wherein the identification information is inserted into a header of a PES packet containing encoded image data for each picture in a payload.
(4) The transmitting device according to (3), wherein the identification information is inserted using a PES priority field of the header.
(5) The transmitting device according to (1), wherein the identification information is inserted into an adaptation field of a TS packet having an adaptation field.
(6) The transmitting apparatus according to (5), wherein the identification information is inserted using the ES priority indicator field of the adaptation field.
(7) The transmitting device according to (1), wherein the identification information is inserted in a header box associated with the track of the corresponding picture.
(8) The image encoding unit
generating a single video stream having encoded image data of pictures in each layer, or generating a predetermined number of video data each having encoded image data of pictures in each layer set;
The transmission device according to any one of (1) to (7), further comprising a configuration information inserting unit that inserts configuration information of a video stream included in the container into the layer of the container.
(9) classifying the image data of each picture constituting the video data into a plurality of layers, encoding the image data of the classified pictures of each layer, and converting the encoded image data of the pictures of each layer; an image encoding step to generate video data having
a transmission step of transmitting, by a transmission unit, a container of a predetermined format containing the generated video data;
dividing the plurality of hierarchies into a predetermined number of hierarchies of two or more, and coding a picture belonging to which hierarchy set the encoded image data of each picture contained in the video data is stored in a packet containing the video data; A transmission method comprising an identification information inserting step of inserting identification information for identifying image data.
(10) A container of a predetermined format containing video data having coded image data of pictures in each hierarchy obtained by classifying and encoding image data of each picture constituting moving image data into a plurality of hierarchies. a receiver for receiving
coded image data of pictures in a hierarchy below a predetermined hierarchy according to decoding capability are selectively fetched into a buffer from the video data contained in the received container, and coded image data of each picture fetched into the buffer and obtains image data of pictures in layers below the predetermined layer by decoding.
(11) The plurality of hierarchies are divided into a predetermined number of hierarchy sets of two or more, and encoded image data of each picture included in the video data is stored in a packet containing the video data to which hierarchy set each picture belongs. identification information is inserted to identify whether it is encoded image data of
The receiving device according to (10), wherein the image decoding unit loads and decodes encoded image data of pictures of a predetermined hierarchy set according to the decoding ability into the buffer based on the identification information.
(12) The receiving device according to (11), wherein the identification information is inserted in a header of a PES packet containing encoded image data for each picture in a payload.
(13) The receiving device according to (11), wherein the identification information is inserted into an adaptation field of a TS packet having an adaptation field.
(14) The transmitting device according to (11), wherein the identification information is inserted in a header box associated with the track of the corresponding picture.
(15) The plurality of layers are divided into a predetermined number of layer sets equal to or greater than two, and the received container contains the predetermined number of video streams each having encoded image data of pictures of the predetermined number of layer sets. contains
The receiving device according to (10), wherein the image encoding unit captures encoded image data of pictures of a predetermined layer set corresponding to the decoding capability into the buffer and decodes the encoded image data based on the stream identification information.
(16) The image decoding unit
When the coded image data of the pictures of the predetermined layer group are included in a plurality of video streams, the coded image data of each picture are made into one stream based on the decoding timing information and taken into the buffer (15). 2. The receiving device according to .
(17) The image decoding unit
The receiving device according to any one of (10) to (16) above, having a function of rewriting the decoding time stamp of the encoded image data of each picture selectively loaded into the buffer to adjust the decoding interval of the low-hierarchical picture. .
(18) The receiving device according to any one of (10) to (17), further comprising a post processing unit that matches the frame rate of the image data of each picture obtained by the image decoding unit to the display capability.
(19) Includes video data having encoded image data of pictures in each layer obtained by classifying and encoding image data of each picture constituting moving image data into a plurality of layers by a receiving unit. a receiving step of receiving a container in a predetermined format;
coded image data of pictures in a hierarchy below a predetermined hierarchy according to decoding capability are selectively fetched into a buffer from the video data contained in the received container, and coded image data of each picture fetched into the buffer is decoded to obtain image data of a picture in a hierarchy below the predetermined hierarchy.

The main feature of this technology is that a packet containing video data contains identification information that identifies to which layer set the encoded image data of each picture included in this video data belongs. By inserting , it becomes possible for the receiving side to selectively decode the encoded image data of the picture in the hierarchy below the predetermined hierarchy according to the decoding capability by using this identification information. (see FIG. 12).

DESCRIPTION OF SYMBOLS 10... Transmission/reception system 100... Transmission apparatus 101... CPU
102 Encoder 103 Compressed data buffer (cpb)
104, 104A Multiplexer 105 Transmitting unit 141 PES priority generating unit 142 Section coding unit 143-1 to 143-N PES packetizing unit 144 Switching unit 145. Transport packetization unit 146 Adaptation field/priority instruction unit 200 Receiving device 201 CPU
202... Receiving unit 203... Demultiplexer 204... Compressed data buffer (cpb)
205 decoder 206 uncompressed data buffer (dpb)
207... Post-processing unit 231... TS adaptation field extractor 232... Clock information extractor 233... TS payload extractor 234... Section extractor 235... PSI table/descriptor extractor 236 PES packet extraction unit 237 PES header extraction unit 238 Timestamp extraction unit 239 Identification information extraction unit 240 PES payload extraction unit 241 Stream configuration unit 242 Identification information extraction unit 251 Temporal ID analysis unit 252 Target hierarchy selection unit 253 Decoding unit 271 Interpolation unit 272 Sub-sampling unit 273 Switch unit

Claims (5)

A first video stream having encoded image data of pictures on the low layer side and encoded image data of pictures on the high layer side, which are generated by hierarchically encoding image data of each picture constituting moving image data, and the coded image data of each picture included in the first video stream corresponding to the first video stream belongs to the coded image data of the picture on the lower hierarchy side and the encoded image data of each picture included in the second video stream corresponding to the second video stream is the encoded image data of the picture on the high layer side. a first descriptor in which a first value based on a frame rate consisting only of pictures on the low layer side corresponding to the first video stream is inserted; receiving a multiplexed stream including a second descriptor in which a second value based on a frame rate is inserted, which is composed of the low-hierarchy picture and the high-hierarchy picture, corresponding to the second video stream; a receiver,
Based on the first stream identification information and the second stream identification information, and the first value and the second value, only the first video stream or the first extracting and decoding both the first video stream and the second video stream, and increasing or decreasing the frame rate of the image data of each decoded picture based on the display capability of the display unit; receiving device, further comprising a processing unit that performs The receiving device according to claim 1, wherein the processing unit increases or decreases the frame rate of the decoded image data of each picture so as to match the display capability of the display unit. 3. The receiving device according to claim 1, wherein the processing unit performs interpolation processing to increase the frame rate of the decoded image data of each picture based on the display capability of the display unit. 4. The receiving device according to any one of claims 1 to 3, wherein layer identification information is added to the encoded image data of each of the hierarchically encoded pictures. A first video stream having encoded image data of a picture on the low layer side and a code of a picture on the high layer side, which is generated by hierarchically encoding image data of each picture constituting moving image data by a receiving unit. a second video stream having coded image data, and the coded image data of each picture included in the first video stream corresponding to the first video stream is the coding of the picture on the lower layer side; first stream identification information indicating that it belongs to the image data, and encoded image data of each picture included in the second video stream corresponding to the second video stream is the code of the picture on the high layer side. second stream identification information indicating that the first video stream belongs to the converted image data, and a first value based on a frame rate consisting only of pictures on the low layer side corresponding to the first video stream. multiplexing including a descriptor, and a second descriptor in which a second value based on a frame rate is inserted, which is composed of the low-hierarchy-side picture and the high-hierarchy-side picture corresponding to the second video stream; having a receiving step for receiving the stream;
a processing unit extracting only the first video stream from the multiplexed stream based on the first stream identification information, the second stream identification information, and the first value and the second value; Alternatively, both the first video stream and the second video stream are extracted and decoded, and the frame rate of the image data of each decoded picture is increased or increased based on the display capability of the display unit. The receiving method, further comprising a processing step of reducing processing.

Images