JP2019062566A - Transmission apparatus, transmission method, reception apparatus, and reception method - Google Patents

Abstract

To allow for excellent decoding depending on the decoding capability on the receiving side.SOLUTION: Image data of each picture composing moving image data is hierarchically encoded to generate a first video stream having encoded image data of a picture on a low hierarchy side and a second video stream having encoded image data of a picture on a high hierarchy side. The first video stream and the second video stream are transmitted. Identification information of the first video stream is transmitted corresponding to the first video stream, and a video stream obtained by combining the first video stream and the second video stream is transmitted corresponding to the second video stream.SELECTED DRAWING: Figure 12

Description

  The present technology relates to a transmission device, a transmission method, a reception device, and a reception method. More specifically, the present technology relates to a transmitting apparatus and the like that hierarchically code and transmit image data of each picture that constitutes moving image data.

  When a compressed moving image is serviced by broadcast, net, etc., the decoding capability of the receiver limits the upper limit of the reproducible frame frequency. Therefore, it is necessary for the service side to limit to only low frame frequency services or to simultaneously provide high and low frame frequency services in consideration of the reproduction capability of popular receivers.

  Receivers are expensive to support high frame frequency services, which is an impediment to early adoption. In the early stage, only low-cost receivers dedicated to low frame frequency services were popularized, and if the service side started offering high frame frequency services in the future, it would not be possible to view them at all without new receivers, and the widespread use of new services. It becomes an inhibiting factor of

  For example, in HEVC (High Efficiency Video Coding), temporal scalability is proposed by hierarchically coding image data of each picture constituting moving image data (see Non-Patent Document 1). On the receiving side, the hierarchy of each picture can be identified based on the temporal ID (temporal_id) inserted in the header of the Network Abstraction Layer (NAL) unit, and selective decoding up to the hierarchy corresponding to the decoding capability is possible. .

Gary J. Sullivan, Jens-Rainer Ohm, Woo-Jin Han, Thomas Wiegand, “Overview of the High Efficiency Video Coding (HEVC) Standard” IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECNOROGY, VOL. 22, NO. 1649-1668, DECEMBER 2012

  An object of the present technology is to enable good decoding processing according to decoding capability on the receiving side.

The concept of this technology is
The image data of each picture constituting moving picture data is classified into a plurality of layers, the image data of the pictures of each classified layer is encoded, and the video data having the image data of the encoded pictures of each layer An image encoding unit that generates
A transmitting unit for transmitting a container of a predetermined format including the generated video data;
The plurality of layers are divided into two or more predetermined number of layer sets, and the packet containing the video data is encoded with the picture to which the encoded image data of each picture included in the video data belongs. A transmission apparatus comprising: an identification information insertion unit that inserts identification information that identifies image data.

  In the present technology, the image coding unit codes the image data of each picture constituting the moving image data to generate video data. In this case, image data of each picture constituting moving image data is classified into a plurality of layers and encoded, and video data having encoded image data of a picture of each layer is generated.

  The transmitting unit transmits a container of a predetermined format including the video data described above. For example, the container may be a transport stream (MPEG-2 TS) adopted in a digital broadcast standard. Also, for example, the container may be a container of MP4 or other format used in distribution of the Internet or the like.

  The identification information insertion unit divides the plurality of layers into two or more predetermined number of layer sets, and in a packet that contains video data, encoded image data of each picture included in the video data belongs to which layer set Identification information for identifying whether it is encoded image data of a picture is inserted. For example, the identification information may be set to be priority information which is set to be higher as the layer set on the lower layer side is higher.

  For example, the identification information may be inserted into the header of a PES packet that includes encoded image data for each picture in the payload. And in this case, for example, identification information may be inserted using the PES priority field of the header. Also, for example, the identification information may be inserted into the adaptation field of the TS packet having the adaptation field. And, in this case, for example, identification information may be inserted using the field of the ES priority indicator of the adaptation field. Also, for example, the identification information may be inserted into the box of the header associated with the track of the corresponding picture.

  As described above, in the present technology, identification information that identifies to which hierarchical set the encoded image data of each picture included in the video data belongs to a picture that belongs to a packet that contains the video data. Is to be inserted. Therefore, on the reception side, by using this identification information, it becomes possible to selectively decode encoded image data of a picture of a hierarchy below a predetermined hierarchy according to the decoding capability.

  In the present technology, for example, the image coding unit generates a single video stream having coded image data of a picture of each layer, or divides a plurality of layers into a predetermined number of layer sets or more. And a configuration information insertion unit that generates a predetermined number of video streams each having encoded image data of pictures of each hierarchical set, and inserts configuration information of the video stream included in the container into the layer of the container. It may be done. In this case, for example, on the receiving side, the configuration of the video stream can be easily grasped based on the configuration information of the video stream included in the container.

Also, the other concept of this technology is
Receive a container of a predetermined format including video data having encoded image data of a picture of each layer obtained by classifying and encoding image data of each picture constituting moving image data into a plurality of layers A receiver,
From the video stream included in the received container, encoded image data of a picture of a layer of a predetermined hierarchy or less according to the decoding capability is selectively taken into a buffer, and encoded image data of each picture taken into the buffer And an image decoding unit for obtaining image data of a picture of a hierarchy below the predetermined hierarchy.

  In the present technology, the receiving unit receives a container of a predetermined format. The container includes video data having image data of pictures of each layer obtained by classifying and coding image data of each picture constituting moving image data into a plurality of layers.

  From the video data contained in the received container, the image decoding unit selectively captures encoded image data of a picture of a hierarchy of a predetermined hierarchy or lower according to the decoding capability into each buffer, and each picture captured in this buffer The encoded image data is decoded to obtain image data of a picture of a hierarchy below a predetermined hierarchy.

  For example, a plurality of layers are divided into two or more predetermined number of layer sets, and the packet containing the video data is encoded with the picture to which the encoded image data of each picture included in the video data belongs. Identification information for identifying whether the data is image data is inserted, and based on the identification information, the image decoding unit takes encoded image data of a picture of a predetermined hierarchical set according to the decoding capability into a buffer and decodes it. , May be.

  In this case, for example, the identification information may be inserted in the header of a PES packet that includes encoded image data for each picture in the payload. Also, in this case, for example, the identification information may be inserted in the adaptation field of the TS packet having the adaptation field. Also, in this case, for example, the identification information may be inserted into the box of the header associated with the track of the corresponding picture.

  Also, for example, the plurality of layers are divided into two or more predetermined number of layer sets, and the received container includes a predetermined number of video streams each having encoded image data of a predetermined number of layer sets of pictures. The image encoding unit may be configured to, based on the stream identification information, capture and decode encoded image data of a picture of a predetermined hierarchical set according to the decoding capability into a buffer. At this time, for example, when the encoded image data of a picture of a predetermined hierarchical set is included in a plurality of video streams, the image decoding unit determines one stream of encoded image data of each picture based on the decode timing information. May be stored in the buffer.

  As described above, in the present technology, encoded image data of a picture of a hierarchy of a predetermined hierarchy or lower according to the decoding capability is selectively taken from a received video data into a buffer and decoded. Therefore, appropriate decoding processing can be performed according to the decoding capability.

  In the present technology, for example, the image decoding unit has a function of rewriting the decoding time stamp of the coded image data of each picture selectively taken into the buffer and adjusting the decoding interval of the low hierarchical picture. It may be done. In this case, even a decoder with low decoding capability can perform reasonable decoding.

  Further, in the present technology, for example, a post processing unit may be further provided that matches the frame rate of the image data of each picture obtained by the image decoding unit with the display capability. In this case, even when the decoding capability is low, it is possible to obtain image data of a frame rate that is suitable for high display capability.

  According to the present technology, it is possible to perform good decoding processing according to the decoding capability on the receiving side. In addition, the effect described here is not necessarily limited, and may be any effect described in the present disclosure.

It is a block diagram showing an example of composition of a transmitting and receiving system as an embodiment. It is a block diagram which shows the structural example of a transmitter. It is a figure which shows an example of the hierarchy coding performed by an encoder. It is a figure which shows the example of a structure (Syntax) of a NAL unit header, and the content (Semantics) of the main parameter in the example of a structure. It is a figure for demonstrating the structure of the coding image data of each picture by HEVC. It is a figure which shows an example of encoding in the case of hierarchical encoding, decoding, a display order, and a delay. It is a figure which shows the coding stream of hierarchy coding, and the display expectation (display order) in a designation | designated hierarchy. It is a figure which shows the structural example (Syntax) of the HEVC descriptor (HEVC_descriptor). It is a figure which shows the content (Semantics) of the main information in the structural example of a HEVC descriptor. It is a figure which shows the structural example (Syntax) of a scalability extension descriptor (scalability_extension_descriptor). It is a figure which shows the content (Semantics) of the main information in the structural example of a scalability extension descriptor. It is a block diagram showing an example of composition of a multiplexer. It is a figure which shows an example of the processing flow of a multiplexer. It is a figure which shows the structural example of transport stream TS in the case of performing delivery by a single stream. It is a block diagram showing an example of composition of a receiving set. It is a block diagram which shows the structural example of a demultiplexer. It is a figure which shows the case where transport stream TS contains a single video stream (coding stream). It is a figure which shows the case where transport stream TS contains two video streams (coding stream) of a base stream and an extended stream. It is a figure for demonstrating the function which rewrites the decoding time stamp of the coding image data of each picture, and adjusts the decoding space | interval of a low hierarchy picture. It is a figure which shows an example of the processing flow (1 frame) of a demultiplexer. It is a figure which shows an example of the processing flow (2 frames) of a demultiplexer. It is a block diagram which shows the structural example of a decoder. It is a figure which shows the structural example of a post process part. It is a figure which shows an example of the processing flow of a decoder and a post process part. It is a figure which shows the example of arrangement | positioning of an adaptation field. It is a block diagram showing an example of composition of a multiplexer in the case of inserting discernment information on a hierarchy set in an adaptation field. It is a figure which shows the structural example of transport stream TS in, when inserting identification information of a hierarchy set in an adaptation field. It is a block diagram which shows the structural example of the demultiplexer in, when inserting identification information of a hierarchy set in an adaptation field. It is a figure which shows the structural example of MP4 stream. It is a figure which shows the structural example of "SampleDependencyTypeBox." It is a figure which shows the content of the main information in the structural example of "SampleDependencyTypeBox". It is a figure which shows the structural example of "SampleScalablePriorityBox." It is a figure which shows the content of the main information in the structural example of "SampleScalablePriorityBox."

Hereinafter, modes for carrying out the invention (hereinafter referred to as “embodiments”) will be described. The description will be made in the following order.
1. Embodiment 2. Modified example

<1. Embodiment>
[Transmit / receive system]
FIG. 1 shows a configuration example of a transmission / reception system 10 as an embodiment. The transmission / reception system 10 is configured to include a transmission device 100 and a reception device 200.

  The transmitting apparatus 100 transmits a transport stream TS as a container on broadcast waves. In the transport stream TS, image data of each picture constituting moving image data is classified into a plurality of layers, and a video stream having encoded data of image data of pictures of each layer is included. In this case, for example, H. Coding such as H.264 / AVC, HEVC, etc. is applied, and the reference picture is coded to belong to the self hierarchy and / or a hierarchy lower than the self hierarchy.

  Layer identification information for identifying the belonging layer is added to the encoded image data of the picture of each layer for each picture. In this embodiment, hierarchy identification information (“nuh_temporal_id_plus1” meaning temporal_id) is disposed in the header portion of the NAL unit (nal_unit) of each picture. Thus, layer identification information is added, so that layer identification of each picture can be made in the layer of the NAL unit on the receiving side, and encoded image data of a layer below a predetermined layer is selectively extracted for decoding processing. It can be carried out.

  In this embodiment, the plurality of layers are divided into two or more predetermined number of layer sets, and in the layer of the video stream, the code of the picture to which the coded image data of each picture possessed by this video stream belongs. Identification information for identifying whether the image data is an image data is inserted.

  In this embodiment, this identification information is set as priority information which is set higher as the layer set on the lower layer side is higher, and is inserted into the header of a PES packet including encoded image data for each picture in the payload. With this identification information, on the receiving side, it becomes possible to take in and process only the coded image data of the picture of the hierarchical set according to its own decoding capability.

  The transport stream TS includes a single video stream having encoded image data of a picture of each layer or a predetermined number of video streams having encoded image data of a picture of each hierarchical set described above. In the transport stream TS, hierarchical information of hierarchical coding and configuration information of a video stream are inserted. With this information, the receiving side can easily grasp the hierarchical configuration and the stream configuration, and can perform appropriate decoding processing.

  The receiving device 200 receives the above-described transport stream TS sent from the transmitting device 100 on a broadcast wave. The receiving apparatus 200 selectively fetches, from the video stream included in the transport stream TS, encoded image data of a picture of a hierarchy of a predetermined hierarchy or less selected according to the decoding capability into a buffer and decodes it. Image data is acquired and image reproduction is performed.

  For example, as described above, the transport stream TS may include a single video stream having encoded image data of pictures of a plurality of layers. In that case, based on the above-mentioned identification information, encoded image data of a picture of a predetermined hierarchical set according to the decoding capability is taken into a buffer and processed.

  Also, for example, as described above, the transport stream TS includes a predetermined number of video streams each having the encoded image data of two or more predetermined number of layers of pictures obtained by dividing a plurality of layers. It may have been. In that case, based on the stream identification information, encoded image data of a picture of a predetermined hierarchical set according to the decoding capability is taken into a buffer and processed.

  Also, the receiving device 200 performs processing of rewriting the decoding time stamp of the encoded image data of each picture selectively taken into the buffer and adjusting the decoding interval of the low-layer picture. By this adjustment processing, even a decoder with low decoding capability can perform reasonable decoding processing.

  Also, the receiving apparatus 200 performs post processing to match the frame rate of the image data of each picture obtained by decoding as described above to the display capability. By this post-processing, for example, even when the decoding capability is low, it is possible to obtain image data of a frame rate that is suitable for high display capability.

"Configuration of transmitter"
FIG. 2 shows a configuration example of the transmission device 100. The transmitting apparatus 100 includes a central processing unit (CPU) 101, an encoder 102, a compressed data buffer (cpb: coded picture buffer) 103, a multiplexer 104, and a transmitting unit 105. The CPU 101 is a control unit and controls the operation of each unit of the transmission apparatus 100.

  The encoder 102 inputs uncompressed moving image data and performs hierarchical coding. The encoder 102 classifies the image data of each picture constituting the moving image data into a plurality of layers. Then, the encoder 102 encodes the image data of the classified picture of each layer to generate a video stream having the encoded image data of the picture of each layer. The encoder 102 may, for example, H.264 / AVC, HEVC, etc. are encoded. At this time, the encoder 102 encodes such that the picture to be referred (reference picture) belongs to a self hierarchy and / or a hierarchy lower than the self hierarchy.

  FIG. 3 shows an example of hierarchical coding performed by the encoder 102. This example is an example of classification into five layers from 0 to 4, and for example, HEVC coding is performed on image data of pictures of each layer.

  The vertical axis shows the hierarchy. 0 to 4 are respectively set as temporal_id (layer identification information) arranged in the header portion of the NAL unit (nal_unit) constituting the encoded image data of the pictures of layers 0 to 4. On the other hand, the horizontal axis indicates the display order (POC: picture order of composition), the left side is the display time before, and the right side is the display time after.

  FIG. 4A shows a structural example (Syntax) of the NAL unit header, and FIG. 4B shows the contents (Semantics) of main parameters in the structural example. A 1-bit field of "Forbidden_zero_bit" requires 0. The 6-bit field of "Nal_unit_type" indicates the NAL unit type. The 6-bit field of "Nuh_layer_id" assumes 0. The 3-bit field of "Nuh_temporal_id_plus1" indicates temporal_id, and takes a value (1 to 7) to which 1 is added.

  Returning to FIG. 3, each rectangular frame indicates a picture, and the numbers indicate the order of the pictures being coded, that is, the encoding order (decoding order on the receiving side). A sub picture group (Sub group of pictures) is composed of 16 pictures from “1” to “17” (except “2”), and “1” is the head picture of the sub picture group . "2" is the leading picture of the next sub-picture group. Alternatively, except for “1”, a sub-picture group is configured by 16 pictures from “2” to “17”, and “2” is the top picture of the sub-picture group.

  The picture of “1” can be the top picture of a GOP (Group Of Pictures). As shown in FIG. 5, the encoded image data of the first picture of the GOP is configured by NAL units of AUD, VPS, SPS, PPS, PSEI, SLICE, SSEI, and EOS. On the other hand, pictures other than the head picture of the GOP are configured by NAL units of AUD, PPS, PSEI, SLICE, SSEI, and EOS. VPS and SPS can be transmitted once in a sequence (GOP) and PPS can be transmitted in My picture.

  Returning to FIG. 3, solid arrows indicate reference relationships of pictures in encoding. For example, the picture of "1" is an I picture and does not refer to other pictures. The picture of “2” is a P picture and is encoded with reference to the picture of “1”. Also, the picture of “3” is a B picture, and is encoded with reference to the pictures of “1” and “3”. Hereinafter, similarly, other pictures are encoded with reference to nearby pictures in display order. Note that the picture of layer 4 has no reference from other pictures.

  The encoder 102 generates a single video stream (single stream) having encoded image data of a picture of each layer, or divides a plurality of layers into two or more predetermined number of layer sets, and each layer set And generate a predetermined number of video streams (multistreams) each having encoded image data of a picture of. For example, in the example of layer coding in FIG. 3, when layers 0 to 3 are divided into lower layer hierarchy sets and layer 4 is divided into upper layer hierarchy sets into two layer sets, the encoder 102 generates each layer group. And generate two video streams (coded streams) each having coded image data of a picture of.

  As described above, the encoder 102 divides a plurality of layers into a predetermined number or more of a predetermined number of layer sets regardless of the number of video streams to be generated, and belongs to the encoded image data of the picture of each layer set. Add identification information to identify the In this case, for example, “general_level_idc”, which is a level designation value of a bit stream included in the SPS, is used as identification information, and the higher the layer set is, the higher the value is. In addition, since "sub_layer_level_idc" can be sent by SPS for every sub layer (sublayer), you may use this "sub_layer_level_idc" as identification information. The above is supplied not only to SPS but also to VPS.

  In this case, the value of the level designation value of each hierarchical set is a value corresponding to the frame rate including the picture of the hierarchical set and the pictures of all hierarchical sets lower than the hierarchical set. For example, in the example of layer coding in FIG. 3, the level designation value of the layer set of layers 0 to 3 is a value corresponding to the frame rate consisting of only pictures of layers 0 to 3, and the level of the layer set of layer 4 The designated value is a value corresponding to a frame rate consisting of pictures of all the layers 0 to 4.

  FIG. 6 shows an example of encoding, decoding, display order and delay in hierarchical coding. This example corresponds to the hierarchical coding example of FIG. 3 described above. This example shows the case of hierarchically coding all layers (all layers) at full time resolution. FIG. 6 (a) shows an encoder input. As shown in FIG. 6B, each picture is encoded in the encoding order with a delay of 16 pictures to obtain a coded stream. Further, FIG. 6 (b) shows a decoder input, in which each picture is decoded in decoding order. Then, as shown in FIG. 6C, the image data of each picture can be obtained in display order with a delay of 4 pictures.

  FIG. 7A shows a coded stream similar to the coded stream shown in FIG. 6B described above, divided into three stages of hierarchy 0 to 2, hierarchy 3 and hierarchy 4. Here, "Tid" indicates temporal_id. FIG. 7B shows display expectation (display order) in the case of selectively decoding each picture of the hierarchical layers 0 to 2, that is, the partial hierarchy of Tid = 0 to 2. Further, FIG. 7C shows display expectation (display order) in the case of selectively decoding each picture of hierarchical layers 0 to 3, that is, a partial hierarchy of Tid = 0 to 3. Further, FIG. 7 (d) shows display expectation (display order) in the case of selectively decoding each picture of layers 0 to 4, that is, all layers of Tid = 0 to 4.

  In order to decode the encoded stream of FIG. 7A separately according to the decoding capability, decoding capability with full time resolution is required. However, when decoding with Tid = 0 to 2, a decoder with 1/4 decoding capability should be able to process for the encoded full time resolution. Also, in the case of decoding Tid = 0 to 3, a decoder having 1/2 decoding capability should be able to process for the encoded full time resolution.

  However, if the pictures belonging to the lower hierarchy referred to in hierarchical coding are continuous and they are encoded at full timing in time resolution, the ability of the decoder to partially decode will not catch up. The period A in FIG. 7A corresponds thereto. A decoder that decodes a partial hierarchy of Tid = 0 to 2 or Tid = 0 to 3 performs decoding and display with a time axis having a capability of 1⁄4 or 1⁄2 as shown in the display example. , A period encoded time resolution is full and decoding of consecutive pictures is not possible.

  Ta indicates the time required for the decoding process for each picture in the decoder that decodes Tid = 0-2. Tb indicates the time required for the decoding process for each picture in the decoder that decodes Tid = 0-3. Tc indicates the time required for the decoding process for each picture in the decoder that decodes Tid = 0 to 4 (all layers). The relationship between these times is Ta> Tb> Tc.

  In this embodiment, as will be described later, the receiving device 200 has a decoder with low decoding capability and rewrites the decoding time stamp (DTS) when selectively decoding low-layer pictures. It is made to have a function to adjust the decoding interval of low hierarchy pictures. As a result, even a decoder with low decoding capability can perform reasonable decoding processing.

  Returning to FIG. 2, the compressed data buffer (cpb) 103 temporarily accumulates a video stream including encoded data of pictures of each layer, generated by the encoder 102. The multiplexer 104 reads the video stream stored in the compressed data buffer 103, converts it into PES packet, and transport packetizes and multiplexes it to obtain the transport stream TS as a multiplexed stream.

  In this embodiment, as described above, the plurality of layers are divided into two or more predetermined number of layer sets. The multiplexer 104 inserts identification information identifying encoded image data of a picture belonging to which hierarchical set into which encoded image data of each picture included in the video stream belongs, in the header (PES header) of the PES packet. With this identification information, on the receiving side, it becomes possible to take in and process only the coded image data of the picture of the hierarchical set according to its own decoding capability.

  The multiplexer 104 uses, for example, when dividing a plurality of layers into a lower layer set and a higher layer set, a 1-bit field of a known PES priority (PES_priority) present in the PES header. This 1-bit field is set to “1”, that is, the priority is high when the PES payload includes encoded image data of a picture of a layer set on the lower layer side. On the other hand, this 1-bit field is set to “0”, that is, the priority is low when the PES payload includes the encoded image data of the picture of the layer set on the high layer side.

  As described above, the transport stream TS includes a single video stream having encoded image data of a picture of each layer or a predetermined number of video streams each having encoded image data of a picture of each hierarchical set described above. Is included. The multiplexer 104 inserts layer information and stream configuration information into the transport stream TS.

  The transport stream TS includes a PMT (Program Map Table) as one of PSI (Program Specific Information). In this PMT, there is a video elementary loop (video ES1 loop) having information related to each video stream. In this video elementary loop, information such as stream type and packet identifier (PID) is arranged corresponding to each video stream, and a descriptor that describes information related to the video stream is also arranged.

  The multiplexer 104 inserts an HEVC descriptor (HEVC_descriptor) as one of the descriptors, and further inserts a scalability extension descriptor (scalability_extension_descriptor) to be newly defined.

  FIG. 8 shows a structural example (Syntax) of the HEVC descriptor (HEVC_descriptor). FIG. 9 also shows the contents (Semantics) of main information in the structural example.

  An 8-bit field of “descriptor_tag” indicates a descriptor type, which is an HEVC descriptor here. The 8-bit field of “descriptor_length” indicates the length (size) of the descriptor, and indicates the number of subsequent bytes as the length of the descriptor.

  An 8-bit field of "level_idc" indicates a bit rate level designation value. Also, when “temporal_layer_subset_flag = 1”, a 5-bit field “temporal_id_min” and a 5-bit field “temporal_id_max” exist. “Temporal_id_min” indicates the value of temporal_id of the lowest layer of the layer encoded data included in the corresponding video stream. “Temporal_id_max” indicates the value of temporal_id of the highest layer of the layer encoded data of the corresponding video stream.

  The 1-bit field of “level_constrained_flag” is newly defined, and indicates that the bit designation level value (general_level_idc) of the bit stream included in the NPS of the VPS can be changed for each picture. “1” indicates that it may change, and “0” indicates that it does not change.

  As described above, for example, “general_level_idc” is used as identification information of a belonging hierarchy set when a plurality of hierarchies are divided into two or more predetermined number of hierarchy sets. Therefore, in the case of a video stream having encoded image data of a plurality of hierarchical sets of pictures, “general_level_idc” may change for each picture. On the other hand, in the case of a video stream having encoded image data of a single hierarchical set of pictures, “general_level_idc” does not change for each picture. Alternatively, “sublayer_level_idc” is attached to each sublayer, and the decoder processes the data of the corresponding layer by reading the packet of the temporal_id in the decodable range.

  The 3-bit field "scalability_id" is newly defined, and is an ID indicating the scalability to be attached to each stream when a plurality of video streams provide scalable services. “0” indicates a base stream, and “1” to “7” are IDs that increase with the degree of scalability from the base stream.

  FIG. 10 shows a structural example (Syntax) of the scalability extension descriptor (scalability_extension_descriptor). FIG. 11 also shows the contents (Semantics) of the main information in the structural example.

  An 8-bit field of “scalability_extension_descriptor_tag” indicates a descriptor type, which indicates that it is a scalability extension descriptor. The 8-bit field of "scalability_extension_descriptor_length" indicates the length (size) of the descriptor, and indicates the number of subsequent bytes as the length of the descriptor. A 1-bit field of "extension_stream_existing_flag" is a flag indicating that there is an extension service by another stream. "1" indicates that there is an extension stream, and "0" indicates that there is no extension stream.

  The 3-bit field "extension_type" indicates the type of extension. "001" indicates that the extension is scalable in the time direction. "010" indicates that the extension is spatially scalable. “011” indicates that the extension is bit rate scalable.

  The 4-bit field "number_of_streams" indicates the total number of streams involved in the delivery service. The 3-bit field "scalability_id" is an ID indicating the scalability to be given to each stream when a plurality of video streams provide scalable services. “0” indicates a base stream, and “1” to “7” are IDs that increase with the degree of scalability from the base stream.

  The 3-bit field of "number_of_layers" indicates the total number of layers in the stream. "The 8-bit field of" sublayer_level_idc "indicates the value of level_idc to which the corresponding sublayer indicated by temporal_id includes the layer lower than that and the decoder corresponds." Number of layers "indicates the NAL unit header (NAL unit header) All values of “Nuh_temporal_id_plus1” are included, and when the demultiplexer (demuxer) detects this, “sublayer_level_idc” indicates in advance to which layer the decoder corresponding to a predetermined level_idc can decode. It becomes possible to recognize.

  As described above, in this embodiment, the level designation value (general_level_idc) of the bit rate included in the SPS is identification information of the belonging hierarchy set when the plurality of hierarchies are divided into two or more predetermined number of hierarchy sets. It is used as The value of the level designation value of each hierarchical set is a value corresponding to the frame rate composed of the picture of this hierarchical set and the pictures of all hierarchical sets lower than the hierarchical set.

  FIG. 12 shows a configuration example of the multiplexer 104. The PES priority generation unit 141, the section coding unit 142, the PES packetization units 143-1 to 143-N, the switch unit 144, and the transport packetization unit 145 are included.

  The PES packetizing units 143-1 to 143-N each read the video streams 1 to N stored in the compressed data buffer 103, and generate PES packets. At this time, the PES packetizing units 143-1 to 143-N attach time stamps of DTS (Decoding Time Stamp) and PTS (Presentation Time Stamp) to the PES header based on the HRD information of the video streams 1 to N. In this case, “cpu_removal_delay” and “dpb_output_delay” of each picture are referred to, converted to DTS and PTS with accuracy synchronized with STC (System Time Clock) time, and arranged at a predetermined position of the PES header.

  Information on the number of layers (Number of layers) and the number of streams (Number of streams) is supplied from the CPU 101 to the PES priority generation unit 141. The PES priority generation unit 141 generates priority information of each layer set in a case where a plurality of layers indicated by the number of layers are divided into a predetermined number of layer sets of 2 or more. For example, in the case of being divided into two, a value to be inserted into a 1-bit field of “PES_priority” of the PES packet header (a low hierarchy set is “1” and a high hierarchy set is “0”) is generated.

  The priority information of each hierarchy set generated by the PES priority generation unit 141 is supplied to the PES packetization units 143-1 to 143-N. The PES packetizing units 143-1 to 143-N insert the priorities of the layer sets into the header of the PES packet including the encoded image data of the picture of the layer set as identification information.

  The process of inserting the priority of the layer set to which the picture belongs to the header of the PES packet as header information in the header of the PES packet for each picture in this way is limited to the case where a single video stream (single stream) is generated by the encoder May be In this case, the processing is performed only by the PES packetizing unit 143-1.

  The switch unit 144 selectively extracts the PES packets generated by the PES packetizing units 143-1 to 143-N based on the packet identifier (PID), and sends the PES packets to the transport packetizing unit 145. The transport packetization unit 145 generates a TS packet including a PES packet in the payload to obtain a transport stream TS.

  The section coding unit 142 generates various section data to be inserted into the transport stream TS. Information on the number of layers (Number of layers) and the number of streams (Number of streams) is supplied from the CPU 101 to the section coding unit 142. The section coding unit 142 generates the above-mentioned HEVC descriptor (HEVC_descriptor) and scalability extension descriptor (scalability_extension_descriptor) based on this information.

  The section coding unit 142 sends various section data to the transport packetization unit 145. The transport packetization unit 145 generates a TS packet including the section data, and inserts the TS packet into the transport stream TS.

  FIG. 13 shows the process flow of the multiplexer 104. This example is an example of dividing a plurality of hierarchies into two, a low hierarchy set and a high hierarchy set. The multiplexer 104 starts processing in step ST1, and then proceeds to processing of step ST2. In step ST2, the multiplexer 104 sets temporal_id_ of each picture of the video stream (video elementary stream) and the number of encoded streams to be configured.

  Next, in step ST3, the multiplexer 104 determines DTS and PTS with reference to HRD information (cpu_removal_delay, dpb_output_delay), and inserts the DTS and PTS into the predetermined position of the PES header.

  Next, in step ST4, the multiplexer 104 determines whether it is a single stream (single video stream). When it is a single stream, the multiplexer 104 proceeds to perform the multiplexing process with one PID (packet identifier) in step ST5, and then proceeds to the process of step ST7.

  In this step ST7, the multiplexer 104 determines whether each of the pictures is a low hierarchical set of pictures (slices). When the picture is a low-layer set picture, in step ST8, the multiplexer 104 sets "PES_priority" of the header of the PES packet including the coded image data of the picture in the payload to "1". On the other hand, when the picture is a high hierarchy set (non-low hierarchy set), the multiplexer 104 sets “PES_priority” of the header of the PES packet including the encoded image data of the picture in the payload to “0” in step ST9. Do. After the processing of step ST8 and step ST9, the multiplexer 104 proceeds to the processing of step ST10.

  Here, the association between a picture and a slice will be described. A picture is a concept and is the same as a slice in structure definition. Although one picture is divided into a plurality of slices, it can be understood from the parameter set that the plurality of slices are the same as an access unit.

  When it is not a single stream in step ST4 described above, the multiplexer 104 proceeds to perform the multiplexing process with a plurality of packets PID (packet identifier) in step ST6, and then proceeds to the process of step ST10. In this step ST10, the multiplexer 104 inserts the encoded stream (video elementary stream) into the PES payload to form a PES packet.

  Next, the multiplexer 104 codes an HEVC descriptor, a scalability extension descriptor, etc. in step ST11. Then, the multiplexer 104 converts it into a transport packet in step ST12 to obtain a transport stream TS. Thereafter, the multiplexer 104 ends the processing in step ST13.

  FIG. 14 shows a configuration example of the transport stream TS in the case of performing delivery by a single stream. The transport stream TS includes one video stream. That is, in this configuration example, a PES packet "video PES1" of a video stream having encoded image data according to HEVC, for example, of pictures of a plurality of layers, and a PES packet "audio PES1" of an audio stream exist.

  In the encoded image data of each picture, NAL units such as VPS, SPS, SEI exist. As described above, in the header of the NAL unit of each picture, temporal_id indicating the layer of the picture is inserted. Also, for example, the VPS includes a bit rate level designation value (general_level_idc). Also, for example, “cpb_removal_delay” and “dpb_output_delay” are included in the picture timing · SEI (Picture timing SEI).

  In addition, a field indicating the priority of one bit of “PES_priority” is present in the header (PES header) of the PES packet. Based on this “PES_priority”, it is possible to identify whether the coded image data of the picture included in the PES payload is of the low hierarchy set picture or the high hierarchy set picture.

  Also, the transport stream TS includes a PMT (Program Map Table) as one of PSI (Program Specific Information). This PSI is information indicating which program each elementary stream included in the transport stream belongs to.

  In PMT, there is a program loop (Program loop) that describes information related to the entire program. Also, in PMT, there is an elementary loop having information related to each elementary stream. In this configuration example, a video elementary loop (video ES1 loop) is present, and an audio elementary loop (audio ES1 loop) is present.

  In the video elementary loop, information such as a stream type and a packet identifier (PID) is arranged corresponding to the video stream (video PES1), and a descriptor describing information related to the video stream is also arranged. Ru. The HEVC descriptor (HEVC_descriptor) and the scalability extension descriptor (scalability_extension_descriptor) described above are inserted as one of the descriptors.

  Returning to FIG. 2, the transmission unit 105 modulates the transport stream TS according to a modulation scheme suitable for broadcasting such as QPSK / OFDM, for example, and transmits an RF modulation signal from a transmission antenna.

  The operation of transmitting apparatus 100 shown in FIG. 2 will be briefly described. Uncompressed moving image data is input to the encoder 102. The encoder 102 performs hierarchical coding on the moving image data. That is, in the encoder 102, the image data of each picture constituting the moving image data is classified into a plurality of layers and encoded, and a video stream having encoded image data of a picture of each layer is generated. At this time, the picture to be referred to is encoded so as to belong to the self hierarchy and / or a hierarchy lower than the self hierarchy.

  In the encoder 102, a single video stream having encoded image data of a picture of each layer is generated, or a plurality of layers is divided into a predetermined number of layer sets of two or more, and the pictures of each layer set are A predetermined number of video streams each having encoded image data are generated.

  In addition, the video stream including the encoded data of the picture of each layer generated by the encoder 102 is supplied to the compressed data buffer (cpb) 103 and temporarily accumulated. In the multiplexer 104, the video stream stored in the compressed data buffer 103 is read, PES packetized, transport packetized and multiplexed, and a transport stream TS is obtained as a multiplexed stream.

  For example, in the case of a single video stream (single stream), the multiplexer 104 encodes a picture belonging to which hierarchical set each of encoded image data of each picture of the video stream belongs to a PES packet header (PES header). Identification information that identifies image data is inserted. For example, when dividing a plurality of layers into a lower layer set and a higher layer set, a 1-bit field of PES priority (PES_priority) of the PES header is used.

  Also, in the multiplexer 104, layer information and stream configuration information are inserted into the transport stream TS. That is, in the multiplexer 104, the HEVC descriptor (HEVC_descriptor) and the scalability extension descriptor (scalability_extension_descriptor) are inserted into the video elementary loop corresponding to each video stream.

  The transport stream TS generated by the multiplexer 104 is sent to the transmitting unit 105. In the transmitting unit 105, this transport stream TS is modulated by a modulation scheme suitable for broadcasting such as QPSK / OFDM, for example, and an RF modulation signal is transmitted from the transmission antenna.

"Configuration of Receiver"
FIG. 15 shows a configuration example of the receiving device 200. The receiving device 200 includes a central processing unit (CPU) 201, a receiving unit 202, a demultiplexer 203, and a compressed data buffer (cpb: coded picture buffer) 204. The receiving apparatus 200 further includes a decoder 205, a non-compressed data buffer (dpb: decoded picture buffer) 206, and a post processing unit 207. The CPU 201 configures a control unit and controls the operation of each unit of the receiving device 200.

  The receiving unit 202 demodulates the RF modulated signal received by the receiving antenna, and acquires the transport stream TS. The demultiplexer 203 selectively extracts, from the transport stream TS, the coded image data of the picture of the hierarchical set according to the decoder temporal layer capability, and sends it to the compressed data buffer (cpb: coded picture buffer) 204. .

  FIG. 16 shows a configuration example of the demultiplexer 203. The demultiplexer 203 includes a TS adaptation field extraction unit 231, a clock information extraction unit 232, a TS payload extraction unit 233, a section extraction unit 234, a PSI table / descriptor extraction unit 235, and a PES packet extraction unit 236. ing. Further, the demultiplexer 203 includes a PES header extraction unit 237, a time stamp extraction unit 238, an identification information extraction unit 239, a PES payload extraction unit 240, and a stream configuration unit (stream composer) 241.

  The TS adaptation field extraction unit 231 extracts the adaptation field from the TS packet having the adaptation field of the transport stream TS. The clock information extraction unit 232 extracts the PCR from the adaptation field including the PCR (Program Clock Reference), and sends the PCR to the CPU 201.

  The TS payload extraction unit 233 extracts the TS payload from the TS packet having the TS payload of the transport stream TS. The section extraction unit 234 extracts the section data from the TS payload including the section data. The PSI table / descriptor extraction unit 235 analyzes the section data extracted by the section extraction unit 234, and extracts a PSI table and a descriptor. Then, the PSI table / descriptor extraction unit 235 sends the minimum value (min) and the maximum value (max) of temporal_id to the CPU 201 and to the stream configuration unit 241.

  The PES packet extraction unit 236 extracts the PES packet from the TS payload including the PES packet. The PES header extraction unit 237 extracts a PES header from the PES packet extracted by the PES packet extraction unit 236. The time stamp extraction unit 238 extracts the time stamp (DTS, PTS) inserted in the PES header for each picture, sends it to the CPU 201, and sends it to the stream construction unit 241.

  The identification information extraction unit 239 extracts, for each picture, identification information identifying a layer set to which the picture belongs, which is inserted in the PES header, and sends the identification information to the stream configuration unit 241. For example, when a plurality of layers are divided into a lower layer set and a higher layer set, priority information of a 1-bit field of “PES_priority” of the PES header is extracted and sent to the stream configuration unit 241. This identification information is always inserted on the transmitting side when a single video stream is included in the transport stream TS, but on the transmitting side when a plurality of video streams are included in the transport stream TS. May not be inserted.

  The PES payload extraction unit 240 extracts a PES payload, that is, encoded image data of a picture in each layer, from the PES packet extracted by the PES packet extraction unit 236. The stream configuration unit 241 selectively extracts encoded image data of pictures of a hierarchical set according to the decoder temporal layer capability from encoded image data of pictures of each layer extracted by the PES payload extraction unit 240. , Compressed data buffer (cpb: coded picture buffer) 204. In this case, the stream configuration unit 241 refers to hierarchy information, stream configuration information, identification information (priority information) extracted by the identification information extraction unit 239, and the like obtained by the PSI table / descriptor extraction unit 235.

  For example, it is assumed that the frame rate of a video stream (coded stream) included in the transport stream TS is 120 fps. For example, it is assumed that a plurality of layers are divided into a layer set on the lower layer side and a layer set on the higher layer side, and the frame rate of the picture of each layer set is 60 fps. For example, in the hierarchical coding example shown in FIG. 3 described above, the hierarchies 0 to 3 are set as a hierarchy set on the lower hierarchy side, and a decoder corresponding to 60 fps level_idc can be decoded. Also, layer 4 is a layer set on the higher layer side, and a decoder compatible with 120 fps level_idc can be decoded.

  In this case, the transport stream TS includes a single video stream (coded stream) having coded data of a picture of each layer, or coded image data of a picture of a lower layer side hierarchical set Two video streams (encoded stream) of the base stream (B_str) and the enhancement stream (E_str) having encoded image data of the picture of the hierarchical set on the higher layer side are included.

  When the decoding capability corresponds to 120 fps, the stream configuration unit 241 extracts encoded image data of pictures of all layers and sends the encoded image data to the compressed data buffer (cpb) 204. On the other hand, when the decoding capability corresponds to 60 fps although the decoding capability does not correspond to 120 fps, the stream configuration unit 241 extracts only the encoded image data of the picture of the lower layer side hierarchical set, and the compressed data buffer (cpb) Send to 204.

  FIG. 17 illustrates an example of picture (slice) selection by the stream configuration unit 241 when the transport stream TS includes a single video stream (coded stream). Here, “High” indicates a picture of a hierarchy set on the high hierarchy side, and “Low” indicates a picture of the hierarchy set on the low hierarchy side. Also, “P” indicates “PES_priority”.

  When the decoding capability corresponds to 120 fps, the stream configuration unit 241 extracts the encoded image data of the pictures of all the layers and sends it to the compressed data buffer (cpb) 204. On the other hand, when the decoding capability does not correspond to 120 fps but corresponds to 60 fps, the stream configuration unit 241 performs filtering based on “PES_priority”, and the picture of the lower layer side hierarchy set is P = 1. It takes only and sends it to the compressed data buffer (cpb) 204.

  FIG. 18 illustrates an example of picture (slice) selection of the stream configuration unit 241 when the transport stream TS includes two video streams (coded streams) of a base stream and an extension stream. Here, “High” indicates a picture of a hierarchy set on the high hierarchy side, and “Low” indicates a picture of the hierarchy set on the low hierarchy side. Also, it is assumed that the packet identifier (PID) of the base stream is PID A, and the packet identifier (PID) of the extension stream is PID B.

  When the decoding capability corresponds to 120 fps, the stream configuration unit 241 extracts the encoded image data of the pictures of all the layers and sends it to the compressed data buffer (cpb) 204. In this case, the stream configuration unit 241 sends the encoded image data of each picture to the compressed data buffer (cpb) 204 as one stream based on the decode timing information.

  In that case, look at the value of DTS as the decode timing, and combine the streams into one so that it monotonically increases between pictures. This picture grouping process itself is performed on a plurality of streams read from the plurality of compressed data buffers (cpb) 204, in which a plurality of compressed data buffers (cpb) 204 for the stream exist, as one stream. Decoding may be performed.

  On the other hand, when the decoding capability does not correspond to 120 fps but corresponds to 60 fps, the stream configuration unit 241 performs filtering based on a packet identifier (PID) and sets the layer set on the lower layer side, which is PID A. Only the picture is taken and sent to the compressed data buffer (cpb) 204.

  The stream configuration unit 241 has a function of rewriting the decode time stamp of the encoded image data of each picture selectively sent to the compressed data buffer (cpb) 204 and adjusting the decode interval of the low hierarchy picture. As a result, even if the decoding capability of the decoder 205 is low, reasonable decoding processing is possible.

  FIG. 19 shows an example of the layer coding example shown in FIG. 3 in which the layer set on the low layer side and the layer set on the high layer side are divided into two, and the picture that belongs to the low layer set The encoded image data is selectively taken out and sent to the compressed data buffer (cpb) 204.

  FIG. 19A shows decode timing before adjustment of the decode interval. In this case, the decoding interval between pictures varies, and the shortest decoding interval is equal to the decoding interval of 120 fps full resolution. On the other hand, FIG. 19B shows the decode timing after adjusting the decode interval. In this case, the decoding interval between pictures is equalized, and the decoding interval is 1/2 of the full resolution decoding interval. Thus, the decoding interval is adjusted according to the capability of the target decoder in each layer.

  FIG. 20 shows an example of the processing flow of the demultiplexer 203. This processing flow shows the case where the transport stream TS includes a single video stream (coded stream).

  The demultiplexer 203 starts processing in step ST31, and then proceeds to processing of step ST32. In this step ST32, the decoding capability (Decoder temporal layer capability) is set from the CPU 201. Next, in step ST33, the demultiplexer 203 determines whether or not there is an ability to decode all the layers (layers).

  When it is possible to decode the entire hierarchy, the demultiplexer 203 demultiplexes all TS packets passing through the corresponding PID filter in step ST34, and performs section parsing. Thereafter, the demultiplexer 203 proceeds to the process of step ST35.

  When there is no ability to decode all the layers in step ST33, the demultiplexer 203 demultiplexes the TS packet whose "PES_priority" is "1" in step ST36, and performs section parsing. Thereafter, the demultiplexer 203 proceeds to the process of step ST35.

  In step ST35, the demultiplexer 203 reads the HEVC descriptor (HEVC_descriptor) and the scalability extension descriptor (scalability_extension_descriptor) in the section of the target PID, and the presence or absence of an extension stream, the scalable type, the number and the ID of streams, The maximum and minimum values of temporal_id, and the decoder corresponding Level of each layer are obtained.

  Next, in step ST37, the demultiplexer 203 transfers the encoded stream to be subjected to the PID to the compressed data buffer (cpb) 204, and notifies the CPU 201 of DTS and PTS. After the processing of step ST37, the demultiplexer 203 ends the processing in step ST38.

  FIG. 21 illustrates an example of the processing flow of the demultiplexer 203. This processing flow shows the case where the transport stream TS includes two video streams (encoded stream) of a base stream and an extension stream.

  The demultiplexer 203 starts the process in step ST41, and then proceeds to the process of step ST42. In this step ST42, the decoding capability (Decoder temporal layer capability) is set from the CPU 201. Next, in step ST43, the demultiplexer 203 determines whether or not there is an ability to decode all the layers (layers).

  When it is possible to decode the entire hierarchy, in step ST44, the demultiplexer 203 demultiplexes a plurality of streams constituting the entire hierarchy using the PID filter, and performs section parsing. Thereafter, the demultiplexer 203 proceeds to the process of step ST45.

  If it is determined in step ST43 that there is no ability to decode all the layers, the demultiplexer 203 demultiplexes the stream of PID = PID A in step ST46, and performs section parsing. Thereafter, the demultiplexer 203 proceeds to the process of step ST45.

  In step ST45, the demultiplexer 203 reads the HEVC descriptor (HEVC_descriptor) and the scalability extension descriptor (scalability_extension_descriptor) in the section of the target PID, and the presence or absence of an extension stream, the scalable type, the number and the ID of streams, The maximum and minimum values of temporal_id, and the decoder corresponding Level of each layer are obtained.

  Next, in step ST47, the demultiplexer 203 combines encoded streams to be subjected to PID into one stream based on DTS (if there is no PTS) information, and transfers the stream to the compressed data buffer (cpb) 204. , DTS, and PTS to the CPU 201. After the processing of step ST47, the demultiplexer 203 ends the processing in step ST48.

  Referring back to FIG. 15, the compressed data buffer (cpb) 204 temporarily accumulates the video stream (coded stream) extracted by the demultiplexer 203. The decoder 205 extracts, from the video stream stored in the compressed data buffer 204, the encoded image data of the picture of the layer designated as the layer to be decoded. Then, the decoder 205 decodes each of the extracted encoded image data of each picture at the decoding timing of the picture and sends the decoded image data to the non-compressed data buffer (dpb) 206.

  Here, in the decoder 205, a hierarchy to be decoded from the CPU 201 is designated by temporal_id. This designated hierarchy is the entire hierarchy included in the video stream (encoded stream) taken out by the demultiplexer 203 or a partial hierarchy on the lower hierarchy side, and is set automatically by the CPU 201 or in response to a user operation. Ru. Further, the decoder 205 is provided with decode timing from the CPU 201 based on the DTS (Decoding Time stamp). The decoder 205 reads out and uses the image data of the reference picture from the non-compression data buffer 206 as necessary when decoding the encoded image data of each picture.

  FIG. 22 shows a configuration example of the decoder 205. The decoder 205 includes a temporal ID analysis unit 251, a target hierarchy selection unit 252, and a decoding unit 253. The temporal ID analysis unit 251 reads the video stream (coded stream) stored in the compressed data buffer 204, and analyzes temporal_id inserted in the NAL unit header of the coded image data of each picture.

  The target hierarchy selection unit 252 extracts, from the video stream read from the compressed data buffer 204, based on the analysis result of the temporal ID analysis unit 251, the encoded image data of the picture of the hierarchy specified as the hierarchy to be decoded. . The decoding unit 253 sequentially decodes the encoded image data of each picture extracted by the target hierarchy selecting unit 252 at the decoding timing and sends the decoded image data to the non-compressed data buffer (dpb) 206.

  In this case, the decoding unit 253 analyzes the VPS and the SPS to grasp, for example, the level designation value “sublayer_level_idc” of the bit rate for each sub layer, and confirms whether or not decoding is possible within the decoding capability. Further, in this case, the decoding unit 253 analyzes the SEI to grasp, for example, “initial_cpb_removal_time” and “cpb_removal_delay”, and confirms whether the decoding timing from the CPU 201 is appropriate.

  When decoding a slice (Slice), the decoding unit 253 acquires “ref_idx_10_active (ref_idx_I1_active) as information indicating a prediction destination in the time direction from the slice header (Slice header), and performs prediction in the time direction. Note that the picture after decoding is processed as a reference by another picture, with “short_term_ref_pic_set_idx” or “it_idx_sps” obtained from a slice header as an index.

  Referring back to FIG. 15, the uncompressed data buffer (dpb) 206 temporarily accumulates the image data of each picture decoded by the decoder 205. The post processing unit 207 performs processing to match the frame rate with the display capability for the image data of each picture sequentially read from the non-compressed data buffer (dpb) 206 at the display timing. In this case, display timing is given from the CPU 201 based on PTS (Presentation Time stamp).

  For example, when the frame rate of image data of each picture after decoding is 120 fps and the display capability is 120 fps, the post processing unit 207 sends the image data of each picture after decoding to the display as it is. Further, for example, when the frame rate of the image data of each picture after decoding is 120 fps and the display capability is 60 fps, the post processing unit 207 determines the time direction resolution of the image data of each picture after decoding. Subsample processing is performed so as to be 1/2 times and sent to the display as image data of 60 fps.

  Also, for example, when the frame rate of the image data of each picture after decoding is 60 fps and the display capability is 120 fps, the post processing unit 207 determines the time direction resolution of the image data of each picture after decoding. Interpolation processing is performed so as to be 2 × and sent to the display as image data of 120 fps. Also, for example, when the frame rate of the image data of each picture after decoding is 60 fps and the display capability is 60 fps, the post processing unit 207 sends the image data of each picture after decoding to the display as it is.

  FIG. 23 shows a configuration example of the post processing unit 207. In this example, as described above, the frame rate of the image data of each picture after decoding is 120 fps or 60 fps and the display capability is 120 fps or 60 fps.

  The post processing unit 207 includes an interpolation unit 271, a subsampling unit 272, and a switch unit 273. The image data of each picture after decoding from the non-compressed data buffer 206 is directly input to the switch unit 273, or after being doubled in frame rate by the interpolation unit 271, is input to the switch unit 273, or a subsample unit The frame rate is halved in 272 and then input to the switch unit 273.

  Selection information is supplied to the switch unit 273 from the CPU 201. The selection information is generated automatically by the CPU 201 with reference to the display capability or in response to a user operation. The switch unit 273 selectively outputs one of the inputs based on the selection information. As a result, the frame rate of the image data of each picture sequentially read out from the non-compressed data buffer (dpb) 206 at the display timing is made to match the display capability.

  FIG. 24 illustrates an example of the processing flow of the decoder 205 and the post processing unit 207. The decoder 205 and the post processing unit 207 start the process in step ST51, and then move to the process in step ST52. In step ST52, the decoder 205 reads the video stream to be decoded stored in the compressed data buffer (cpb) 204, and selects a picture of a hierarchy designated as a decoding target by the CPU 201 based on temporal_id.

  Next, in step ST53, the decoder 205 sequentially decodes the encoded image data of each selected picture at decode timing, and transfers the image data of each picture after decoding to the non-compression data buffer (dpb) 206. , Temporarily accumulate. Next, in step ST54, the post processing unit 207 reads out the image data of each picture from the non-compressed data buffer (dpb) 206 at display timing.

  Next, the post processing unit 207 determines whether the frame rate of the read image data of each picture is in the display capability. If the frame rate does not match the display capability, the post processing unit 207 sends the frame rate to the display according to the display capability in step ST56, and then ends the process in step ST57. On the other hand, when the frame rate matches the display capability, the post processing unit 207 sends the frame rate as it is to the display in step ST58, and then ends the process in step ST57.

  The operation of the receiving device 200 shown in FIG. 15 will be briefly described. The receiving unit 202 demodulates the RF modulated signal received by the receiving antenna to obtain the transport stream TS. The transport stream TS is sent to the demultiplexer 203. The demultiplexer 203 selectively extracts encoded image data of a picture of a hierarchical set according to the decoder temporal layer capability from the transport stream TS, and sends it to the compressed data buffer (cpb) 204 for temporary use. Accumulated.

  The decoder 205 extracts, from the video stream stored in the compressed data buffer 204, the coded image data of the picture of the layer designated as the layer to be decoded. Then, in the decoder 205, the encoded image data of each picture taken out is decoded at the decode timing of that picture, sent to the non-compressed data buffer (dpb) 206, and temporarily stored. In this case, when the encoded image data of each picture is decoded, the image data of the reference picture is read out from the non-compression data buffer 206 and used as necessary.

  Image data of each picture sequentially read out from the non-compressed data buffer (dpb) 206 at display timing is sent to the post processing unit 207. The post-processing unit 207 performs interpolation or sub-sampling on the image data of each picture to match the frame rate to the display capability. The image data of each picture processed by the post processing unit 207 is supplied to the display, and a moving image is displayed by the image data of each picture.

  As described above, in the transmitting and receiving system 10 shown in FIG. 1, on the transmitting side, in the layer of the video stream (the header of the PES packet), which layer set each of the coded image data of each picture included in the video stream Identification information is inserted to identify whether it is encoded image data of a picture belonging to. Therefore, for example, on the reception side, it is possible to selectively decode the coded image data of a picture of a hierarchy of a predetermined hierarchy or lower according to the decoding capability by using the identification information.

  Further, in the transmission and reception system 10 shown in FIG. 1, the scalability extension descriptor (scalability_extension_descriptor) and the like are inserted in the layer of the transport stream TS on the transmission side. Therefore, for example, on the reception side, hierarchical information in hierarchical coding, configuration information of a video stream included in the transport stream TS, and the like can be easily grasped, and appropriate decoding processing can be performed.

  Further, in the transmitting and receiving system 10 shown in FIG. 1, on the receiving side, encoded image data of a picture of a layer of a predetermined layer or less according to the decoding capability (Decoder temporal layer capability) is selectively compressed from the received video stream. The data is fetched into the data buffer 204 and decoded. Therefore, for example, appropriate decoding processing can be performed according to the decoding capability.

  Further, in the transmission / reception system 10 shown in FIG. 1, the function of adjusting the decoding interval of the low-layer picture by rewriting the decoding time stamp of the coded image data of each picture selectively taken into the compressed data buffer 204 on the receiving side. It is the one with Therefore, for example, even when the decoding capability of the decoder 205 is low, reasonable decoding processing is possible.

  Further, in the transmission and reception system 10 shown in FIG. 1, the frame rate of the image data of each picture after decoding is matched with the display capability by the post processing unit 207 on the reception side. Therefore, for example, even when the decoding capability is low, it is possible to obtain image data of a frame rate that is suitable for high display capability.

<2. Modified example>
In the above-described embodiment, identification information for identifying coded image data of a picture belonging to which hierarchical set among a predetermined number of hierarchical sets is used as the identification information of the coded image data of each picture included in the video stream, An example of inserting into the header (PES header) of the PES packet has been shown. However, the insertion position of this identification information is not limited to this.

  For example, the multiplexer 104 (see FIG. 2) may insert this identification information into the adaptation field of the TS packet having the adaptation field. The multiplexer 104 utilizes, for example, one bit field of a well-known elementary stream priority indicator (elementary_stream_priority_indicator) present in the adaptation field when dividing a plurality of layers into a lower layer set and a higher layer set.

  This 1-bit field is set to “1”, that is, the priority is high when the payload of the subsequent TS packet includes a PES packet having encoded image data of a picture of a layer set on the lower layer side as the payload. On the other hand, this 1-bit field is set to "0" when the payload of the subsequent TS packet includes a PES packet having encoded image data of a picture of a layer set on the lower layer side in the payload, that is, the priority is low. .

  FIG. 25 shows an arrangement example of the adaptation field. This example is a case where a plurality of layers are divided into a lower layer set and a higher layer set, and one bit field of an elementary stream priority indicator (elementary_stream_priority_indicator) is used.

  In the illustrated example, a TS packet having an adaptation field is arranged immediately before each of a predetermined number of TS packet groups having divided PES packets having one picture of encoded image data in the payload. In this case, when the one picture is a picture of the lower layer side hierarchy set, the 1 bit field of the elementary stream priority indicator is set to “1”. On the other hand, when the one picture is a picture of a hierarchy set on the higher hierarchy side, the 1-bit field of the elementary stream priority indicator is set to "0".

  As shown in FIG. 25, by arranging TS packets having an adaptation field, on the receiving side, it is encoded data of a picture belonging to any hierarchical set for each of encoded image data of a picture included in a video stream. Can be easily identified. In the arrangement example shown in FIG. 25, TS packets having an adaptation field are arranged for each picture, but every time the layer set to which a picture belongs is switched, a TS packet having an adaptation field is arranged immediately before that. You may be asked to

  FIG. 26 shows a configuration example of the multiplexer 104A of the transmission apparatus 100 in the case of inserting the identification information of the hierarchical set into the adaptation field as described above. In FIG. 26, parts corresponding to FIG. 12 are assigned the same reference numerals, and the detailed description thereof will be omitted. The multiplexer 104A has an adaptation field priority instructing unit 146 instead of the PES priority generating unit 141 in the multiplexer 104 of FIG.

  Information of the number of layers (Number of layers) and the number of streams (Number of streams) is supplied from the CPU 101 to the priority instruction unit 146. The priority instructing unit 146 generates priority information of each layer set in a case where a plurality of layers indicated by the number of layers are divided into a predetermined number of layer sets of 2 or more. For example, when divided into two, a value to be inserted into a 1-bit field of an elementary stream priority indicator (a low hierarchy set is “1”, a high hierarchy set is “0”) is generated.

  The priority information of each hierarchical set generated by the priority designation unit 146 is supplied to the transport packetization unit 145. The transport packetization unit 145 arranges a TS packet having an adaptation field immediately before each of a predetermined number of TS packet groups having a PES packet divided into a payload having one picture encoded image data as a payload. Then, in that case, the transport packetization unit 145 inserts, in the adaptation field, priority information corresponding to the layer set to which the picture belongs as identification information.

  FIG. 27 shows a configuration example of the transport stream TS in the case where the identification information of the hierarchy set is inserted into the adaptation field as described above. This configuration example is configured substantially the same as the configuration example shown in FIG. 14 described above. In this configuration example, there are TS packets having an adaptation field, and identification information for identifying a hierarchical set to which each picture belongs is inserted in this adaptation field. For example, when a plurality of layers are divided into a lower layer set and a higher layer set, a 1-bit field of an elementary stream priority indicator (elementary_stream_priority_indicator) is used.

  FIG. 28 shows a configuration example of the demultiplexer 203A of the reception apparatus 200 in the case of inserting the identification information of the hierarchy set into the adaptation field as described above. In FIG. 28, parts corresponding to FIG. 16 are assigned the same reference numerals, and the detailed description thereof will be omitted. The demultiplexer 203A has an identification information extraction unit 242 in place of the identification information extraction unit 239 in the demultiplexer 203 of FIG.

  The identification information extraction unit 242 extracts identification information from the adaptation field, and sends the identification information to the stream configuration unit 241. For example, when a plurality of layers are divided into a lower layer set and a higher layer set, priority information of a 1-bit field of “elementary_stream_priority_indicator” of the adaptation field is extracted and sent to the stream configuration unit 241.

  The stream configuration unit 241 selectively extracts encoded image data of pictures of a hierarchical set according to the decoder temporal layer capability from encoded image data of pictures of each layer extracted by the PES payload extraction unit 240. , Sent to the compressed data buffer (cpb) 204. In this case, the stream configuration unit 241 refers to hierarchy information, stream configuration information, identification information (priority information) extracted by the identification information extraction unit 242, etc. obtained by the PSI table / descriptor extraction unit 235.

  Moreover, although the transmission / reception system 10 which consists of the transmission apparatus 100 and the receiving apparatus 200 was shown in the above-mentioned embodiment, the structure of the transmission / reception system which can apply this technique is not limited to this. For example, the part of the receiving apparatus 200 may be, for example, a configuration of a set top box and a monitor connected by a digital interface such as (HDMI (High-Definition Multimedia Interface), etc.) Note that “HDMI” is registered It is a trademark.

  Moreover, in the above-mentioned embodiment, the container showed the example which is a transport stream (MPEG-2 TS). However, the present technology is similarly applicable to a system configured to be distributed to receiving terminals using a network such as the Internet. In Internet delivery, it is often delivered in containers of MP4 or other formats. That is, as the container, containers of various formats, such as transport stream (MPEG-2 TS) adopted in digital broadcasting standard and MP4 used in Internet distribution, correspond.

For example, FIG. 29 shows a configuration example of an MP4 stream. In this MP4 stream, "moov
There are boxes such as "moof" and "mdat". In the "mdat" box, there is a video elementary stream "track1: video ES1" which is a video encoded stream as a track, and an audio elementary stream "track1: audio ES1" which is an audio encoded stream. Exist

  Also, in the box of "moof", "mfhd (movie fragment header) exists as a header part, and" track fragment "corresponding to each track exists as a data part of the video elementary stream" track1: In “track1 fragment (video)” corresponding to “video ES1”, “Independent and disposable samples” exist, and in it, a box “SampleDependencyTypeBox” corresponding to each picture is inserted.

  In this box, identification information can be inserted which identifies encoded image data of a picture which belongs to each hierarchical set of encoded image data of each picture. For example, in the case of dividing a plurality of layers into two layer sets of the top layer and other lower layers, the 2-bit field of "sample_depends_on" and the 2-bit field of "sample_is_depended_on" are used to Insertion is possible.

  FIG. 30 shows a structural example (Syntax) of “SampleDependencyTypeBox”. FIG. 31 also shows the contents (Semantics) of the main information in the structural example. In this case, “sample_depends_on” is “1” to indicate that the reference is to another picture and not to an I picture, and “sample_is_dependent_on” is indicated to “2” to indicate that no reference is made to other pictures. It is possible to identify that the picture belongs to a set of Also, in any other state, it is possible to identify that the picture is a picture belonging to a hierarchical set of hierarchical layers.

  Note that instead of using the box of “SampleDependencyTypeBox”, it is also conceivable to use a box “SampleScalablePriorityBox” that is newly defined. FIG. 32 shows a structural example (Syntax) of “SampleScalablePriorityBox”. Also, FIG. 33 shows the contents (Semantics) of main information in the structural example.

  In this case, in the case of dividing a plurality of layers into two layer sets of the lowest layer set and the high layer set, the identification information is inserted using a 2-bit field of “base_and_priority”. That is, by setting “base_and_priority” to “1”, for example, it is possible to identify that the picture has low priority and belongs to the high hierarchy set. On the other hand, by setting “base_and_priority” to “2”, for example, it is possible to identify that the picture has high priority and belongs to the low hierarchy set.

Furthermore, the present technology can also be configured as follows.
(1) The image data of each picture constituting moving image data is classified into a plurality of layers, the image data of the classified picture of each layer is encoded, and the image data of the encoded picture of each layer is encoded An image encoding unit that generates video data, and
A transmitting unit for transmitting a container of a predetermined format including the generated video data;
The plurality of layers are divided into two or more predetermined number of layer sets, and the packet containing the video data is encoded with the picture to which the encoded image data of each picture included in the video data belongs. A transmission apparatus comprising: an identification information insertion unit that inserts identification information that identifies image data.
(2) The transmission apparatus according to (1), wherein the identification information is priority information which is set to be higher as the layer set on the lower layer side is higher.
(3) The transmission apparatus according to (1), wherein the identification information is inserted into a header of a PES packet including encoded image data for each picture in a payload.
(4) The transmission apparatus according to (3), wherein the identification information is inserted using a field of PES priority of the header.
(5) The transmission apparatus according to (1), wherein the identification information is inserted into an adaptation field of a TS packet having an adaptation field.
(6) The transmission apparatus according to (5), wherein the identification information is inserted using a field of the ES priority indicator of the adaptation field.
(7) The transmission apparatus according to (1), wherein the identification information is inserted into a box of a header associated with a track of a corresponding picture.
(8) The image coding unit
Generate a single video stream having encoded image data of the picture of each layer, or generate a predetermined number of video data having encoded image data of the picture of each layer set,
The transmission device according to any one of (1) to (7), further including a configuration information insertion unit that inserts configuration information of a video stream included in the container into the layer of the container.
(9) The image data of each picture constituting moving image data is classified into a plurality of layers, the image data of the classified picture of each layer is encoded, and the image data of the encoded picture of each layer is encoded An image encoding step of generating video data
A transmitting step of transmitting a container of a predetermined format including the generated video data by the transmitting unit;
The plurality of layers are divided into two or more predetermined number of layer sets, and the packet containing the video data is encoded with the picture to which the encoded image data of each picture included in the video data belongs. A transmission method comprising: an identification information insertion step of inserting identification information that identifies image data.
(10) A container of a predetermined format including video data having encoded image data of a picture of each layer obtained by classifying and encoding image data of each picture constituting moving image data into a plurality of layers A receiver for receiving
From the video data contained in the received container, encoded image data of a picture of a layer of a predetermined hierarchy or less according to the decoding capability is selectively taken into a buffer, and encoded image data of each picture taken into the buffer And an image decoding unit for obtaining image data of a picture of a hierarchy below the predetermined hierarchy.
(11) The plurality of layers are divided into two or more predetermined number of layer sets, and in the packet containing the video data, the picture including the encoded image data of each picture included in the video data belongs to which layer set Identification information identifying whether it is encoded image data of
The receiving device according to (10), wherein the image decoding unit fetches, in the buffer, encoded image data of a picture of a predetermined hierarchical set according to the decoding capability based on the identification information.
(12) The reception apparatus according to (11), wherein the identification information is inserted in a header of a PES packet including encoded image data for each picture in a payload.
(13) The reception apparatus according to (11), wherein the identification information is inserted in an adaptation field of a TS packet having an adaptation field.
(14) The transmission apparatus according to (11), wherein the identification information is inserted in a box of a header associated with a track of a corresponding picture.
(15) The plurality of layers are divided into two or more predetermined numbers of layer groups, and the predetermined number of video streams having encoded image data of pictures of the predetermined number of layer groups in the received container Is included,
The receiving device according to (10), wherein the image coding unit takes encoded image data of a picture of a predetermined hierarchical set according to the decoding capability into the buffer based on stream identification information.
(16) The image decoding unit
When the encoded image data of the picture of the predetermined hierarchical set is included in a plurality of video streams, the encoded image data of each picture is taken as one stream based on the decode timing information to the buffer (15) Receiving device according to.
(17) The image decoding unit
The receiving device according to any one of (10) to (16), having a function of rewriting the decoding time stamp of the encoded image data of each picture selectively taken into the buffer and adjusting the decoding interval of the low hierarchy picture. .
(18) The receiving device according to any one of (10) to (17), further including a post processing unit that matches a frame rate of image data of each picture obtained by the image decoding unit with a display capability.
(19) Includes video data having coded image data of a picture of each layer obtained by the image data of each picture constituting moving image data being classified into a plurality of layers and coded by the receiving unit Receiving a container of a predetermined format;
From the video data contained in the received container, encoded image data of a picture of a layer of a predetermined hierarchy or less according to the decoding capability is selectively taken into a buffer, and encoded image data of each picture taken into the buffer Receiving the image data of the picture of the hierarchy below the predetermined hierarchy by decoding H.

  A main feature of the present technology is identification information for identifying, in a packet set that contains video data, encoded image data of each picture included in the video data, which encoded image data belongs to which hierarchical set In the receiving side, it is easily possible to selectively decode the coded image data of the picture of the hierarchy below the predetermined hierarchy according to the decoding capability by inserting the (See FIG. 12).

10: Transmission / reception system 100: Transmission device 101: CPU
102: Encoder 103: Compressed data buffer (cpb)
104, 104A ... multiplexer 105 ... transmission section 141 ... PES priority generation section 142 ... section coding section 143-1-143-N ... PES packetization section 144 ... switch section 145 · · · Transport packetization unit 146 ··· Adaptation field · Priority instruction unit 200 ··· Receiving unit 201 ··· CPU
202: Reception unit 203: Demultiplexer 204: Compressed data buffer (cpb)
205: Decoder 206: Uncompressed data buffer (dpb)
207 ... post processing unit 231 ... TS adaptation field extraction unit 232 ... clock information extraction unit 233 ... TS payload extraction unit 234 ... section extraction unit 235 ... PSI table / descriptor extraction unit 236 ... PES packet extraction unit 237 ... PES header extraction unit 238 ... Time stamp extraction unit 239 ... Identification information extraction unit 240 ... PES payload extraction unit 241 ... Stream construction unit 242 ... Identification information extraction unit 251 ... Temporal ID analysis unit 252 ... Target hierarchy selection unit 253 ... Decoding unit 271 ... Interpolation unit 272 ... Subsample unit 273 ... Switch unit

Claims (5)

The image data of each picture making up the moving image data is hierarchically encoded, and the first video stream having the encoded image data of the lower layer side picture and the second video having the encoded image data of the higher layer side picture An image encoding unit that generates a stream;
The video stream includes the first video stream and the second video stream generated by the image coding unit, and is included in each video stream corresponding to the first video stream and the second video stream, respectively. Stream identification information indicating to which hierarchical set the encoded image data of each picture to be belong belongs, and a value corresponding to a frame rate consisting only of the picture on the lower layer side corresponding to the first video stream is inserted. Multiplex, including a first descriptor, and a second descriptor in which a value corresponding to a frame rate made up of the picture on the lower layer side and the picture on the higher layer side corresponding to the second video stream is inserted A transmitter comprising: a transmitter configured to transmit a coded stream. The image coding unit hierarchically codes the image data of each picture constituting the moving image data, and the first video stream having the coded image data of the picture on the lower layer side and the coded image data of the picture on the higher layer side An image coding step to generate a second video stream having
A transmitting unit includes the first video stream and the second video stream generated in the image coding step, and corresponds to the first video stream and the second video stream, respectively. It corresponds to a stream identification information indicating to which hierarchical set the encoded image data of each picture included in the video stream belongs, and a frame rate consisting of only the pictures on the lower layer side corresponding to the first video stream A second descriptor in which a value corresponding to a frame rate made up of a first descriptor into which a value is inserted, a picture on the lower layer side and a picture on the higher layer side corresponding to the second video stream is inserted A transmission method, comprising: a transmission step of transmitting a multiplexed stream including a descriptor. The first video stream having the encoded image data of the lower layer side picture and the encoded image data of the higher layer side picture generated by hierarchically encoding the image data of each picture constituting the moving image data Containing a second video stream, and corresponding to the first video stream and the second video stream, to which hierarchical set the encoded image data of each picture included in each video stream belongs. Corresponding to the first video stream, a first descriptor in which a value corresponding to a frame rate consisting only of the lower layer side picture corresponding to the first video stream is inserted, and the second video stream A value corresponding to a frame rate consisting of the lower layer picture and the higher layer picture. A receiver for receiving a multiplexed stream including the descriptors,
Only the first video stream or the first video stream from the multiplexed stream based on the stream identification information and the value corresponding to the frame rate inserted in the first descriptor and the second descriptor. A processing unit that extracts and decodes both the video stream of and the second video stream. The receiving apparatus according to claim 3, wherein the processing unit further performs a process of adjusting a frame rate of image data of each picture obtained by performing the decoding to a display capability. The receiver is a code of a first video stream having encoded image data of a picture on the lower layer side generated by hierarchically encoding image data of each picture constituting moving image data and a code of the picture on the upper layer side Layer set including encoded video data of each picture included in each video stream including the second video stream having the encoded image data and corresponding to the first video stream and the second video stream respectively And a first descriptor in which a value corresponding to a frame rate consisting of only the lower layer side picture corresponding to the first video stream is inserted, and the second video A value corresponding to a frame rate consisting of the picture on the lower layer side and the picture on the higher layer side corresponding to the stream is inserted A receiving step of receiving a multiplexed stream including a second descriptor that,
A processing unit generates only the first video stream from the multiplexed stream based on the stream identification information and the value corresponding to the frame rate inserted in the first descriptor and the second descriptor. Alternatively, the method further includes a processing step of extracting and decoding both the first video stream and the second video stream.

Images