JP6648811B2 - Transmitting device, transmitting method, receiving device and receiving method - Google Patents

Description

  The present technology relates to a transmission device, a transmission method, a reception device, and a reception method. More specifically, the present technology relates to a transmission device or the like that hierarchically encodes and transmits image data of each picture constituting moving image data.

  When a compressed moving image is provided through broadcasting, a network, or the like, the upper limit of a reproducible frame frequency is limited by the decoding capability of the receiver. Therefore, it is necessary for the service side to restrict the service to only low frame frequencies or to simultaneously provide services of a plurality of high and low frame frequencies in consideration of the reproduction capability of a widespread receiver.

  Receivers are costly to support services at high frame frequencies, and hinder early adoption. Initially, only low-cost receivers dedicated to low-frame-frequency services are widespread, and if the service side starts high-frame-frequency services in the future, it will be impossible to view at all without a new receiver. It becomes a hindrance factor.

  For example, in HEVC (High Efficiency Video Coding), temporal scalability has been proposed by hierarchically encoding image data of each picture constituting moving image data (see Non-Patent Document 1). On the receiving side, the layer of each picture can be identified based on the temporal ID (temporal_id) inserted in the header of the NAL (Network Abstraction Layer) unit, and selective decoding up to the layer corresponding to the decoding capability becomes possible. .

Gary J. Sullivan, Jens-Rainer Ohm, Woo-Jin Han, Thomas Wiegand, “Overview of the High Efficiency Video Coding (HEVC) Standard” IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECNOROGY, VOL. 22, NO. 12, pp . 1649-1668, DECEMBER 2012

  An object of the present technology is to enable good decoding processing according to decoding capability on a receiving side.

The concept of this technology is
Video data that classifies the image data of each picture constituting the moving image data into a plurality of layers, encodes the image data of the classified pictures of each layer, and has the image data of the encoded pictures of each layer. An image encoding unit that generates
A transmitting unit that transmits a container of a predetermined format including the generated video data,
The plurality of hierarchies are divided into a predetermined number of hierarchies of two or more, and in a packet for container of the video data, the coded image data of each picture included in the video data is coded for each of the hierarchies. A transmission device including an identification information insertion unit for inserting identification information for identifying whether the data is image data.

  In the present technology, the image encoding unit encodes image data of each picture constituting moving image data to generate video data. In this case, the image data of each picture constituting the moving image data is classified and encoded into a plurality of layers, and video data having encoded image data of the pictures of each layer is generated.

  The transmitting unit transmits a container of a predetermined format including the above-described video data. For example, the container may be a transport stream (MPEG-2 TS) adopted in a digital broadcasting standard. Further, for example, the container may be an MP4 used for distribution on the Internet or the like, or a container of another format.

  The identification information insertion unit divides the plurality of hierarchies into a predetermined number of hierarchies of two or more, and in a packet for container of video data, the coded image data of each picture included in the video data belongs to any hierarchies. Identification information for identifying whether the data is coded image data of a picture is inserted. For example, the identification information may be priority information that is set higher for a lower layer group.

  For example, the identification information may be inserted into a header of a PES packet whose payload includes encoded image data for each picture. In this case, for example, the identification information may be inserted using the PES priority field of the header. Further, for example, the identification information may be inserted into the adaptation field of the TS packet having the adaptation field. In this case, for example, the identification information may be inserted by using the field of the ES priority indicator of the adaptation field. Also, for example, the identification information may be inserted into a box of a header related to a track of a corresponding picture.

  As described above, in the present technology, the packet that contains the video data includes, in the packet that contains the video data, the identification information that identifies the coded image data of the picture belonging to which hierarchical group. Is inserted. Therefore, on the receiving side, by using the identification information, it becomes easy to selectively decode the coded image data of the picture of the hierarchy of the predetermined hierarchy or less according to the decoding capability.

  In the present technology, for example, the image encoding unit generates a single video stream having encoded image data of pictures of each layer, or divides a plurality of layers into two or more predetermined number of layer sets. A configuration information insertion unit configured to generate a predetermined number of video streams each having encoded image data of a picture of each hierarchical set, and to insert configuration information of a video stream included in the container into a layer of the container, It may be done as follows. In this case, for example, the receiving side can easily grasp the configuration of the video stream based on the configuration information of the video stream included in the container.

Another concept of the present technology is
Receives a container of a predetermined format including video data having coded image data of a picture of each layer obtained by classifying and coding the image data of each picture constituting the moving image data into a plurality of layers. A receiving unit,
From the video stream included in the received container, coded image data of a picture of a layer lower than a predetermined layer corresponding to the decoding capability is selectively captured into a buffer, and the coded image data of each picture captured by the buffer is captured. And a receiving device including an image decoding unit that obtains image data of a picture of a layer lower than the predetermined layer.

  In the present technology, the receiving unit receives a container of a predetermined format. This container includes video data having image data of pictures of each layer obtained by classifying and encoding image data of each picture constituting moving image data into a plurality of layers.

  The image decoding unit selectively fetches coded image data of a picture of a layer lower than a predetermined layer according to the decoding capability from the video data included in the received container into a buffer, and stores each picture fetched into the buffer. Is decoded to obtain image data of a picture of a layer lower than a predetermined layer.

  For example, a plurality of hierarchies are divided into a predetermined number of hierarchic sets of two or more, and a packet that contains video data includes, Identification information for identifying whether the image data is image data is inserted, and based on the identification information, the image decoding unit fetches coded image data of a picture of a predetermined hierarchical set according to the decoding capability into a buffer and decodes the coded image data. And so on.

  In this case, for example, the identification information may be inserted into the header of a PES packet that includes encoded image data for each picture in the payload. In this case, for example, the identification information may be inserted into the adaptation field of the TS packet having the adaptation field. In this case, for example, the identification information may be inserted in a box of a header related to the track of the corresponding picture.

  Also, for example, the plurality of layers are divided into two or more predetermined number of layer sets, and the received container includes a predetermined number of video streams each having encoded image data of pictures of the predetermined number of layer sets. In this case, the image encoding unit may be configured to fetch encoded image data of a picture of a predetermined hierarchical group according to the decoding capability into a buffer and decode the encoded image data based on the stream identification information. At this time, for example, when the coded image data of the picture of the predetermined hierarchical group is included in a plurality of video streams, the image decoding unit may convert the coded image data of each picture into one stream based on the decode timing information. In the buffer.

  As described above, in the present technology, the coded image data of the picture of the hierarchy lower than the predetermined hierarchy corresponding to the decoding capability is selectively taken into the buffer and decoded from the received video data. Therefore, appropriate decoding processing according to the decoding capability can be performed.

  Note that, in the present technology, for example, the image decoding unit has a function of adjusting a decoding interval of a low-layer picture by rewriting a decode time stamp of encoded image data of each picture selectively taken into a buffer. May be done. In this case, even a decoder having a low decoding ability can perform reasonable decoding processing.

  In addition, the present technology may further include, for example, a post-processing unit that adjusts a frame rate of image data of each picture obtained by an image decoding unit to display capability. In this case, even if the decoding capability is low, it is possible to obtain image data at a frame rate that is compatible with the high display capability.

  According to the present technology, it is possible to perform a good decoding process according to the decoding capability on the receiving side. Note that the effects described here are not necessarily limited, and may be any of the effects described in the present disclosure.

FIG. 1 is a block diagram illustrating a configuration example of a transmission / reception system according to an embodiment. FIG. 3 is a block diagram illustrating a configuration example of a transmission device. FIG. 3 is a diagram illustrating an example of hierarchical encoding performed by an encoder. It is a figure which shows the structural example (Syntax) of a NAL unit header, and the content (Semantics) of the main parameter in the structural example. FIG. 3 is a diagram for describing a configuration of encoded image data of each picture according to HEVC. FIG. 9 is a diagram illustrating an example of encoding, decoding, display order, and delay in hierarchical encoding. It is a figure which shows the encoding stream of hierarchical encoding, and the display expectation (display order) in a designated hierarchy. It is a figure showing the example of structure (Syntax) of HEVC descriptor (HEVC_descriptor). It is a figure which shows the content (Semantics) of the main information in the structural example of a HEVC descriptor. It is a figure which shows the example of a structure (Syntax) of a scalability extension descriptor (scalability_extension_descriptor). It is a figure which shows the content (Semantics) of the main information in the example of a structure of a scalability extension descriptor. FIG. 3 is a block diagram illustrating a configuration example of a multiplexer. FIG. 7 is a diagram illustrating an example of a processing flow of a multiplexer. [Fig. 3] Fig. 3 is a diagram illustrating a configuration example of a transport stream TS when distribution is performed by a single stream. It is a block diagram which shows the example of a structure of a receiver. FIG. 3 is a block diagram illustrating a configuration example of a demultiplexer. FIG. 3 is a diagram illustrating a case where a single video stream (encoded stream) is included in a transport stream TS. FIG. 3 is a diagram illustrating a case where a transport stream TS includes two video streams (encoded streams) of a base stream and an extension stream. FIG. 7 is a diagram for explaining a function of adjusting a decoding interval of a lower hierarchical picture by rewriting a decoding time stamp of encoded image data of each picture. It is a figure showing an example of a processing flow (one frame) of a demultiplexer. It is a figure showing an example of a processing flow (two frames) of a demultiplexer. FIG. 3 is a block diagram illustrating a configuration example of a decoder. FIG. 3 is a diagram illustrating a configuration example of a post-processing unit. FIG. 9 is a diagram illustrating an example of a processing flow of a decoder and a post-processing unit. FIG. 4 is a diagram illustrating an example of the arrangement of adaptation fields. FIG. 10 is a block diagram illustrating a configuration example of a multiplexer in a case where identification information of a hierarchical set is inserted into an adaptation field. It is a figure which shows the example of a structure of the transport stream TS at the time of inserting the identification information of a hierarchy set into an adaptation field. FIG. 9 is a block diagram illustrating a configuration example of a demultiplexer in a case where identification information of a hierarchical set is inserted into an adaptation field. FIG. 3 is a diagram illustrating a configuration example of an MP4 stream. It is a figure showing the example of structure of “SampleDependencyTypeBox”. It is a figure which shows the content of the main information in the structural example of "SampleDependencyTypeBox". It is a figure showing the example of structure of “SampleScalablePriorityBox”. It is a figure which shows the content of the main information in the structural example of "SampleScalablePriorityBox".

Hereinafter, embodiments for carrying out the invention (hereinafter, referred to as “embodiments”) will be described. The description will be made in the following order.
1. Embodiment 2. Modified example

<1. Embodiment>
[Transmission and reception system]
FIG. 1 shows a configuration example of a transmission / reception system 10 as an embodiment. The transmission / reception system 10 has a configuration including a transmission device 100 and a reception device 200.

  The transmitting device 100 transmits a transport stream TS as a container on a broadcast wave. The transport stream TS includes a video stream in which the image data of each picture constituting the moving image data is classified into a plurality of layers, and has encoded data of the image data of the picture in each layer. In this case, for example, Encoding such as H.264 / AVC and HEVC is performed, and the referenced picture is encoded so as to belong to the own layer and / or a layer lower than the own layer.

  Layer identification information for identifying the layer to which the layer belongs is added to the coded image data of the picture of each layer for each picture. In this embodiment, hierarchical identification information (“nuh_temporal_id_plus1” meaning temporal_id) is arranged in a header portion of a NAL unit (nal_unit) of each picture. By adding the layer identification information in this manner, the reception side can identify the layer of each picture in the layer of the NAL unit, and selectively extracts encoded image data of a layer lower than a predetermined layer to perform decoding processing. It can be carried out.

  In this embodiment, the plurality of hierarchies are divided into a predetermined number of hierarchies of two or more, and the coded image data of each picture in Identification information for identifying whether the data is coded image data is inserted.

  In this embodiment, this identification information is set as priority information that is set higher for a lower layer group, and is inserted into the header of a PES packet that includes encoded image data for each picture in the payload. With this identification information, it becomes possible for the receiving side to take in only the coded image data of the picture of the hierarchical group according to its own decoding capability into the buffer and process it.

  The transport stream TS includes a single video stream having coded image data of pictures of each layer or a predetermined number of video streams each having coded image data of pictures of each layer set. In this transport stream TS, layer information of layer coding and configuration information of a video stream are inserted. With this information, the receiving side can easily grasp the hierarchical configuration and the stream configuration, and perform appropriate decoding processing.

  The receiving device 200 receives the above-described transport stream TS transmitted from the transmitting device 100 on a broadcast wave. The receiving device 200 selectively fetches coded image data of a picture of a layer lower than a predetermined layer selected according to the decoding capability from the video stream included in the transport stream TS into a buffer and decodes the coded image data. The image data is acquired and the image is reproduced.

  For example, as described above, the transport stream TS may include a single video stream having coded image data of pictures of a plurality of layers. In this case, based on the identification information described above, the coded image data of the picture of the predetermined hierarchical group corresponding to the decoding capability is fetched into the buffer and processed.

  Also, for example, as described above, the transport stream TS includes a predetermined number of video streams each having encoded image data of a picture of a predetermined number of two or more layers obtained by dividing a plurality of layers. May have been In this case, based on the stream identification information, the coded image data of the picture of the predetermined hierarchical group according to the decoding capability is fetched into the buffer and processed.

  In addition, the receiving apparatus 200 performs a process of adjusting the decoding interval of the lower hierarchical picture by rewriting the decode time stamp of the encoded image data of each picture selectively taken into the buffer. With this adjustment process, a reasonable decoding process can be performed even with a decoder having a low decoding capability.

  Further, the receiving apparatus 200 performs post processing for adjusting the frame rate of the image data of each picture obtained by decoding as described above to the display capability. By this post-processing, for example, even when the decoding capability is low, it is possible to obtain image data at a frame rate that is compatible with high display capability.

"Configuration of Transmitter"
FIG. 2 shows a configuration example of the transmission device 100. The transmission device 100 includes a CPU (Central Processing Unit) 101, an encoder 102, a compressed data buffer (cpb: coded picture buffer) 103, a multiplexer 104, and a transmission unit 105. The CPU 101 is a control unit, and controls the operation of each unit of the transmission device 100.

  The encoder 102 inputs uncompressed moving image data and performs hierarchical coding. The encoder 102 classifies the image data of each picture constituting the moving image data into a plurality of layers. Then, the encoder 102 encodes the classified image data of the picture of each layer, and generates a video stream having the encoded image data of the picture of each layer. The encoder 102 is, for example, an H.264 encoder. Encoding such as H.264 / AVC and HEVC is performed. At this time, the encoder 102 performs encoding such that the picture to be referred to (the referenced picture) belongs to the own layer and / or a layer lower than the own layer.

  FIG. 3 shows an example of hierarchical coding performed by the encoder 102. This example is an example in which image data of pictures of each layer is classified into five layers from 0 to 4, and for example, HEVC encoding is performed.

  The vertical axis indicates the hierarchy. 0 to 4 are set as temporal_id (hierarchy identification information) arranged in the header portion of the NAL unit (nal_unit) constituting the coded image data of the pictures of layers 0 to 4, respectively. On the other hand, the horizontal axis indicates the display order (POC: picture order of composition), with the display time on the left being earlier and the display time on the right being later.

  FIG. 4A shows a structural example (Syntax) of the NAL unit header, and FIG. 4B shows contents (Semantics) of main parameters in the structural example. In the 1-bit field of “Forbidden_zero_bit”, 0 is essential. A 6-bit field of “Nal_unit_type” indicates a NAL unit type. It is assumed that the 6-bit field of “Nuh_layer_id” is 0. The 3-bit field of “Nuh_temporal_id_plus1” indicates the temporal_id, and takes a value (1 to 7) obtained by adding “1”.

  Returning to FIG. 3, each of the rectangular frames indicates a picture, and the numbers indicate the order of encoded pictures, that is, the encoding order (decoding order on the receiving side). A sub picture group (Sub group of pictures) is composed of 16 pictures from "1" to "17" (excluding "2"), and "1" is the first picture of the sub picture group. . “2” is the first picture of the next sub-picture group. Alternatively, a sub-picture group is composed of 16 pictures from "2" to "17" except for "1", and "2" is the first picture of the sub-picture group.

  The picture “1” can be the first picture of a GOP (Group Of Pictures). As shown in FIG. 5, the coded image data of the first picture of the GOP is composed of NAL units of AUD, VPS, SPS, PPS, PSEI, SLICE, SSEI, and EOS. On the other hand, pictures other than the first picture of the GOP are constituted by NAL units of AUD, PPS, PSEI, SLICE, SSEI, and EOS. The VPS can be transmitted once in a sequence (GOP) together with the SPS, and the PPS can be transmitted in My Picture.

  Returning to FIG. 3, the solid arrow indicates the reference relationship of pictures in encoding. For example, the picture “1” is an I picture and does not refer to other pictures. The picture “2” is a P picture and is encoded with reference to the picture “1”. The picture “3” is a B picture, and is encoded with reference to the pictures “1” and “3”. Hereinafter, similarly, other pictures are coded with reference to a nearby picture in display order. It should be noted that the picture of layer 4 has no reference from other pictures.

  The encoder 102 generates a single video stream (single stream) having coded image data of pictures of each layer, or divides a plurality of layers into two or more predetermined number of layer sets, and A predetermined number of video streams (multi-streams) each having the coded image data of the picture are generated. For example, in the example of the layer coding in FIG. 3, when the layers 0 to 3 are divided into two layer sets with a layer set of a lower layer and a layer 4 as a layer set of a higher layer, the encoder 102 , Two video streams (coded streams) each having the coded image data of the picture No. are generated.

  The encoder 102 divides the plurality of layers into two or more predetermined number of layer sets as described above regardless of the number of video streams to be generated, and assigns the coded image data of the picture of each layer set to Is added. In this case, for example, “general_level_idc”, which is the level designation value of the bit stream included in the SPS, is used as the identification information, and the higher the layer set on the higher layer side, the higher the value. Since “sub_layer_level_idc” can be transmitted by SPS for each sublayer (sublayer), this “sub_layer_level_idc” may be used as identification information. The above is supplied not only in the SPS but also in the VPS.

  In this case, the value of the level designation value of each layer set is a value corresponding to the frame rate composed of the pictures of this layer set and the pictures of all layer sets lower than this layer set. For example, in the example of the layer coding of FIG. 3, the level designation value of the layer set of layers 0 to 3 is a value corresponding to the frame rate including only the pictures of layers 0 to 3, and the level of the layer set of layer 4 is The specified value is a value corresponding to a frame rate composed of pictures of all the layers 0 to 4.

  FIG. 6 shows an example of encoding, decoding, display order and delay in hierarchical encoding. This example corresponds to the hierarchical coding example of FIG. 3 described above. This example shows a case in which all layers (all layers) are hierarchically encoded at full time resolution. FIG. 6A shows an encoder input. As shown in FIG. 6B, each picture is encoded in the encoding order with a delay of 16 pictures, and an encoded stream is obtained. FIG. 6B shows a decoder input, in which each picture is decoded in decoding order. Then, as shown in FIG. 6C, the image data of each picture is obtained in the display order with a delay of 4 pictures.

  FIG. 7A shows an encoded stream similar to the encoded stream shown in FIG. 6B described above, divided into three stages of layers 0 to 2, layer 3, and layer 4. Here, “Tid” indicates temporal_id. FIG. 7B shows a display expectation (display order) in a case where each picture of the sub-layers of layers 0 to 2, ie, Tid = 0 to 2, is selectively decoded. FIG. 7C shows the display expectation (display order) in the case where each picture in the sub-layers of layers 0 to 3, ie, Tid = 0 to 3, is selectively decoded. Further, FIG. 7D shows a display expectation (display order) in the case where each picture of the layers 0 to 4, that is, all the layers of Tid = 0 to 4 is selectively decoded.

  To decode the encoded stream of FIG. 7A according to the decoding capability, a decoding capability with a full-rate time resolution is required. However, in the case of performing decoding of Tid = 0 to 2, a decoder having a decoding capability of 1/4 with respect to the encoded full time resolution should be able to process. Further, when decoding Tid = 0 to 3, a decoder having a decoding capability of 1/2 should be able to process the encoded full time resolution.

  However, if the pictures belonging to the lower layer referred to in the layer coding are continuous and are coded at the full timing with the time resolution, the ability of the decoder to partially decode cannot catch up. The period A in FIG. 7A corresponds to this. A decoder that decodes a partial hierarchy of Tid = 0 to 2 or Tid = 0 to 3 performs decoding and display with a time axis of 1/4 or 1/2 as shown in the display example. , A, cannot decode continuous pictures coded in full time resolution.

  Ta indicates the time required for the decoding process for each picture in the decoder that decodes Tid = 0 to 2. Tb indicates a time required for a decoding process for each picture in a decoder that decodes Tid = 0 to 3. Tc indicates a time required for a decoding process for each picture in a decoder that decodes Tid = 0 to 4 (all layers). The relationship between these times is Ta> Tb> Tc.

  In this embodiment, as will be described later, the receiving apparatus 200 has a decoder with a low decoding capability and, when selectively decoding a low-level picture, rewrites a decoding time stamp (DTS). It has a function of adjusting a decoding interval of a low-layer picture. As a result, even a decoder having a low decoding ability can perform a reasonable decoding process.

  Returning to FIG. 2, the compressed data buffer (cpb) 103 temporarily stores the video stream including the encoded data of the picture of each layer generated by the encoder 102. The multiplexer 104 reads the video stream stored in the compressed data buffer 103, converts the video stream into PES packets, and further multiplexes the packets into transport packets to obtain a transport stream TS as a multiplexed stream.

  In this embodiment, as described above, the plurality of hierarchies are divided into two or more predetermined number of hierarchies. The multiplexer 104 inserts, into the header (PES header) of the PES packet, identification information for identifying the coded image data of each picture included in the video stream as the coded image data of the picture belonging to which hierarchical group. With this identification information, it becomes possible for the receiving side to take in only the coded image data of the picture of the hierarchical group according to its own decoding capability into the buffer and process it.

  For example, when dividing a plurality of layers into a lower layer set and a higher layer set, the multiplexer 104 uses a well-known 1-bit field of PES priority (PES_priority) present in the PES header. This one-bit field is set to “1” when the PES payload includes the coded image data of the picture of the lower layer set, that is, the priority is set higher. On the other hand, this 1-bit field is set to “0” when the PES payload includes the coded image data of the picture in the higher layer group, that is, the priority is set lower.

  As described above, the transport stream TS includes a single video stream having coded image data of pictures of each layer, or a predetermined number of video streams each having coded image data of pictures of each layer set described above. Is included. The multiplexer 104 inserts layer information and stream configuration information into the transport stream TS.

  The transport stream TS includes a PMT (Program Map Table) as one of PSI (Program Specific Information). The PMT includes a video elementary loop (video ES1 loop) having information related to each video stream. In the video elementary loop, information such as a stream type and a packet identifier (PID) is arranged corresponding to each video stream, and a descriptor describing information related to the video stream is arranged.

  The multiplexer 104 inserts an HEVC descriptor (HEVC_descriptor) as one of the descriptors, and further inserts a newly defined scalability extension descriptor (scalability_extension_descriptor).

  FIG. 8 illustrates a structural example (Syntax) of the HEVC descriptor (HEVC_descriptor). FIG. 9 shows the content (Semantics) of the main information in the structural example.

  An 8-bit field of “descriptor_tag” indicates a descriptor type, and here indicates that it is a HEVC descriptor. The 8-bit field of “descriptor_length” indicates the length (size) of the descriptor, and indicates the number of subsequent bytes as the length of the descriptor.

  An 8-bit field of “level_idc” indicates a level designation value of the bit rate. When “temporal_layer_subset_flag = 1”, there is a 5-bit field of “temporal_id_min” and a 5-bit field of “temporal_id_max”. “Temporal_id_min” indicates the value of temporal_id of the lowest layer of the layer coded data included in the corresponding video stream. “Temporal_id_max” indicates the value of temporal_id of the highest layer of the layer encoded data of the corresponding video stream.

  The 1-bit field of “level_constrained_flag” is newly defined, and indicates that the level specification value (general_level_idc) of the bit stream included in the NAL unit of the VPS can change for each picture. “1” indicates that it can change, and “0” indicates that it does not change.

  As described above, for example, “general_level_idc” is used as identification information of a belonging layer set when a plurality of layers are divided into two or more predetermined number of layer sets. Therefore, in the case of a video stream having coded image data of pictures of a plurality of hierarchical sets, “general_level_idc” can change for each picture. On the other hand, in the case of a video stream having coded image data of pictures of a single hierarchical group, “general_level_idc” does not change for each picture. Alternatively, “sublayer_level_idc” is added to each sublayer, and the decoder processes the data of the corresponding layer by reading the packets of temporal_id within the range that can be decoded.

  The 3-bit field of “scalability_id” is newly defined, and is an ID indicating scalability assigned to each stream when a plurality of video streams provide a scalable service. “0” indicates a base stream, and “1” to “7” are IDs that increase according to the degree of scalability from the base stream.

  FIG. 10 illustrates a structural example (Syntax) of the scalability extension descriptor (scalability_extension_descriptor). FIG. 11 shows the content (Semantics) of main information in the structural example.

  An 8-bit field of “scalability_extension_descriptor_tag” indicates a descriptor type, and here indicates that it is a scalability extension descriptor. An 8-bit field of “scalability_extension_descriptor_length” indicates the length (size) of the descriptor, and indicates the number of subsequent bytes as the length of the descriptor. The 1-bit field of “extension_stream_existing_flag” is a flag indicating that there is an extended service by another stream. “1” indicates that there is an extension stream, and “0” indicates that there is no extension stream.

  A 3-bit field of “extension_type” indicates an extension type. “001” indicates that the extension is scalable in the time direction. “010” indicates that the extension is spatially scalable. “011” indicates that the extension is bit rate scalable.

  The 4-bit field of “number_of_streams” indicates the total number of streams involved in the distribution service. The 3-bit field of “scalability_id” is an ID indicating scalability assigned to each stream when a plurality of video streams provide a scalable service. “0” indicates a base stream, and “1” to “7” are IDs that increase according to the degree of scalability from the base stream.

  The 3-bit field of “number_of_layers” indicates the total number of layers of the stream. The “8-bit field of“ sublayer_level_idc ”indicates the level_idc value corresponding to the decoder, including the lower layer of the corresponding sublayer indicated by“ temporal_id. ”“ Number of layers ”indicates the NAL unit header. It includes all the values of “Nuh_temporal_id_plus1”, and by detecting this, the demultiplexer (demuxer) detects in advance which layer the decoder corresponding to the predetermined level_idc can decode by using “sublayer_level_idc”. It becomes possible to recognize.

  As described above, in this embodiment, the level designation value (general_level_idc) of the bit rate included in the SPS is the identification information of the belonging layer set when a plurality of layers are divided into two or more predetermined number of layer sets. Used as The value of the level designation value of each hierarchical set is a value corresponding to a frame rate composed of pictures of this hierarchical set and pictures of all hierarchical sets lower than this hierarchical set.

  FIG. 12 shows a configuration example of the multiplexer 104. It has a PES priority generation unit 141, a section coding unit 142, PES packetization units 143-1 to 143-N, a switch unit 144, and a transport packetization unit 145.

  The PES packetizers 143-1 to 143-N read the video streams 1 to N stored in the compressed data buffer 103, respectively, and generate PES packets. At this time, the PES packetizers 143-1 to 143-N add DTS (Decoding Time Stamp) and PTS (Presentation Time Stamp) time stamps to the PES header based on the HRD information of the video streams 1 to N. In this case, the "cpu_removal_delay" and "dpb_output_delay" of each picture are referred to, converted into DTS and PTS, respectively, with an accuracy synchronized with the STC (System Time Clock) time, and arranged at a predetermined position in the PES header.

  The PES priority generating unit 141 is supplied with information on the number of layers (Number of layers) and the number of streams (Number of streams) from the CPU 101. The PES priority generation unit 141 generates priority information of each layer set when a plurality of layers indicated by the number of layers are divided into two or more predetermined number of layer sets. For example, when the data is divided into two, a value to be inserted into the 1-bit field of “PES_priority” of the PES packet header (“1” for the lower layer set and “0” for the higher layer set) is generated.

  The priority information of each hierarchical set generated by the PES priority generation unit 141 is supplied to the PES packetization units 143-1 to 143-N. The PES packetizers 143-1 to 143-N insert the priority of each layer set as identification information into the header of the PES packet containing the coded image data of the picture of the layer set.

  Note that the process of inserting the priority of the hierarchical group to which the picture belongs into the header of the PES packet for each picture as the header information is limited to the case where the encoder 102 generates a single video stream (single stream). You may. In this case, the processing is performed only by the PES packetizing unit 143-1.

  The switch unit 144 selectively extracts the PES packets generated by the PES packetizing units 143-1 to 143-N based on the packet identifier (PID), and sends the packets to the transport packetizing unit 145. The transport packetizer 145 generates a TS packet including a PES packet in a payload, and obtains a transport stream TS.

  The section coding unit 142 generates various section data to be inserted into the transport stream TS. The section coding unit 142 is supplied with information on the number of layers (Number of layers) and the number of streams (Number of streams) from the CPU 101. The section coding unit 142 generates the above-mentioned HEVC descriptor (HEVC_descriptor) and scalability extension descriptor (scalability_extension_descriptor) based on this information.

  The section coding unit 142 sends various section data to the transport packetizing unit 145. The transport packetizing unit 145 generates a TS packet including the section data and inserts the generated TS packet into the transport stream TS.

  FIG. 13 shows a processing flow of the multiplexer 104. In this example, a plurality of hierarchies are divided into a lower hierarchy group and a higher hierarchy group. The multiplexer 104 starts the process in step ST1, and then proceeds to the process in step ST2. In step ST2, the multiplexer 104 sets the temporal_id_ of each picture of the video stream (video elementary stream) and the number of encoded streams to be configured.

  Next, in step ST3, the multiplexer 104 refers to the HRD information (cpu_removal_delay, dpb_output_delay), determines DTS and PTS, and inserts the DTS and PTS at a predetermined position in the PES header.

  Next, in step ST4, the multiplexer 104 determines whether or not the stream is a single stream (single video stream). If the stream is a single stream, the multiplexer 104 proceeds with the multiplexing process using one PID (packet identifier) in step ST5, and then proceeds to the process in step ST7.

  In step ST7, the multiplexer 104 determines whether each of the pictures is a picture (slice) of a lower hierarchical set. If the picture belongs to a low hierarchical set, the multiplexer 104 sets “PES_priority” of the header of the PES packet including the encoded image data of the picture in the payload to “1” in step ST8. On the other hand, when the picture is a high-layer set (non-low-layer set), the multiplexer 104 sets “PES_priority” of the header of the PES packet including the coded image data of the picture in the payload to “0” in step ST9. I do. The multiplexer 104 proceeds to the processing of step ST10 after the processing of steps ST8 and ST9.

  Here, the association between a picture and a slice will be described. A picture is a concept, and its structure is the same as a slice. One picture is divided into a plurality of slices, and the fact that the plurality of slices are the same as an access unit can be understood from a parameter set.

  If it is not a single stream in step ST4, the multiplexer 104 proceeds with the multiplexing process using a plurality of packets PIDs (packet identifiers) in step ST6, and then proceeds to the process in step ST10. In this step ST10, the multiplexer 104 inserts the encoded stream (video elementary stream) into the PES payload to make a PES packet.

  Next, in step ST11, the multiplexer 104 codes a HEVC descriptor, a scalability extension descriptor, and the like. Then, in step ST12, the multiplexer 104 converts the packet into a transport packet to obtain a transport stream TS. Thereafter, the multiplexer 104 ends the process in step ST13.

  FIG. 14 illustrates a configuration example of the transport stream TS in the case of performing distribution by a single stream. This transport stream TS includes one video stream. That is, in this configuration example, a PES packet “video PES1” of a video stream having encoded image data of, for example, HEVC of a plurality of layers of a picture exists, and a PES packet “audio PES1” of an audio stream exists.

  The coded image data of each picture has NAL units such as VPS, SPS, and SEI. As described above, temporal_id indicating the hierarchy of the picture is inserted into the header of the NAL unit of each picture. Also, for example, the VPS includes a bit rate level designation value (general_level_idc). In addition, for example, “cpb_removal_delay” and “dpb_output_delay” are included in the picture timing SEI (Picture timing SEI).

  Further, a field indicating the 1-bit priority of “PES_priority” exists in the header (PES header) of the PES packet. By this “PES_priority”, it is possible to identify whether the coded image data of the picture included in the PES payload is that of a picture of a lower layer set or a picture of a higher layer set.

  In addition, the transport stream TS includes a PMT (Program Map Table) as one of PSI (Program Specific Information). This PSI is information describing to which program each elementary stream included in the transport stream belongs.

  The PMT has a program loop that describes information related to the entire program. Further, the PMT has an elementary loop having information related to each elementary stream. In this configuration example, a video elementary loop (video ES1 loop) exists and an audio elementary loop (audio ES1 loop) exists.

  In the video elementary loop, information such as a stream type and a packet identifier (PID) is arranged corresponding to the video stream (video PES1), and a descriptor describing information related to the video stream is also arranged. You. As one of the descriptors, the above-described HEVC descriptor (HEVC_descriptor) and scalability extension descriptor (scalability_extension_descriptor) are inserted.

  Returning to FIG. 2, transmitting section 105 modulates transport stream TS by a modulation method suitable for broadcasting such as QPSK / OFDM, and transmits an RF modulated signal from a transmitting antenna.

  The operation of the transmitting apparatus 100 shown in FIG. 2 will be briefly described. Uncompressed moving image data is input to the encoder 102. The encoder 102 performs hierarchical coding on the moving image data. That is, in the encoder 102, the image data of each picture constituting the moving image data is classified and encoded into a plurality of layers, and a video stream having encoded image data of the picture of each layer is generated. At this time, the pictures are coded so that the referenced picture belongs to the own layer and / or a layer lower than the own layer.

  The encoder 102 generates a single video stream having the coded image data of the picture of each layer, or divides a plurality of layers into a predetermined number of two or more layer sets, and A predetermined number of video streams each having encoded image data are generated.

  The video stream including the encoded data of the picture of each layer generated by the encoder 102 is supplied to the compressed data buffer (cpb) 103 and is temporarily stored. In the multiplexer 104, the video stream stored in the compressed data buffer 103 is read, converted into PES packets, further converted into transport packets, and multiplexed, thereby obtaining a transport stream TS as a multiplexed stream.

  In the multiplexer 104, for example, in the case of a single video stream (single stream), in the header (PES header) of the PES packet, the coded image data of each picture included in the video stream is encoded into a picture belonging to any hierarchical set. Identification information for identifying whether the data is image data is inserted. For example, when a plurality of layers are divided into a low layer set and a high layer set, a 1-bit field of the PES priority (PES_priority) of the PES header is used.

  In the multiplexer 104, the layer information and the stream configuration information are inserted into the transport stream TS. That is, in the multiplexer 104, the HEVC descriptor (HEVC_descriptor) and the scalability extension descriptor (scalability_extension_descriptor) are inserted into the video elementary loop corresponding to each video stream.

  The transport stream TS generated by the multiplexer 104 is sent to the transmitting unit 105. In the transmission section 105, the transport stream TS is modulated by a modulation method suitable for broadcasting such as QPSK / OFDM, and an RF modulation signal is transmitted from a transmission antenna.

"Receiver configuration"
FIG. 15 illustrates a configuration example of the receiving device 200. The receiving apparatus 200 includes a CPU (Central Processing Unit) 201, a receiving unit 202, a demultiplexer 203, and a compressed data buffer (cpb: coded picture buffer) 204. The receiving apparatus 200 includes a decoder 205, a non-compressed data buffer (dpb: decoded picture buffer) 206, and a post-processing unit 207. The CPU 201 configures a control unit and controls the operation of each unit of the receiving device 200.

  The receiving unit 202 demodulates the RF modulated signal received by the receiving antenna, and acquires a transport stream TS. The demultiplexer 203 selectively extracts, from the transport stream TS, coded image data of a picture in a hierarchical group according to a decoding capability (Decoder temporal layer capability), and sends the coded image data to a compressed data buffer (cpb: coded picture buffer) 204. .

  FIG. 16 shows a configuration example of the demultiplexer 203. The demultiplexer 203 includes a TS adaptation field extractor 231, a clock information extractor 232, a TS payload extractor 233, a section extractor 234, a PSI table / descriptor extractor 235, and a PES packet extractor 236. ing. Further, the demultiplexer 203 includes a PES header extraction unit 237, a time stamp extraction unit 238, an identification information extraction unit 239, a PES payload extraction unit 240, and a stream configuration unit (stream composer) 241.

  The TS adaptation field extraction unit 231 extracts the adaptation field from the TS packet having the adaptation field of the transport stream TS. The clock information extraction unit 232 extracts the PCR from the adaptation field including the PCR (Program Clock Reference), and sends the extracted PCR to the CPU 201.

  The TS payload extracting unit 233 extracts the TS payload from a TS packet having the TS payload of the transport stream TS. The section extraction unit 234 extracts the section data from the TS payload including the section data. The PSI table / descriptor extraction unit 235 analyzes the section data extracted by the section extraction unit 234, and extracts a PSI table and a descriptor. Then, the PSI table / descriptor extraction unit 235 sends the minimum value (min) and the maximum value (max) of temporal_id to the CPU 201 and also sends it to the stream composition unit 241.

  The PES packet extracting unit 236 extracts the PES packet from the TS payload including the PES packet. The PES header extractor 237 extracts a PES header from the PES packet extracted by the PES packet extractor 236. The time stamp extracting unit 238 extracts a time stamp (DTS, PTS) inserted in the PES header for each picture, and sends the extracted time stamp to the CPU 201 and also sends it to the stream composition unit 241.

  The identification information extraction unit 239 extracts identification information, which is inserted into the PES header for each picture and identifies the hierarchical set to which the picture belongs, and sends it to the stream composition unit 241. For example, when a plurality of layers are divided into a lower layer set and a higher layer set, the priority information of the 1-bit field of “PES_priority” of the PES header is extracted and sent to the stream configuration unit 241. Note that this identification information is always inserted on the transmission side when the transport stream TS includes a single video stream, but is included when the transport stream TS includes a plurality of video streams. May not be inserted.

  The PES payload extraction unit 240 extracts a PES payload, that is, encoded image data of a picture of each layer, from the PES packet extracted by the PES packet extraction unit 236. The stream configuration unit 241 selectively extracts encoded image data of a picture of a layer set corresponding to a decoding capability (Decoder temporal layer capability) from encoded image data of a picture of each layer extracted by the PES payload extraction unit 240. To a compressed data buffer (cpb: coded picture buffer) 204. In this case, the stream configuration unit 241 refers to the hierarchy information obtained by the PSI table / descriptor extraction unit 235, the stream configuration information, the identification information (priority information) extracted by the identification information extraction unit 239, and the like.

  For example, consider a case where the frame rate of a video stream (encoded stream) included in the transport stream TS is 120 fps. For example, it is assumed that a plurality of layers are divided into a lower layer group and a higher layer group, and the frame rate of the picture in each layer group is 60 fps. For example, in the layer coding example shown in FIG. 3 described above, layers 0 to 3 are set as a layer set on the lower layer side, and a decoder corresponding to level_idc of 60 fps can decode. Further, the layer 4 is a layer set on the higher layer side, and a decoder that supports level_idc of 120 fps can decode.

  In this case, the transport stream TS includes a single video stream (coded stream) having the coded data of the picture of each layer, or the coded image data of the picture of the layer set of the lower layer. The video stream includes two video streams (encoded streams): a base stream (B_str) having the extended stream and an extended stream (E_str) having encoded image data of a picture in a higher layer.

  When the decoding capability corresponds to 120 fps, the stream configuration unit 241 extracts the coded image data of the pictures of all the layers and sends the coded image data to the compressed data buffer (cpb) 204. On the other hand, when the decoding capability is not compatible with 120 fps but is compatible with 60 fps, the stream configuration unit 241 extracts only the coded image data of the picture of the lower hierarchical set and compresses the compressed data buffer (cpb). Send to 204.

  FIG. 17 illustrates an example of a picture (slice) selection of the stream configuration unit 241 when a single video stream (encoded stream) is included in the transport stream TS. Here, “High” indicates a picture of a layer set on a higher layer side, and “Low” indicates a picture of a layer set on a lower layer side. “P” indicates “PES_priority”.

  When the decoding capability corresponds to 120 fps, the stream forming unit 241 extracts the coded image data of the pictures of all the layers and sends the coded image data to the compressed data buffer (cpb) 204. On the other hand, when the decoding capability is not compatible with 120 fps but is compatible with 60 fps, the stream configuration unit 241 performs filtering based on “PES_priority” and performs picture processing of the lower hierarchical set where P = 1. And sends it to the compressed data buffer (cpb) 204.

  FIG. 18 illustrates an example of picture (slice) selection of the stream configuration unit 241 when the transport stream TS includes two video streams (encoded streams), a base stream and an extension stream. Here, “High” indicates a picture of a layer set on a higher layer side, and “Low” indicates a picture of a layer set on a lower layer side. The packet identifier (PID) of the base stream is PID A, and the packet identifier (PID) of the extension stream is PID B.

  When the decoding capability corresponds to 120 fps, the stream forming unit 241 extracts the coded image data of the pictures of all the layers and sends the coded image data to the compressed data buffer (cpb) 204. In this case, the stream configuration unit 241 converts the encoded image data of each picture into one stream based on the decode timing information and sends the stream to the compressed data buffer (cpb) 204.

  In that case, the value of DTS is viewed as the decoding timing, and the streams are combined into one such that the value increases monotonously between pictures. The picture grouping process itself is performed on a plurality of streams read from the plurality of compressed data buffers (cpb) 204 when there are a plurality of compressed data buffers (cpb) 204 for the number of streams. A decoding process may be performed.

  On the other hand, when the decoding capability does not support 120 fps but does support 60 fps, the stream configuration unit 241 performs filtering based on the packet identifier (PID) to generate a PID A of the lower hierarchical set. Only the picture is taken out and sent to the compressed data buffer (cpb) 204.

  Note that the stream configuration unit 241 has a function of selectively rewriting the decode time stamp of the coded image data of each picture to be sent to the compressed data buffer (cpb) 204 to adjust the decoding interval of the lower hierarchical picture. As a result, even when the decoding capability of the decoder 205 is low, a reasonable decoding process can be performed.

  FIG. 19 shows an example of the layer coding shown in FIG. 3, in which the stream is divided into a lower layer set and a higher layer set, and the stream configuration unit 241 determines whether the picture belongs to the lower layer set. Is selectively extracted and sent to the compressed data buffer (cpb) 204.

  FIG. 19A shows the decode timing before the decode interval adjustment. In this case, there is a variation in the decoding interval between pictures, and the shortest decoding interval is equal to the 120 fps full resolution decoding interval. On the other hand, FIG. 19B shows the decode timing after the adjustment of the decode interval. In this case, the decoding intervals between pictures are made equal, and the decoding intervals are １ of the decoding intervals of the full resolution. Thus, the decoding interval is adjusted in each hierarchy according to the capability of the target decoder.

  FIG. 20 illustrates an example of a processing flow of the demultiplexer 203. This processing flow illustrates a case where the transport stream TS includes a single video stream (encoded stream).

  The demultiplexer 203 starts the process in step ST31, and then proceeds to the process in step ST32. In this step ST32, a decoding capability (Decoder temporal layer capability) is set by the CPU 201. Next, in step ST33, the demultiplexer 203 determines whether or not there is a capability of decoding all layers (layers).

  When all layers are capable of decoding, in step ST34, the demultiplexer 203 demultiplexes all TS packets passing through the corresponding PID filter and performs section parsing. After that, the demultiplexer 203 proceeds to the process of step ST35.

  In step ST33, when there is no ability to decode all layers, in step ST36, the demultiplexer 203 demultiplexes the TS packet whose “PES_priority” is “1” and performs section parsing. After that, the demultiplexer 203 proceeds to the process of step ST35.

  In step ST35, the demultiplexer 203 reads the HEVC descriptor (HEVC_descriptor) and the scalability extension descriptor (scalability_extension_descriptor) in the target PID section, and determines whether or not there is an extension stream, the scalable type, the number and ID of the stream, Obtain the maximum and minimum values of temporal_id, and the level corresponding to the decoder of each layer.

  Next, in step ST37, the demultiplexer 203 transfers the encoded stream to be subjected to the PID to the compressed data buffer (cpb) 204, and notifies the CPU 201 of the DTS and PTS. After the process in step ST37, the demultiplexer 203 ends the process in step ST38.

  FIG. 21 illustrates an example of a processing flow of the demultiplexer 203. This processing flow shows a case where the transport stream TS includes two video streams (encoded streams) of a base stream and an extension stream.

  The demultiplexer 203 starts the process in step ST41, and then proceeds to the process in step ST42. In this step ST42, a decoding capability (Decoder temporal layer capability) is set by the CPU 201. Next, in step ST43, the demultiplexer 203 determines whether or not there is a capability of decoding all layers (layers).

  When there is a capability to decode all layers, the demultiplexer 203 demultiplexes a plurality of streams constituting all layers by a PID filter in step ST44, and performs section parsing. After that, the demultiplexer 203 proceeds to the process of step ST45.

  If there is no ability to decode all layers in step ST43, the demultiplexer 203 demultiplexes the stream of PID = PID A in step ST46 and performs section parsing. After that, the demultiplexer 203 proceeds to the process of step ST45.

  In step ST45, the demultiplexer 203 reads the HEVC descriptor (HEVC_descriptor) and the scalability extension descriptor (scalability_extension_descriptor) in the target PID section, and determines whether or not there is an extension stream, the scalable type, the number and ID of the stream, Obtain the maximum and minimum values of temporal_id, and the level corresponding to the decoder of each layer.

  Next, in step ST47, the demultiplexer 203 combines the coded stream to be subjected to the PID into one stream based on the DTS (PTS if not), transfers the stream to the compressed data buffer (cpb) 204, and , DTS, and PTS to the CPU 201. After the process of step ST47, the demultiplexer 203 ends the process in step ST48.

  Returning to FIG. 15, the compressed data buffer (cpb) 204 temporarily stores the video stream (encoded stream) extracted by the demultiplexer 203. The decoder 205 extracts, from the video stream stored in the compressed data buffer 204, coded image data of a picture of a layer specified as a layer to be decoded. Then, the decoder 205 decodes the extracted coded image data of each picture at the decoding timing of the picture, and sends the decoded image data to the uncompressed data buffer (dpb) 206.

  Here, the hierarchy to be decoded by the CPU 201 is designated by the temporal_id in the decoder 205. The designated layer is the entire layer included in the video stream (encoded stream) extracted by the demultiplexer 203 or a part of the lower layer, and is set automatically by the CPU 201 or in response to a user operation. You. The decoder 205 is given a decoding timing from the CPU 201 based on a DTS (Decoding Time stamp). When decoding the coded image data of each picture, the decoder 205 reads out and uses the image data of the referenced picture from the uncompressed data buffer 206 as necessary.

  FIG. 22 shows a configuration example of the decoder 205. The decoder 205 has a temporal ID analysis unit 251, a target hierarchy selection unit 252, and a decoding unit 253. The temporal ID analysis unit 251 reads the video stream (encoded stream) stored in the compressed data buffer 204, and analyzes the temporal_id inserted in the NAL unit header of the encoded image data of each picture.

  The target layer selection unit 252 extracts, from the video stream read from the compressed data buffer 204, the coded image data of the picture of the layer specified as the layer to be decoded based on the analysis result of the temporal ID analysis unit 251. . The decoding unit 253 sequentially decodes the encoded image data of each picture extracted by the target hierarchy selection unit 252 at a decoding timing, and sends the decoded image data to the uncompressed data buffer (dpb) 206.

  In this case, the decoding unit 253 analyzes the VPS and the SPS, grasps, for example, the level designation value “sublayer_level_idc” of the bit rate for each sublayer, and confirms whether the decoding can be performed within the decoding capability. Also, in this case, the decoding unit 253 analyzes the SEI, grasps, for example, “initial_cpb_removal_time” and “cpb_removal_delay”, and checks whether the decoding timing from the CPU 201 is appropriate.

  When decoding the slice (Slice), the decoding unit 253 acquires “ref_idx_l0_active (ref_idx_l1_active)” from the slice header (Slice header) as information indicating a prediction destination in the time direction, and performs prediction in the time direction. The decoded picture is processed as a reference by another picture using “short_term_ref_pic_set_idx” or “it_idx_sps” obtained from the slice header as an index.

  Returning to FIG. 15, the uncompressed data buffer (dpb) 206 temporarily stores the image data of each picture decoded by the decoder 205. The post-processing unit 207 performs a process of adjusting the frame rate of the image data of each picture sequentially read from the uncompressed data buffer (dpb) 206 at the display timing according to the display capability. In this case, the display timing is given from the CPU 201 based on the PTS (Presentation Time stamp).

  For example, when the frame rate of the decoded image data of each picture is 120 fps and the display capability is 120 fps, the post-processing unit 207 sends the decoded image data of each picture to the display as it is. Further, for example, when the frame rate of the image data of each decoded picture is 120 fps and the display capability is 60 fps, the post-processing unit 207 determines that the temporal resolution of the decoded image data is A sub-sampling process is performed so as to be 1/2 times, and sent to a display as 60 fps image data.

  Further, for example, when the frame rate of the image data of each picture after decoding is 60 fps and the display capability is 120 fps, the post-processing unit 207 determines that the temporal resolution of the image data of each picture after decoding is Interpolation processing is performed so that the image data becomes twice, and the image data is sent to a display as 120 fps image data. For example, when the frame rate of the decoded image data of each picture is 60 fps and the display capability is 60 fps, the post-processing unit 207 sends the decoded image data of each picture to the display as it is.

  FIG. 23 illustrates a configuration example of the post-processing unit 207. This example is an example in which the frame rate of the image data of each picture after decoding is 120 fps or 60 fps and the display capability is 120 fps or 60 fps as described above.

  The post-processing unit 207 includes an interpolation unit 271, a sub-sampling unit 272, and a switch unit 273. The image data of each picture after decoding from the uncompressed data buffer 206 is directly input to the switch unit 273, or input to the switch unit 273 after being set to a double frame rate by the interpolation unit 271, or to the sub-sample unit. After the frame rate is set to 倍 in 272, the frame rate is input to the switch unit 273.

  The switch unit 273 is supplied with selection information from the CPU 201. This selection information is generated automatically by the CPU 201 with reference to the display capability or in response to a user operation. The switch unit 273 selectively outputs any one of the inputs based on the selection information. As a result, the frame rate of the image data of each picture sequentially read from the uncompressed data buffer (dpb) 206 at the display timing matches the display capability.

  FIG. 24 illustrates an example of a processing flow of the decoder 205 and the post-processing unit 207. The decoder 205 and the post-processing unit 207 start the process in step ST51, and then proceed to the process in step ST52. In this step ST52, the decoder 205 reads out the video stream to be decoded stored in the compressed data buffer (cpb) 204, and selects a picture of a hierarchy designated as a decoding target by the CPU 201 based on the temporal_id.

  Next, in step ST53, the decoder 205 sequentially decodes the coded image data of each selected picture at a decoding timing, and transfers the decoded image data of each picture to the uncompressed data buffer (dpb) 206. , Temporarily accumulate. Next, in step ST54, the post-processing unit 207 reads the image data of each picture from the uncompressed data buffer (dpb) 206 at the display timing.

  Next, the post-processing unit 207 determines whether or not the frame rate of the read image data of each picture satisfies the display capability. When the frame rate does not match the display capability, the post-processing unit 207 sends the frame rate to the display according to the display capability in step ST56, and then ends the process in step ST57. On the other hand, when the frame rate matches the display capability, the post-processing unit 207 sends the frame rate to the display as it is in step ST58, and ends the process in step ST57.

  The operation of receiving apparatus 200 shown in FIG. 15 will be briefly described. In the receiving unit 202, the RF modulation signal received by the receiving antenna is demodulated, and a transport stream TS is obtained. This transport stream TS is sent to the demultiplexer 203. In the demultiplexer 203, coded image data of a picture in a hierarchical group according to the decoding capability (Decoder temporal layer capability) is selectively extracted from the transport stream TS, sent to the compressed data buffer (cpb) 204, and temporarily Is accumulated.

  The decoder 205 extracts, from the video stream stored in the compressed data buffer 204, the coded image data of the picture of the layer specified as the layer to be decoded. Then, the decoder 205 decodes the extracted coded image data of each picture at the decoding timing of the picture, sends the coded image data to the uncompressed data buffer (dpb) 206, and temporarily stores the coded image data. In this case, when the encoded image data of each picture is decoded, the image data of the referenced picture is read from the uncompressed data buffer 206 and used as needed.

  Image data of each picture sequentially read from the uncompressed data buffer (dpb) 206 at the display timing is sent to the post-processing unit 207. In the post-processing unit 207, interpolation or sub-sampling is performed on the image data of each picture so that the frame rate matches the display capability. The image data of each picture processed by the post-processing unit 207 is supplied to a display, and a moving image is displayed based on the image data of each picture.

  As described above, in the transmission / reception system 10 shown in FIG. 1, on the transmitting side, the layer of the video stream (the header of the PES packet) includes the encoded image data of each picture included in the video stream in any hierarchical group. The identification information for identifying whether the picture belongs to the coded image data of the picture belonging to. Therefore, for example, on the receiving side, by using this identification information, it becomes easy to selectively decode the coded image data of the picture of the hierarchy of the predetermined hierarchy or less according to the decoding capability.

  In the transmission / reception system 10 shown in FIG. 1, a scalability extension descriptor (scalability_extension_descriptor) or the like is inserted into a layer of the transport stream TS on the transmission side. Therefore, for example, the receiving side can easily grasp the layer information in the layer coding, the configuration information of the video stream included in the transport stream TS, and perform an appropriate decoding process.

  In the transmission / reception system 10 illustrated in FIG. 1, the receiving side selectively compresses coded image data of a picture of a layer of a predetermined layer or less according to a decoding capability (Decoder temporal layer capability) from a received video stream. The data is captured by the data buffer 204 and decoded. Therefore, for example, it is possible to perform an appropriate decoding process according to the decoding capability.

  Further, in the transmission / reception system 10 shown in FIG. 1, the function of adjusting the decoding interval of the low-level picture by rewriting the decoding time stamp of the coded image data of each picture selectively taken into the compressed data buffer 204 on the receiving side. With Therefore, for example, even when the decoding capability of the decoder 205 is low, a reasonable decoding process can be performed.

  In the transmission / reception system 10 shown in FIG. 1, on the receiving side, the post-processing unit 207 adjusts the frame rate of the image data of each decoded picture to the display capability. Therefore, for example, even when the decoding capability is low, it is possible to obtain image data at a frame rate that is compatible with the high display capability.

<2. Modification>
Note that, in the above-described embodiment, identification information for identifying which coded image data of each picture included in the video stream is coded image data of a picture belonging to which hierarchical group out of a predetermined number of hierarchical groups, An example in which the PES packet is inserted into the header (PES header) of the PES packet has been described. However, the insertion position of the identification information is not limited to this.

  For example, the multiplexer 104 (see FIG. 2) may insert this identification information into the adaptation field of the TS packet having the adaptation field. For example, when dividing a plurality of layers into a lower layer set and a higher layer set, the multiplexer 104 uses a 1-bit field of a well-known elementary stream priority indicator (elementary_stream_priority_indicator) existing in the adaptation field.

  This 1-bit field is set to “1” when the payload of the subsequent TS packet includes a PES packet having coded image data of a picture of a lower hierarchical set in the payload, that is, a higher priority. On the other hand, this 1-bit field is set to “0” when the payload of the subsequent TS packet includes a PES packet having the coded image data of the picture of the lower layer set in the payload, that is, the priority is set lower. .

  FIG. 25 shows an example of the arrangement of the adaptation fields. This example is a case where a plurality of hierarchies are divided into a lower hierarchy set and a higher hierarchy set, and a case where a 1-bit field of an elementary stream priority indicator (elementary_stream_priority_indicator) is used.

  In the example shown in the figure, a TS packet having an adaptation field is arranged immediately before each of a predetermined number of TS packet groups having divided PES packets each having coded image data of one picture as a payload. In this case, when the one picture is a picture of the lower layer set, the 1-bit field of the elementary stream priority indicator is set to “1”. On the other hand, when the one picture is a picture of a layer set on the higher layer side, the 1-bit field of the elementary stream priority indicator is set to “0”.

  As shown in FIG. 25, by arranging a TS packet having an adaptation field, on the receiving side, encoded image data of a picture included in a video stream is encoded data of a picture belonging to any hierarchical group. Can be easily identified. In the arrangement example of FIG. 25, a TS packet having an adaptation field is arranged for each picture. However, each time a hierarchical group to which a picture belongs is switched, a TS packet having an adaptation field is arranged immediately before. It may be made to do.

  FIG. 26 illustrates a configuration example of the multiplexer 104A of the transmission device 100 when the identification information of the hierarchical set is inserted into the adaptation field as described above. 26, portions corresponding to those in FIG. 12 are denoted by the same reference numerals, and detailed description thereof will be omitted. This multiplexer 104A has an adaptation field priority instructing unit 146 instead of the PES priority generating unit 141 in the multiplexer 104 of FIG.

  The priority instruction unit 146 is supplied with information on the number of layers (Number of layers) and the number of streams (Number of streams) from the CPU 101. The priority instructing unit 146 generates priority information of each layer set when a plurality of layers indicated by the number of layers are divided into two or more predetermined number of layer sets. For example, in the case of division into two, a value to be inserted into the 1-bit field of the elementary stream priority indicator (“1” for the lower layer set and “0” for the higher layer set) is generated.

  The priority information of each layer set generated by the priority instructing unit 146 is supplied to the transport packetizing unit 145. The transport packetizing unit 145 arranges a TS packet having an adaptation field immediately before each of a predetermined number of TS packet groups having divided PES packets each having coded image data of one picture as a payload. Then, in that case, the transport packetizing unit 145 inserts, into the adaptation field, priority information corresponding to the layer set to which the picture belongs as identification information.

  FIG. 27 illustrates a configuration example of the transport stream TS when the identification information of the hierarchical set is inserted into the adaptation field as described above. This configuration example is substantially the same as the configuration example shown in FIG. 14 described above. In this configuration example, there is a TS packet having an adaptation field, and identification information for identifying a layer set to which each picture belongs is inserted into the adaptation field. For example, when a plurality of layers are divided into a lower layer set and a higher layer set, a 1-bit field of an elementary stream priority indicator (elementary_stream_priority_indicator) is used.

  FIG. 28 illustrates a configuration example of the demultiplexer 203A of the reception device 200 when the identification information of the hierarchical set is inserted into the adaptation field as described above. 28, portions corresponding to those in FIG. 16 are denoted by the same reference numerals, and detailed description thereof will be omitted. This demultiplexer 203A has an identification information extraction unit 242 instead of the identification information extraction unit 239 in the demultiplexer 203 of FIG.

  The identification information extraction unit 242 extracts identification information from the adaptation field and sends it to the stream composition unit 241. For example, when a plurality of layers are divided into a lower layer group and a higher layer group, the priority information of the 1-bit field of “elementary_stream_priority_indicator” of the adaptation field is extracted and sent to the stream configuration unit 241.

  The stream configuration unit 241 selectively extracts encoded image data of a picture of a layer set according to a decoding capability (Decoder temporal layer capability) from encoded image data of a picture of each layer extracted by the PES payload extraction unit 240. , To the compressed data buffer (cpb) 204. In this case, the stream configuration unit 241 refers to the layer information obtained by the PSI table / descriptor extraction unit 235, the stream configuration information, the identification information (priority information) extracted by the identification information extraction unit 242, and the like.

  Further, in the above-described embodiment, the transmission / reception system 10 including the transmission device 100 and the reception device 200 has been described, but the configuration of the transmission / reception system to which the present technology can be applied is not limited to this. For example, the part of the receiving device 200 may be, for example, a configuration of a set-top box and a monitor connected by a digital interface such as (High-Definition Multimedia Interface (HDMI). Is a trademark.

  Further, in the above-described embodiment, an example has been described in which the container is a transport stream (MPEG-2 TS). However, the present technology can be similarly applied to a system configured to be distributed to a receiving terminal using a network such as the Internet. In the Internet distribution, it is often distributed in a container of MP4 or another format. That is, as containers, containers of various formats such as a transport stream (MPEG-2 TS) adopted in the digital broadcasting standard and MP4 used in Internet distribution correspond.

For example, FIG. 29 illustrates a configuration example of an MP4 stream. This MP4 stream contains "moov
, "Moof", "mdat", etc. In the box of “mdat”, as a track, a video elementary stream “track1: video ES1” that is an encoded stream of video exists, and an audio elementary stream “track1: audio ES1” that is an encoded stream of audio exists. Exists

  In the box of “moof”, “mfhd (movie fragment header)” exists as a header part, and “track fragment” corresponding to each track exists as a data part thereof. In “track1 fragment (video)” corresponding to “video ES1”, “Independent and disposal samples” exist, and a box called “SampleDependencyTypeBox” corresponding to each picture is inserted therein.

  In this box, it is possible to insert identification information for identifying the coded image data of each picture to which the coded image data of each picture belongs. For example, when dividing a plurality of hierarchies into a two-layer set of the highest layer and the other lower layers, the 2-bit field of "sample_depends_on" and the 2-bit field of "sample_is_depended_on" Insertion is possible.

  FIG. 30 illustrates a structural example (Syntax) of “SampleDependencyTypeBox”. FIG. 31 shows the content (Semantics) of main information in the structure example. In this case, “sample_depends_on” is set to “1” to refer to another picture to indicate that it is not an I picture, and “sample_is_depended_on” is set to “2” to indicate that it is not referred to by other pictures. Can be identified as belonging to the set of. In other states, the picture can be identified as a picture belonging to a hierarchical set of hierarchical layers.

  Instead of using the box of “SampleDependencyTypeBox”, a newly defined box called “SampleScalablePriorityBox” may be used. FIG. 32 illustrates a structural example (Syntax) of “SampleScalablePriorityBox”. FIG. 33 shows the contents (Semantics) of main information in the structural example.

  In this case, when dividing a plurality of hierarchies into two hierarchies, a lowest hierarchy and a higher hierarchy, the identification information is inserted using a 2-bit field of “base_and_priority”. That is, by setting “base_and_priority” to, for example, “1”, it is possible to identify a picture having a low priority and belonging to a higher hierarchical set. On the other hand, by setting “base_and_priority” to, for example, “2”, it is possible to identify a picture that has a high priority and belongs to a low hierarchical set.

In addition, the present technology may have the following configurations.
(1) The image data of each picture constituting the moving image data is classified into a plurality of layers, the image data of the classified picture of each layer is encoded, and the encoded image data of the picture of each layer is encoded. An image encoding unit for generating video data having
A transmitting unit that transmits a container of a predetermined format including the generated video data,
The plurality of hierarchies are divided into a predetermined number of hierarchies of two or more, and in a packet for container of the video data, the coded image data of each picture included in the video data is coded for each of the hierarchies. A transmission device including an identification information insertion unit for inserting identification information for identifying whether the data is image data.
(2) The transmitting device according to (1), wherein the identification information is priority information that is set higher for a lower layer group.
(3) The transmission device according to (1), wherein the identification information is inserted into a header of a PES packet including encoded image data for each picture in a payload.
(4) The transmitting device according to (3), wherein the identification information is inserted using a PES priority field of the header.
(5) The transmitting device according to (1), wherein the identification information is inserted into the adaptation field of a TS packet having an adaptation field.
(6) The transmitting device according to (5), wherein the identification information is inserted using an ES priority indicator field of the adaptation field.
(7) The transmitting device according to (1), wherein the identification information is inserted into a box of a header related to a track of a corresponding picture.
(8) The image encoding unit includes:
Either generate a single video stream having the coded image data of the picture of each layer, or generate a predetermined number of video data each having the coded image data of the picture of each layer set,
The transmission device according to any one of (1) to (7), further including a configuration information insertion unit configured to insert configuration information of a video stream included in the container into a layer of the container.
(9) The image data of each picture constituting the moving image data is classified into a plurality of layers, the image data of the classified pictures of each layer is encoded, and the encoded image data of the pictures of each layer is encoded. An image encoding step for generating video data having
A transmitting unit for transmitting a container of a predetermined format including the generated video data,
The plurality of hierarchies are divided into a predetermined number of hierarchies of two or more, and in a packet for container of the video data, the coded image data of each picture included in the video data is coded for each of the hierarchies. A transmission method comprising an identification information insertion step of inserting identification information for identifying whether the data is image data.
(10) A container of a predetermined format including video data having coded image data of a picture of each layer obtained by classifying and coding image data of each picture constituting moving image data into a plurality of layers. A receiving unit for receiving the
From the video data included in the received container, encoded image data of a picture of a layer lower than a predetermined layer corresponding to a decoding capability is selectively captured into a buffer, and the encoded image data of each picture captured in the buffer is stored. And a picture decoding unit that decodes the picture and obtains picture data of a picture of a layer lower than the predetermined layer.
(11) The plurality of hierarchies are divided into a predetermined number of hierarchies of two or more, and a packet that contains the video data includes a picture to which the coded image data of each picture included in the video data belongs. Identification information for identifying whether the image data is encoded image data is inserted,
The receiving device according to (10), wherein the image decoding unit fetches coded image data of a picture of a predetermined hierarchical group according to the decoding capability into the buffer and decodes the coded image data based on the identification information.
(12) The receiving device according to (11), wherein the identification information is inserted in a header of a PES packet including encoded image data for each picture in a payload.
(13) The receiving device according to (11), wherein the identification information is inserted into the adaptation field of a TS packet having an adaptation field.
(14) The transmitting device according to (11), wherein the identification information is inserted in a box of a header related to a track of a corresponding picture.
(15) The plurality of layers are divided into two or more predetermined number of layer sets, and the received container includes the predetermined number of video streams each having encoded image data of pictures of the predetermined number of layer sets. Is included,
The receiving device according to (10), wherein the image encoding unit captures encoded image data of a picture of a predetermined hierarchical group corresponding to the decoding capability into the buffer and decodes the encoded image data based on the stream identification information.
(16) The image decoding unit includes:
When the coded image data of the picture of the predetermined hierarchical set is included in a plurality of video streams, the coded image data of each picture is taken into the buffer as one stream based on the decode timing information. The receiving device according to claim 1.
(17) The image decoding unit includes:
The receiving device according to any one of (10) to (16), having a function of adjusting a decoding interval of a low-layer picture by rewriting a decode time stamp of encoded image data of each picture selectively taken into a buffer. .
(18) The receiving device according to any one of (10) to (17), further including a post-processing unit that adjusts a frame rate of image data of each picture obtained by the image decoding unit to a display capability.
(19) Includes video data having coded image data of pictures of each layer obtained by classifying and encoding the image data of each picture constituting the moving image data into a plurality of layers by the receiving unit. A receiving step of receiving a container of a predetermined format;
From the video data included in the received container, coded image data of a picture of a predetermined layer or less according to the decoding capability is selectively captured into a buffer, and the coded image data of each picture captured in the buffer is stored. A decoding method for decoding image data to obtain image data of a picture of a layer lower than the predetermined layer.

  The main feature of the present technology is that, in a packet for container of video data, identification information for identifying the coded image data of each picture belonging to each of the coded image data of the pictures included in the video data. , The receiving side can easily use the identification information to selectively decode the coded image data of a picture of a layer lower than a predetermined layer corresponding to the decoding capability. (See FIG. 12).

10 Transmission / Reception System 100 Transmission Device 101 CPU
102: encoder 103: compressed data buffer (cpb)
104, 104A... Multiplexer 105... Transmitting section 141... PES priority generating section 142... Section coding section 143-1 to 143-N... PES packetizing section 144. ..Transport packetizer 146 Adaptation field priority indicator 200 Receiver 201 CPU
202: receiving unit 203: demultiplexer 204: compressed data buffer (cpb)
205: decoder 206: uncompressed data buffer (dpb)
207: Post-processing unit 231: TS adaptation field extracting unit 232: Clock information extracting unit 233: TS payload extracting unit 234: Section extracting unit 235: PSI table / descriptor extracting unit 236 ... PES packet extraction unit 237 ... PES header extraction unit 238 ... time stamp extraction unit 239 ... identification information extraction unit 240 ... PES payload extraction unit 241 ... stream construction unit 242 ... Identification information extracting unit 251 Temporal ID analyzing unit 252 Target layer selecting unit 253 Decoding unit 271 Interpolating unit 272 Sub-sample unit 273 Switch unit

Claims (5)

The image data of each picture constituting the moving image data is hierarchically encoded, and a first video stream having encoded image data of a lower layer side picture and a second video stream having encoded image data of a higher layer side picture includes the picture coding unit for generating a stream,
A level specification value of the first video stream and a level specification value of a video stream combining the first video stream and the second video stream are inserted into the NAL unit of the SPS of the first video stream. And
Together comprising said first video stream and the second video stream generated by the picture coding unit, reference numerals of the pictures included in the video stream of the first in correspondence with the first video stream of each picture of image data is included in the first and stream identification information, the second video stream of the second corresponding to the video stream as belonging to the coded image data of the picture of the lower layer side Second stream identification information indicating that the coded image data belongs to the coded image data of the higher layer picture, and a frame rate consisting of only the lower layer picture corresponding to the first video stream first and descriptors, and the picture of the second corresponding to a video stream the low hierarchy side of the first value has been inserted, based on A multiplexed stream generation unit that generates a multiplexed stream a second value based on the frame rate consisting of a picture of the serial high hierarchy side comprises a second descriptor that is inserted,
A transmission device further comprising a transmission unit that transmits the multiplexed stream generated by the multiplexed stream generation unit . An image encoding unit that hierarchically encodes image data of each picture constituting the moving image data, and a first video stream having encoded image data of a lower layer picture and encoded image data of a higher layer picture Image encoding to generate a second video stream having
A level specification value of the first video stream and a level specification value of a video stream combining the first video stream and the second video stream are inserted into the NAL unit of the SPS of the first video stream. And
Multiplexed stream generating unit, the image coding with including the generated said first video stream and the second video stream in step, the first corresponding to a video stream of the first video stream And first stream identification information indicating that the coded image data of each picture included in the second video stream belongs to the coded image data of the lower layer picture, and the second video stream corresponding to the second video stream. Second stream identification information indicating that the coded image data of each picture included in the stream belongs to the coded image data of the higher layer picture, and the lower layer side corresponding to the first video stream. a first descriptor first value has been inserted based on the frame rate comprising only picture, to the second video stream A multiplexed stream generating step of generating a multiplexed stream by response comprises a second descriptor second value based on the frame rate consisting of a picture of the picture and the high hierarchy side of the lower layer side is inserted ,
A transmission method, further comprising : a transmission step of transmitting the multiplexed stream generated in the multiplexed stream generation step . A first video stream having encoded image data of a lower layer side picture and encoded image data of a higher layer side picture, which are generated by hierarchically encoding the image data of each picture constituting the moving image data, together comprising a second video stream having, in the first encoded image data of each picture included in the video stream of the first corresponding to the video stream of the picture of the lower layer side coded image data First stream identification information indicating that the image data belongs to the second video stream, and coded image data of each picture included in the second video stream corresponding to the second video stream. a second stream identification information indicating that it belongs to, in correspondence with the first video stream frame comprising only a picture of the lower layer side A first descriptor first value based on the rate is inserted, the second based on the frame rate corresponding to the second video stream consists of a picture of the lower layer side of the picture and the high hierarchy side A receiving unit that receives a multiplexed stream including the second descriptor in which the value of
A level specification value of the first video stream and a level specification value of a video stream combining the first video stream and the second video stream are inserted into the NAL unit of the SPS of the first video stream. And
Based on the first stream identification information and the second stream identification information and the first value and the second value, from the multiplexed stream, the first video stream only or the first video stream The receiving apparatus further comprising a processing unit that extracts and decodes both the video stream and the second video stream. The receiving device according to claim 3, wherein the processing unit further performs a process of adjusting a frame rate of image data of each picture obtained by performing the decoding to a display capability. A receiving unit configured to generate a first video stream having encoded image data of a lower layer side picture and a code of a higher layer side picture, which are generated by hierarchically encoding the image data of each picture constituting the moving image data; along with comprising a second video stream with the image data, the encoded image data of each picture included in the first video stream corresponding to the first video stream of the picture of the lower layer side code First stream identification information indicating that the picture belongs to the coded image data, and the coded image data of each picture included in the second video stream corresponding to the second video stream is the a second stream identification information indicating that it belongs to the encoded image data, the only picture corresponding the lower layer side to the first video stream A first descriptor first value has been inserted, based on the frame rate that, based on the frame rate corresponding to the second video stream consists of a picture of the picture and the high hierarchy side of the lower layer side has a receiving step in which the second value receives a multiplexed stream including a second descriptor that is inserted,
A level specification value of the first video stream and a level specification value of a video stream combining the first video stream and the second video stream are inserted into the NAL unit of the SPS of the first video stream. Yes,
A processing unit that, based on the first stream identification information and the second stream identification information and the first value and the second value, outputs only the first video stream from the multiplexed stream; Alternatively, a receiving method further comprising a processing step of extracting and decoding both the first video stream and the second video stream.

Images