JP7257646B2 - Transmission method, reception method, transmission device and reception device - Google Patents

Description

The present invention relates to a transmission method, a reception method, a transmission device and a reception device.

With the advancement of broadcasting and communication services, the introduction of ultra-high-definition video content such as 8K (7680 x 4320 pixels; also referred to as 8K4K below) and 4K (3840 x 2160 pixels; also referred to as 4K2K below) is being considered. It is The receiving device needs to decode and display the encoded data of the received ultra-high-definition moving image in real time. It is difficult to decode moving images in real time with one decoder. Therefore, a method of parallelizing the decoding process using a plurality of decoders to reduce the processing load per decoder and to achieve real-time processing is being studied.

Also, encoded data is multiplexed based on a multiplexing scheme such as MPEG-2 TS (Transport Stream) or MMT (MPEG Media Transport) before being transmitted. For example, Non-Patent Document 1 discloses a technique for transmitting encoded media data in units of packets according to MMT.

Information technology - High efficiency coding and media delivery in heterogeneous environment - Part 1: MPEG media transport (MMT), ISO/IEC DIS 23008-1

MMT is a system that supports hybrid distribution using broadcasting and communication, is capable of transmitting media in MP4 format, and has various functions. However, if MMT is used for data transmission when reproducing a broadcast stream in a receiving device, the configuration of the transmitting device and the receiving device becomes complicated due to MMT having more functions than necessary, and the amount of processing increases. may be lost. In addition, the transmission band may be wasted by transmitting unnecessary data.

The present invention provides a transmitting device and a receiving device that can simplify the configuration of the device and reduce the processing amount of the device when data is transmitted using a method such as MMT.

To achieve the above object, a transmission method according to an aspect of the present invention provides sample data, which is data obtained by encoding a video signal or an audio signal, and reconfigures the sample data as an MP4 format file on the receiving side. a adding step of adding header information including MP4 configuration information for performing, the content of which differs depending on whether or not the presentation time of the sample data is set; and the header information is added. and a transmitting step of transmitting the sample data, and in the adding step, if metadata corresponding to the sample data is not transmitted in the transmitting step, depending on whether the presentation time of the sample data is set. Then, header information that does not contain the MP4 configuration information is added to the sample data.

Further, a receiving method according to an aspect of the present invention provides sample data that is data in which a video signal or an audio signal is encoded, and MP4 configuration information for reconstructing the sample data as an MP4 format file. a receiving step of receiving sample data to which header information not included is added; and a case where metadata corresponding to the sample data is not received in the receiving step, and the presentation time of the sample data is determined. a decoding step of decoding said sample data without using said MP4 configuration information.

In addition, these general or specific aspects may be realized by a system, method, integrated circuit, computer program, or a recording medium such as a computer-readable CD-ROM. and any combination of recording media.

INDUSTRIAL APPLICABILITY The present invention simplifies the configuration of a device and reduces the processing amount of the device when data is transmitted using a method such as MMT.

FIG. 1 is a diagram illustrating an example of dividing a picture into slice segments. FIG. 2 is a diagram showing an example of a PES packet sequence in which picture data is stored. FIG. 3 is a diagram showing an example of picture division according to the first embodiment. FIG. 4 is a diagram illustrating an example of picture division according to a comparative example of Embodiment 1. FIG. 5 is a diagram showing an example of data of an access unit according to Embodiment 1. FIG. 6 is a block diagram of a transmitting device according to Embodiment 1. FIG. 7 is a block diagram of a receiving device according to Embodiment 1. FIG. 8 is a diagram showing an example of an MMT packet according to Embodiment 1. FIG. 9 is a diagram showing another example of an MMT packet according to Embodiment 1. FIG. 10 is a diagram showing an example of data input to each decoding unit according to Embodiment 1. FIG. 11 is a diagram showing an example of an MMT packet and header information according to Embodiment 1. FIG. 12 is a diagram showing another example of data input to each decoding unit according to Embodiment 1. FIG. 13 is a diagram illustrating an example of picture division according to Embodiment 1. FIG. 14 is a flowchart of a transmission method according to Embodiment 1. FIG. 15 is a block diagram of a receiving device according to Embodiment 1. FIG. 16 is a flowchart of a reception method according to Embodiment 1. FIG. 17 is a diagram showing an example of an MMT packet and header information according to Embodiment 1. FIG. 18 is a diagram showing an example of an MMT packet and header information according to Embodiment 1. FIG. FIG. 19 is a diagram showing the configuration of the MPU. FIG. 20 is a diagram showing the configuration of MF metadata. FIG. 21 is a diagram for explaining the data transmission order. FIG. 22 is a diagram showing an example of a decoding method without using header information. 23 is a block diagram of a transmitting device according to Embodiment 2. FIG. 24 is a flowchart of a transmission method according to Embodiment 2. FIG. 25 is a block diagram of a receiving device according to Embodiment 2. FIG. FIG. 26 is a flowchart of operations for specifying the MPU head position and NAL unit position. FIG. 27 is a flowchart of operations for obtaining initialization information based on the transmission order type and decoding media data based on the initialization information. FIG. 28 is a flow chart of the operation of the receiving device when the low-delay presentation mode is provided. FIG. 29 is a diagram illustrating an example of the transmission order of MMT packets when auxiliary data is transmitted. FIG. 30 is a diagram for explaining an example in which a transmitting device generates auxiliary data based on a moof configuration. FIG. 31 is a diagram for explaining reception of auxiliary data. FIG. 32 is a flow chart of a reception operation using auxiliary data. FIG. 33 is a diagram showing the configuration of an MPU made up of multiple movie fragments. FIG. 34 is a diagram for explaining the transmission order of MMT packets when the MPU having the configuration of FIG. 33 is transmitted. FIG. 35 is a first diagram for explaining an operation example of the receiving device when one MPU is composed of a plurality of movie fragments. FIG. 36 is a second diagram for explaining an operation example of the receiving device when one MPU is composed of a plurality of movie fragments. FIG. 37 is a flow chart of the operation of the receiving method described in FIGS. FIG. 38 is a diagram showing a case where non-VCL NAL units are individually treated as data units and aggregated. FIG. 39 is a diagram showing a case where non-VCL NAL units are collectively used as data units. FIG. 40 is a flow chart of the operation of the receiver when packet loss occurs. FIG. 41 is a flow chart of the receiving operation when the MPU is divided into multiple movie fragments. FIG. 42 is a diagram showing an example of a picture prediction structure for each TemporalId when temporal scalability is realized. FIG. 43 is a diagram showing the relationship between the decoding time (DTS) and the display time (PTS) in each picture in FIG. FIG. 44 is a diagram showing an example of a prediction structure of pictures that require picture delay processing and reorder processing. FIG. 45 is a diagram showing an example in which an MPU configured in MP4 format is divided into a plurality of movie fragments and stored in MMTP payloads and MMTP packets. FIG. 46 is a diagram for explaining a method of calculating PTS and DTS and problems. FIG. 47 is a flowchart of the reception operation when DTS is calculated using information for DTS calculation. FIG. 48 is a diagram for explaining a method of storing a data unit in a payload in MMT. 49 is an operation flow of a transmitting device according to Embodiment 3. FIG. 50 is an operation flow of the receiving device according to Embodiment 3. FIG. 51 is a diagram showing an example of a specific configuration of a transmitting apparatus according to Embodiment 3. FIG. 52 is a diagram showing an example of a specific configuration of a receiving device according to Embodiment 3. FIG.

In a transmission method according to an aspect of the present invention, sample data, which is data obtained by encoding a video signal or an audio signal, is provided with MP4 configuration information for reconstructing the sample data as an MP4 format file on the receiving side. a adding step of adding header information including MP4 configuration information whose contents differ depending on whether or not the presentation time of the sample data is set; and a transmitting step of transmitting the sample data to which the header information is added. and, in the adding step, if the metadata corresponding to the sample data is not transmitted in the transmitting step, the MP4 configuration information is included depending on whether the presentation time of the sample data is set. Header information that does not exist is added to the sample data.

Such a transmission method can simplify the configuration of the device and reduce the processing amount of the device when data is transmitted using a method such as MMT.

Further, in the adding step, if the metadata corresponding to the sample data is not transmitted in the transmitting step, and if the presentation time of the sample data is determined, the header information not including the MP4 configuration information is transmitted. It is added to the sample data, and if the presentation time of the sample data is not determined, header information including the MP4 configuration information is added to the sample data.

Further, in the transmitting step, the sample data to which the header information is attached may be packetized by an MMT (MPEG Media Transport) method and transmitted.

Further, the header information attached to the sample data whose presentation time is determined includes at least one of movie_fragment_sequence_number, sample_number, offset, priority, and dependency_counter in the MMTP (MMT Protocol) payload as the MP4 configuration information. , item_id in the MP4MMTP payload may be included as the MP4 configuration information in the header information added to the sample data for which the presentation time is not defined.

Further, the sample data for which the presentation time is set is timed-MFU (Movie
Fragment Unit) and the sample data for which presentation time is not defined may be non-timed-MFU.

Also, the metadata may include MPU (Media Processing Unit) metadata and movie fragment metadata.

Further, a receiving method according to an aspect of the present invention provides sample data that is data in which a video signal or an audio signal is encoded, and MP4 configuration information for reconstructing the sample data as an MP4 format file. a receiving step of receiving sample data to which header information not included is added; and a case where metadata corresponding to the sample data is not received in the receiving step, and the presentation time of the sample data is determined. a decoding step of decoding said sample data without using said MP4 configuration information.

Such a reception method can simplify the configuration of the device and reduce the processing amount of the device when data is transmitted using a method such as MMT.

A transmission device according to an aspect of the present invention provides sample data, which is data obtained by encoding a video signal or an audio signal, and MP4 configuration information for reconstructing the sample data as an MP4 format file on the receiving side. a adding unit that adds header information including MP4 configuration information whose contents differ depending on whether the presentation time of the sample data is set or not; and a transmitting unit that transmits the sample data to which the header information is added. wherein, when the transmitting unit does not transmit the metadata corresponding to the sample data, the adding unit includes the MP4 configuration information depending on whether the presentation time of the sample data is set. Header information that does not exist is added to the sample data.

A receiving device according to an aspect of the present invention provides sample data that is data obtained by encoding a video signal or an audio signal, and does not include MP4 configuration information for reconstructing the sample data as an MP4 format file. a receiving unit that receives sample data to which header information is attached; and when metadata corresponding to the sample data is not received by the receiving unit and the presentation time of the sample data is determined. and a decoding unit for decoding the sample data without using the MP4 configuration information.

In addition, these general or specific aspects may be realized by a system, method, integrated circuit, computer program, or a recording medium such as a computer-readable CD-ROM. It may be implemented in any combination of computer programs and recording media.

Hereinafter, embodiments will be specifically described with reference to the drawings.

It should be noted that the embodiments described below are all comprehensive or specific examples. Numerical values, shapes, materials, components, arrangement positions and connection forms of components, steps, order of steps, and the like shown in the following embodiments are examples and are not intended to limit the present invention. In addition, among the constituent elements in the following embodiments, constituent elements that are not described in independent claims representing the highest concept will be described as arbitrary constituent elements.

(Knowledge on which the present invention is based)
2. Description of the Related Art In recent years, the resolution of displays for TVs, smartphones, tablet terminals, and the like has been increasing. Especially in broadcasting in Japan, 8K4K (resolution is 8K×4K) service is scheduled in 2020. Since it is difficult to decode super-high-resolution video such as 8K4K in real time with a single decoder, a method of decoding in parallel using a plurality of decoders is being studied.

Since coded data is multiplexed and transmitted based on a multiplexing method such as MPEG-2 TS or MMT, the receiving device needs to separate the coded video data from the multiplexed data prior to decoding. . The process of separating encoded data from multiplexed data is hereinafter referred to as demultiplexing.

When parallelizing the decoding process, it is necessary to distribute the encoded data to be decoded to each decoder. When distributing the encoded data, it is necessary to analyze the encoded data itself, and the processing load related to the analysis is particularly large for content such as 8K because the bit rate is extremely high. Therefore, there is a problem that the inverse multiplexing part becomes a bottleneck and reproduction in real time cannot be performed.

By the way, H.264 standardized by MPEG and ITU. 264 and H. In a video coding method such as H.265, a transmitting device divides a picture into a plurality of regions called slices or slice segments, and encodes each divided region so that it can be decoded independently. Thus, for example, H. In the case of H.265, a receiver that receives broadcast separates the data for each slice segment from the received data and outputs the data for each slice segment to separate decoders, thereby realizing parallel decoding processing. .

FIG. 1 is a diagram showing an example of dividing one picture into four slice segments in HEVC. For example, the receiving device has four decoders, each decoding one of the four slice segments.

In conventional broadcasting, a transmitting device stores one picture (access unit in the MPEG system standard) in one PES packet, and multiplexes the PES packet into a TS packet train. Therefore, the receiving device separates the payload of the PES packet, analyzes the data of the access unit stored in the payload, separates each slice segment, and decodes the data of each separated slice segment. I had to output to

However, the present inventor has found that there is a problem that it is difficult to perform this processing in real time because the amount of processing required to analyze access unit data and separate slice segments is large.

FIG. 2 is a diagram showing an example in which picture data divided into slice segments is stored in the payload of a PES packet.

As shown in FIG. 2, for example, data of multiple slice segments (slice segments 1 to 4) are stored in the payload of one PES packet. Also, the PES packets are multiplexed into the TS packet train.

(Embodiment 1)
In the following description, H.264 is used as a video encoding method. 265 will be described as an example. This embodiment can also be applied when using other coding schemes such as H.264.

FIG. 3 is a diagram showing an example of dividing an access unit (picture) into division units according to the present embodiment. The access unit is H.264. A feature called tiles, introduced by H.265, bisects each horizontally and vertically into a total of four tiles. Also, slice segments and tiles are associated one-to-one.

The reason for such halving in the horizontal and vertical directions will be described. First, decoding generally requires a line memory for storing one horizontal line of data. However, when the resolution is super high such as 8K4K, the size of the line memory increases because the size in the horizontal direction increases. In implementation of the receiver, it is desirable to be able to reduce the size of the line memory. Vertical partitioning is required to reduce the size of the line memory. Vertical partitioning requires a data structure called a tile. For these reasons tiles are used.

On the other hand, since images generally have high correlation in the horizontal direction, the ability to refer to a wider range in the horizontal direction improves the coding efficiency. Therefore, from the viewpoint of coding efficiency, it is desirable to divide the access unit horizontally.

By dividing the access unit into two halves in the horizontal and vertical directions, these two characteristics are compatible, and both implementation and coding efficiency can be considered. If a single decoder can decode a 4K2K moving image in real time, the 8K4K image is divided into four equal parts, and each slice segment is divided into 4K2K, so that the receiving device can decode 8K4K images in real time.

Next, the reason why tiles obtained by dividing access units in the horizontal and vertical directions and slice segments are associated one-to-one will be described. H. In H.265, an access unit is composed of a plurality of units called NAL (Network Adaptation Layer) units.

The payload of the NAL unit includes an access unit delimiter that indicates the start position of an access unit, an SPS (Sequence Parameter Set) that is initialization information commonly used in decoding for each sequence, and an initialization information commonly used in a picture when decoding. PPS (Picture Parameter Set) which is encoding information, SEI (Supplemental Enhancement Information) which is unnecessary for the decoding process itself but is necessary for processing and displaying the decoding result, and encoded data of slice segments, etc. to store The NAL unit header contains type information to identify the data stored in the payload.

Here, the transmission device converts encoded data into MPEG-2 TS, MMT (MPEG Media Transport), MPEG DASH (Dynamic Adaptive
When multiplexing using a multiplexing format such as Streaming over HTTP) or RTP (Real-time Transport Protocol), the basic unit can be set to the NAL unit. In order to store one slice segment in one NAL unit, it is desirable to divide the access unit into slice segments when dividing the access unit into regions. For this reason, the transmitter associates tiles with slice segments on a one-to-one basis.

Note that, as shown in FIG. 4, the transmitting device can collectively set tiles 1 to 4 as one slice segment. However, in this case, all tiles are stored in one NAL unit, making it difficult for the receiving device to separate the tiles in the multiplexing layer.

Slice segments include independent slice segments that can be decoded independently and reference slice segments that refer to independent slice segments. Here, a case where independent slice segments are used will be described.

FIG. 5 is a diagram showing an example of access unit data divided so that the boundaries between tiles and slice segments are aligned as shown in FIG. The access unit data consists of the NAL unit storing the access unit delimiter placed at the beginning, the SPS, PPS, and SEI NAL units placed after that, and the tiles 1 to 4 placed after that. and data of the slice segment in which the data is stored. Access unit data may not include some or all of the NAL units of SPS, PPS, and SEI.

Next, the configuration of transmitting apparatus 100 according to this embodiment will be described. FIG. 6 is a block diagram showing a configuration example of transmitting apparatus 100 according to this embodiment. This transmitting apparatus 100 comprises an encoding section 101 , a multiplexing section 102 , a modulating section 103 and a transmitting section 104 .

The encoding unit 101 converts an input image into H.264, for example. The encoded data is generated by encoding according to H.265. Also, the encoding unit 101 divides an access unit into four slice segments (tiles) and encodes each slice segment, as shown in FIG. 3, for example.

Multiplexing section 102 multiplexes the encoded data generated by encoding section 101 . Modulation section 103 modulates the data obtained by multiplexing. The transmitting section 104 transmits the modulated data as a broadcast signal.

Next, the configuration of receiving apparatus 200 according to this embodiment will be described. FIG. 7 is a block diagram showing a configuration example of receiving apparatus 200 according to this embodiment. This receiver 200 comprises a tuner 201 , a demodulator 202 , a demultiplexer 203 , a plurality of decoders 204A to 204D, and a display 205 .

Tuner 201 receives a broadcast signal. Demodulator 202 demodulates the received broadcast signal. The demodulated data is input to demultiplexing section 203 .

Demultiplexing section 203 demultiplexes the demodulated data into division units, and outputs the data for each division unit to decoding sections 204A to 204D. Here, a division unit is a division area obtained by dividing an access unit. It is a slice segment in H.265. Also here, the 8K4K image is split into four 4K2K images. Therefore, there are four decoding units 204A-204D.

A plurality of decoding units 204A to 204D operate in synchronization with each other based on a predetermined reference clock. Each decoding unit decodes the encoded data in division units according to the DTS (Decoding Time Stamp) of the access unit, and outputs the decoding result to the display unit 205 .

The display unit 205 generates an 8K4K output image by integrating multiple decoding results output from the multiple decoding units 204A to 204D. The display unit 205 displays the generated output image according to the separately acquired PTS (Presentation Time Stamp) of the access unit. Note that when integrating the decoding results, the display unit 205 performs filter processing such as a deblocking filter in a boundary area between division units adjacent to each other, such as a boundary between tiles, so that the boundary becomes visually inconspicuous. good too.

In the above description, the transmission device 100 and the reception device 200 that transmit or receive broadcasts have been described as examples, but content may be transmitted and received via a communication network. When the receiving device 200 receives content via a communication network, the receiving device 200 separates multiplexed data from IP packets received via a network such as Ethernet.

In broadcasting, the transmission path delay from when a broadcast signal is transmitted until it reaches the receiver 200 is constant. On the other hand, in a communication network such as the Internet, due to the effects of congestion, the transmission path delay until data transmitted from a server reaches the receiving device 200 is not constant. Therefore, the receiving apparatus 200 often does not perform strictly synchronous reproduction based on a reference clock such as PCR in broadcast MPEG-2 TS. Therefore, the receiving device 200 may display the 8K4K output image on the display unit according to the PTS without strictly synchronizing the decoding units.

Also, due to congestion of the communication network, etc., there is a case where decoding processing for all division units has not been completed at the time indicated by the PTS of the access unit. In this case, the receiving device 200 skips the display of the access unit, or delays the display until the decoding of at least four division units is completed and the generation of the 8K4K image is completed.

Note that content may be transmitted and received using both broadcasting and communication. This method can also be applied when reproducing multiplexed data stored in a recording medium such as a hard disk or memory.

Next, a method of multiplexing access units divided into slice segments when MMT is used as the multiplexing method will be described.

FIG. 8 is a diagram showing an example of packetizing HEVC access unit data into MMT. Although SPS, PPS, SEI, etc. do not necessarily have to be included in the access unit, the case where they exist is exemplified here.

The NAL units, such as the access unit delimiter, SPS, PPS, and SEI, which are arranged before the head slice segment in the access unit, are collectively stored in the MMT packet #1. Subsequent slice segments are stored in separate MMT packets for each slice segment.

Note that, as shown in FIG. 9, the NAL unit arranged before the top slice segment in the access unit may be stored in the same MMT packet as the top slice segment.

Also, when NAL units such as End-of-Sequence or End-of-Bitstream, which indicate the end of a sequence or stream, are added after the final slice segment, these are included in the same MMT packet as the final slice segment. Stored. However, since NAL units such as End-of-Sequence or End-of-Bitstream are inserted at the end point of the decoding process, or at the connection point of two streams, the receiving device 200 inserts these NAL units. , can be easily obtained at the multiplexing layer. In this case, these NAL units may be stored in MMT packets separate from slice segments. This allows receiving apparatus 200 to easily separate these NAL units in the multiplexing layer.

Note that TS, DASH, RTP, or the like may be used as the multiplexing method. In these schemes as well, transmitting apparatus 100 stores different slice segments in different packets. This ensures that receiving apparatus 200 can separate slice segments in the multiplexing layer.

For example, when TS is used, encoded data is PES-packetized in units of slice segments. When RTP is used, encoded data is RTP-packetized in units of slice segments. Also in these cases, the NAL unit arranged before the slice segment and the slice segment may be packetized separately like MMT packet #1 shown in FIG.

When TS is used, transmitting apparatus 100 indicates the unit of data stored in a PES packet by using data alignment descriptors. Also, since DASH is a method of downloading MP4 format data units called segments using HTTP or the like, the transmitting apparatus 100 does not packetize encoded data for transmission. Therefore, transmitting apparatus 100 creates subsamples in units of slice segments and stores information indicating the storage positions of subsamples in the MP4 header so that receiving apparatus 200 can detect slice segments in the multiplexing layer of MP4. You may

MMT packetization of slice segments is described in detail below.

As shown in FIG. 8, by packetizing encoded data, data commonly referred to when decoding all slice segments in an access unit such as SPS and PPS is stored in MMT packet #1. In this case, receiving device 200 concatenates the payload data of MMT packet #1 and the data of each slice segment, and outputs the obtained data to the decoding unit. In this way, receiving apparatus 200 can easily generate input data to the decoding unit by concatenating payloads of a plurality of MMT packets.

FIG. 10 is a diagram showing an example in which input data to decoding units 204A to 204D are generated from the MMT packets shown in FIG. Demultiplexing section 203 concatenates the payload data of MMT packet #1 and MMT packet #2, thereby generating data necessary for decoding section 204A to decode slice segment 1. FIG. Demultiplexing section 203 similarly generates input data for decoding section 204B to decoding section 204D. That is, demultiplexing section 203 generates input data for decoding section 204B by concatenating the payload data of MMT packet #1 and MMT packet #3. Demultiplexing section 203 generates input data for decoding section 204C by concatenating the payload data of MMT packet #1 and MMT packet #4. Demultiplexing section 203 generates input data for decoding section 204D by concatenating the payload data of MMT packet #1 and MMT packet #5.

Note that the demultiplexing unit 203 removes NAL units that are not necessary for decoding processing, such as access unit delimiters and SEI, from the payload data of MMT packet #1, and removes only the NAL units of SPS and PPS that are necessary for decoding processing. may be separated and added to the slice segment data.

When the encoded data is packetized as shown in FIG. 9, demultiplexing section 203 outputs MMT packet #1 including the leading data of the access unit in the multiplexing layer to first decoding section 204A. . In addition, the demultiplexing unit 203 analyzes the MMT packet including the leading data of the access unit in the multiplexing layer, separates the SPS and PPS NAL units, and divides the separated SPS and PPS NAL units into the second and subsequent slices. Input data for each of the second and subsequent decoding units is generated by adding it to each of the segment data.

Furthermore, receiving device 200 uses the information included in the header of the MMT packet to determine the type of data stored in the MMT payload and the slice segment in the access unit when the slice segment is stored in the payload. It is desirable to be able to identify the index number. Here, the type of data includes pre-slice segment data (the NAL units arranged before the head slice segment in the access unit are collectively called in this way), and slice segment data. Either. When storing an MPU fragmented unit such as a slice segment in an MMT packet, a mode for storing an MFU (Media Fragment Unit) is used. When using this mode, transmitting apparatus 100 converts Data Unit, which is the basic unit of data in MFU, to samples (data units in MMT and corresponds to access units) or subsamples (samples). divided unit) can be set.

At this time, the header of the MMT packet includes a field called Fragmentation indicator and a field called Fragment counter.

Fragmentation indicator indicates whether the data stored in the payload of the MMT packet is a fragmented Data unit, and if the fragment is fragmented, the fragment is the first or last fragment in the Data unit, or , indicates whether the fragment is neither the first nor the last. In other words, the fragmentation indicator included in the header of a certain packet is (1) only the packet is included in the Data unit, which is a basic data unit, (2) the Data unit is divided and stored in a plurality of packets, and (3) the data unit is divided and stored in a plurality of packets, and the packet is a packet other than the first and last packets of the data unit; and (4) Identification information indicating whether a Data unit is divided and stored in a plurality of packets, or whether the packet is the last packet of the Data unit.

Fragment counter is an index number indicating which fragment in the data unit the data stored in the MMT packet corresponds to.

Therefore, transmitting apparatus 100 sets samples in MMT to Data unit, and sets pre-slice segment data and each slice segment in units of fragment of Data unit, respectively, so that receiving apparatus 200 can generate the header of the MMT packet. can be used to identify the type of data stored in the payload. That is, the demultiplexing unit 203 can refer to the header of the MMT packet to generate input data to each of the decoding units 204A to 204D.

FIG. 11 is a diagram showing an example in which the sample is set in the Data unit, and the pre-slice segment data and the slice segment are packetized as fragments of the Data unit.

The slice segment pre-data and the slice segment are divided into five fragments from fragment #1 to fragment #5. Each fragment is stored in a separate MMT packet. At this time, the values of the fragmentation indicator and the fragment counter included in the header of the MMT packet are as illustrated.

For example, the Fragment indicator is a binary 2-bit value. The fragment indicator of MMT packet #1, which is the head of the data unit, the fragment indicator of MMT packet #5, which is the last, and the fragment indicators of MMT packet #2 to MMT packet #4, which are packets in between, are different. set to the value Specifically, the Fragment indicator of MMT packet #1, which is the head of the Data unit, is set to 01, the Fragment indicator of MMT packet #5, which is the end, is set to 11, and from MMT packet #2, which is the packet between Fragment indicators up to MMT packet #4 are set to 10. Note that the Fragment indicator is set to 00 when the Data unit contains only one MMT packet.

In addition, the fragment counter is 4, which is a value obtained by subtracting 1 from 5, which is the total number of fragments, in MMT packet #1, decreases by 1 in subsequent packets, and is 0 in the last MMT packet #5. be.

Therefore, receiving apparatus 200 can identify an MMT packet storing pre-slice segment data using either a Fragment indicator or a Fragment counter. Also, receiving device 200 can identify the MMT packet storing the N-th slice segment by referring to the Fragment counter.

The header of the MMT packet separately includes the sequence number within the MPU of the Movie Fragment to which the Data unit belongs, the sequence number of the MPU itself, and the sequence number within the Movie Fragment of the sample to which the Data unit belongs. The demultiplexing unit 203 can uniquely determine the sample to which the Data unit belongs by referring to these.

Furthermore, since the demultiplexing unit 203 can determine the index number of the fragment in the data unit from the fragment counter or the like, even when packet loss occurs, the slice segment stored in the fragment can be uniquely identified. For example, even if fragment #4 shown in FIG. 11 cannot be acquired due to packet loss, demultiplexing section 203 knows that the fragment received next to fragment #3 is fragment #5. can be correctly output to decoding section 204D instead of decoding section 204C.

Note that when a transmission line that guarantees that no packet loss occurs is used, the demultiplexing unit 203 refers to the header of the MMT packet to determine the type of data stored in the MMT packet, or the slice segment. It suffices to periodically process arriving packets without determining the index number of the . For example, when an access unit is transmitted by pre-slice data and a total of 5 MMT packets of 4 slice segments, after determining the pre-slice data of the access unit to start decoding, the receiving apparatus 200 , by sequentially processing the received MMT packets, the pre-slice data and the data of the four slice segments can be obtained in order.

Modifications of packetization will be described below.

The slice segment does not necessarily have to be obtained by dividing the plane of the access unit in both the horizontal direction and the vertical direction. As shown in FIG. 1, the slice segment may be obtained by dividing the access unit only in the horizontal direction. However, it may be divided only in the vertical direction.

Also, if the access unit is divided only horizontally, tiles need not be used.

Also, the number of in-plane divisions in the access unit is arbitrary and is not limited to four. However, the region sizes of slice segments and tiles conform to H.264. It must be at least the lower bound of an encoding standard such as H.265.

Transmitting apparatus 100 may store identification information indicating an intra-plane division method in an access unit in an MMT message, a TS descriptor, or the like. For example, information may be stored that indicates the number of divisions in the plane in the horizontal direction and the number of divisions in the vertical direction. Alternatively, identification information unique to the division method, such as halving in the horizontal direction and the vertical direction as shown in FIG. 3, or dividing into 4 equal parts in the horizontal direction as shown in FIG. may be assigned. For example, when the access unit is divided as shown in FIG. 3, the identification information indicates mode 1, and when the access unit is divided as shown in FIG. 1, the identification information indicates mode 1. .

In addition, the multiplexing layer may include information indicating restrictions on coding conditions related to the intra-plane division method. For example, information indicating that one slice segment consists of one tile may be used. Alternatively, reference blocks for motion compensation when decoding slice segments or tiles are limited to slice segments or tiles at the same position in the screen, or are limited to blocks within a predetermined range in adjacent slice segments. Information indicating such as that may be used.

Also, transmitting apparatus 100 may switch whether to divide an access unit into a plurality of slice segments according to the resolution of a moving image. For example, if the moving image to be processed has a resolution of 4K2K, the transmission device 100 does not perform intra-plane division, and if the moving image to be processed has a resolution of 8K4K, the access unit may be divided into four. good. By prescribing the division method in the case of 8K4K moving images, the receiving apparatus 200 obtains the resolution of the received moving image to determine whether or not to divide the screen and the division method, and decode. You can switch between actions.

In addition, receiving device 200 can detect the presence or absence of in-plane division by referring to the header of the MMT packet. For example, if the access unit is not fragmented, if the MMT Data unit is set to sample, the Data unit is not fragmented. Therefore, receiving apparatus 200 can determine that the access unit is not divided when the value of Fragment counter included in the header of the MMT packet is always zero. Alternatively, receiving device 200 may detect whether the value of the fragmentation indicator is always 01. Receiving apparatus 200 can determine that the access unit is not fragmented even when the value of the fragmentation indicator is always 01.

In addition, receiving apparatus 200 can cope with cases where the number of in-plane divisions in an access unit does not match the number of decoding units. For example, when receiving device 200 includes two decoding units 204A and 204B capable of decoding 8K2K encoded data in real time, demultiplexing unit 203 sends 8K4K encoded data to decoding unit 204A. outputs two of the four slice segments that make up the .

FIG. 12 is a diagram showing an operation example when MMT packetized data as shown in FIG. 8 is input to two decoding units 204A and 204B. Here, it is desirable that receiving apparatus 200 can integrate and output the decoding results of decoding sections 204A and 204B as they are. Therefore, demultiplexing section 203 selects slice segments to be output to decoding sections 204A and 204B so that the decoding results of decoding sections 204A and 204B are spatially continuous.

Also, the demultiplexing unit 203 may select a decoding unit to be used according to the resolution or frame rate of the encoded data of the moving image. For example, when receiving device 200 includes four 4K2K decoding units, if the resolution of the input image is 8K4K, receiving device 200 performs decoding processing using all four decoding units. Also, if the resolution of the input image is 4K2K, the reception device 200 performs decoding processing using only one decoding unit. Alternatively, the demultiplexing unit 203, even if the plane is divided into four, if 8K4K can be decoded in real time by a single decoding unit, all the division units are integrated into one decoding unit output to

Furthermore, receiving apparatus 200 may determine the decoding unit to be used in consideration of the frame rate. For example, if the receiving apparatus 200 includes two decoding units whose upper limit of the frame rate that can be decoded in real time is 60 fps when the resolution is 8K4K, there is a case where 8K4K encoded data of 120 fps is input. be. At this time, assuming that the plane is composed of four division units, slice segments 1 and 2 are input to the decoding unit 204A, and slice segments 3 and 4 are input as in the example of FIG. It is input to the decoding unit 204B. Since each of the decoding units 204A and 204B can decode in real time up to 120 fps for 8K2K (half the resolution of 8K4K), decoding processing is performed by these two decoding units 204A and 204B.

Also, even if the resolution and frame rate are the same, the profile or level in the encoding system, or the H.264 standard may be used. 264 or H.264. If the encoding method itself such as H.265 is different, the amount of processing will be different. Therefore, receiving apparatus 200 may select a decoding unit to be used based on these pieces of information. In addition, if the receiving apparatus 200 cannot decode all the encoded data received by broadcasting or communication, or if all the slice segments or tiles that make up the region selected by the user cannot be decoded, the decoding unit may automatically determine decodable slice segments or tiles within the processing range of . Alternatively, receiving device 200 may provide a user interface for the user to select a region to decode. At this time, receiving device 200 may display a warning message indicating that all regions cannot be decoded, or may display information indicating the number of decodable regions, slice segments, or tiles.

Moreover, the above method can also be applied when MMT packets storing slice segments of the same encoded data are transmitted and received using a plurality of transmission paths such as broadcasting and communication.

In addition, transmitting apparatus 100 may perform encoding such that the regions of each slice segment overlap in order to obscure boundaries between division units. In the example shown in FIG. 13, an 8K4K picture is divided into four slice segments 1-4. Each of slice segments 1-3 is, for example, 8K×1.1K, and slice segment 4 is 8K×1K. Adjacent slice segments also overlap each other. By doing this, the motion compensation at the time of encoding can be executed efficiently at the boundaries of the four divisions indicated by the dotted lines, so that the image quality of the boundary portions is improved. In this way, deterioration of image quality in the boundary portion is reduced.

In this case, the display unit 205 cuts out an 8K×1K area from an 8K×1.1K area, and integrates the obtained areas. In addition, transmitting apparatus 100 includes information indicating whether or not slice segments are encoded with overlap and the range of overlap in the multiplexing layer or encoded data, and separately transmits the information. good too.

A similar technique can be applied when tiles are used.

The operation flow of the transmission device 100 will be described below. FIG. 14 is a flow chart showing an operation example of the transmission device 100 .

First, the encoding unit 101 divides a picture (access unit) into a plurality of slice segments (tiles), which are a plurality of regions (S101). Next, the encoding unit 101 generates encoded data corresponding to each of the plurality of slice segments by encoding each of the plurality of slice segments so that they can be independently decoded (S102). Note that the encoding unit 101 may encode a plurality of slice segments by a single encoding unit, or may perform parallel processing by a plurality of encoding units.

Next, the multiplexing unit 102 multiplexes the multiple encoded data by storing the multiple encoded data generated by the encoding unit 101 in multiple MMT packets (S103). Specifically, as shown in FIGS. 8 and 9, multiplexing section 102 combines a plurality of encoded data into a plurality of slice segments so that encoded data corresponding to different slice segments are not stored in one MMT packet. Store in MMT packet. Further, as shown in FIG. 8, multiplexing section 102 sets control information commonly used for all decoding units in a picture to a plurality of MMT packets #2 to # in which a plurality of encoded data are stored. 5 is stored in MMT packet #1 different from 5. Here, the control information includes at least one of access unit delimiter, SPS, PPS and SEI.

Note that multiplexing section 102 may store the control information in the same MMT packet as one of a plurality of MMT packets in which a plurality of encoded data are stored. For example, as shown in FIG. 9, multiplexing section 102 stores control information in the first MMT packet (MMT packet #1 in FIG. 9) among multiple MMT packets storing multiple pieces of encoded data. You may

Finally, transmitting device 100 transmits a plurality of MMT packets. Specifically, the modulation section 103 modulates the data obtained by multiplexing, and the transmission section 104 transmits the modulated data (S104).

FIG. 15 is a block diagram showing a configuration example of receiving apparatus 200, and is a diagram showing in detail the configuration of demultiplexing section 203 shown in FIG. 7 and its subsequent stage. As shown in FIG. 15 , receiving device 200 further includes decoding instruction section 206 . Demultiplexing section 203 also includes type determining section 211 , control information acquiring section 212 , slice information acquiring section 213 , and decoded data generating section 214 .

The operation flow of the receiving device 200 will be described below. FIG. 16 is a flow chart showing an operation example of the receiving device 200 . Here, the operation for one access unit is shown. When the decryption processing of a plurality of access units is executed, the processing of this flowchart is repeated.

First, the receiving device 200 receives, for example, a plurality of packets (MMT packets) generated by the transmitting device 100 (S201).

Next, the type determination unit 211 acquires the type of encoded data stored in the received packet by analyzing the header of the received packet (S202).

Next, the type determination unit 211 determines whether the data stored in the received packet is pre-slice segment data or slice segment data based on the type of the acquired encoded data (S203). .

If the data stored in the received packet is pre-slice segment data (Yes in S203), the control information acquisition unit 212 acquires pre-slice segment data of the access unit to be processed from the payload of the received packet, The pre-segment data is stored in memory (S204).

On the other hand, if the data stored in the received packet is slice segment data (No in S203), receiving apparatus 200 uses the header information of the received packet to determine whether the data stored in the received packet is plural. It is determined which region of the region the coded data belongs to. Specifically, the slice information acquisition unit 213 acquires the index number Idx of the slice segment stored in the received packet by analyzing the header of the received packet (S205). Specifically, the index number Idx is the index number in the Movie Fragment of the access unit (sample in MMT).

Note that the process of step S205 may be collectively performed in step S202.

Next, the decoded data generation unit 214 determines a decoding unit that decodes the slice segment (S206). Specifically, the index number Idx and a plurality of decoding units are associated in advance, and the decoded data generation unit 214 causes the decoding unit corresponding to the index number Idx acquired in step S205 to decode the slice segment. Decide which decoder to use.

12, the decoded data generation unit 214 uses the resolution of the access unit (picture), the method of dividing the access unit into multiple slice segments (tiles), and the multiple A decoder for decoding the slice segment may be determined based on at least one of the processing capabilities of the decoder. For example, the decoded data generator 214 determines the access unit division method based on the identification information in the MMT message or the descriptor such as the TS section.

Next, the decoded data generation unit 214 generates control information commonly used for all decoding units in a picture included in any of the plurality of packets, and each of the plurality of encoded data of the plurality of slice segments. are combined to generate a plurality of input data (combined data) to be input to a plurality of decoding units. Specifically, the decoded data generation unit 214 acquires slice segment data from the payload of the received packet. The decoded data generation unit 214 combines the pre-slice segment data stored in the memory in step S204 and the acquired slice segment data to generate the input data to the decoding unit determined in step S206. (S207).

After step S204 or S207, if the data of the received packet is not the final data of the access unit (No in S208), the processing after step S201 is performed again. That is, the above processing is repeated until input data to the plurality of decoding units 204A to 204D corresponding to all slice segments included in the access unit are generated.

Note that the timing at which packets are received is not limited to the timing shown in FIG. 16, and a plurality of packets may be received in advance or sequentially and stored in a memory or the like.

On the other hand, if the data of the received packet is the final data of the access unit (Yes in S208), the decoding instruction unit 206 outputs the multiple pieces of input data generated in step S207 to the corresponding decoding units 204A-204D. (S209).

Next, the multiple decoding units 204A to 204D generate multiple decoded images by decoding multiple input data in parallel according to the DTS of the access unit (S210).

Finally, the display unit 205 generates a display image by combining the multiple decoded images generated by the multiple decoding units 204A to 204D, and displays the display image according to the PTS of the access unit (S211).

Receiving apparatus 200 acquires the DTS and PTS of the access unit by analyzing payload data of an MMT packet that stores MPU header information or Movie Fragment header information. Also, when TS is used as the multiplexing method, receiving device 200 acquires the DTS and PTS of the access unit from the header of the PES packet. Receiving device 200 acquires the DTS and PTS of the access unit from the header of the RTP packet when RTP is used as the multiplexing method.

Further, the display unit 205 may perform filter processing such as a deblocking filter at the boundary between adjacent division units when integrating the decoding results of the plurality of decoding units. Since filtering is not necessary when displaying the decoding results of a single decoding unit, the display unit 205 performs processing according to whether filtering is to be performed on the boundaries of the decoding results of a plurality of decoding units. You can switch. Whether or not filter processing is necessary may be defined in advance according to the presence or absence of division. Alternatively, information indicating whether filtering is required may be stored separately in the multiplexing layer. Information required for filtering, such as filter coefficients, may also be stored in SPS, PPS, SEI, or slice segments. The decoding units 204A to 204D or the demultiplexing unit 203 acquires the information by analyzing the SEI, and outputs the acquired information to the display unit 205. FIG. The display unit 205 performs filtering using these pieces of information. Note that when these pieces of information are stored in slice segments, it is desirable that the decoding units 204A to 204D acquire these pieces of information.

In the above description, an example in which two types of data are stored in a fragment, that is, pre-slice segment data and slice segment data, is shown, but the number of data types may be three or more. In this case, classification is performed according to the type in step S203.

Also, when the data size of the slice segment is large, transmitting apparatus 100 may fragment the slice segment and store the fragment in the MMT packet. That is, transmitting apparatus 100 may fragment the pre-slice segment data and the slice segment. In this case, setting the access unit equal to the data unit as in the example of packetization shown in FIG. 11 causes the following problem.

For example, when slice segment 1 is divided into three fragments, slice segment 1 is divided into three packets with Fragment counter values of 1 to 3 and transmitted. Also, after slice segment 2, the Fragment counter value is 4 or more, and the value of the Fragment counter cannot be associated with the data stored in the payload. Therefore, receiving device 200 cannot identify the packet that stores the head data of the slice segment from the information in the header of the MMT packet.

In such a case, receiving device 200 may analyze the payload data of the MMT packet to identify the start position of the slice segment. Here, H.I. 264 or H.264. As a format for storing NAL units in the multiplexing layer in H.265, a format called a byte stream format in which a start code consisting of a specific bit string is added immediately before the NAL unit header, and a field indicating the size of the NAL unit is added. There are two types: a format called NAL size format.

Byte stream formats are used in MPEG-2 systems, RTP, and the like. The NAL size format is used in MP4, DASH and MMT, etc. that use MP4.

When the byte stream format is used, receiving device 200 analyzes whether the head data of the packet matches the start code. If the head data of the packet matches the start code, receiving device 200 obtains the NAL unit type from the following NAL unit header, and determines whether the data included in the packet is slice segment data. can detect what

On the other hand, in the case of the NAL size format, receiving device 200 cannot detect the starting position of the NAL unit based on the bit string. Therefore, in order to acquire the starting position of the NAL unit, receiving apparatus 200 needs to shift the pointer by reading data by the size of the NAL unit in order from the leading NAL unit of the access unit.

However, in the header of the MPU or Movie Fragment in MMT, the size of the sub-sample unit is indicated, and if the sub-sample corresponds to the pre-slice data or slice segment, the receiving device 200 is based on the size information of the sub-sample The starting position of each NAL unit can be specified. Therefore, transmitting apparatus 100 may include information indicating whether sub-sample unit information exists in MPU or Movie Fragment in information such as MPT in MMT that receiving apparatus 200 acquires at the start of data reception. .

The MPU data is based on the MP4 format and expanded. In MP4, H. 264 or H.264. There are a mode in which parameter sets such as H.265 SPS and PPS can be stored as sample data and a mode in which they cannot be stored. Also, information for specifying this mode is indicated as the entry name of SampleEntry. If the storable mode is used and the parameter set is included in the sample, the receiving device 200 acquires the parameter set by the method described above.

On the other hand, if a non-storable mode is used, the parameter set is stored as Decoder Specific Information in the SampleEntry or stored using a stream for the parameter set. Here, since parameter set streams are not generally used, transmitting apparatus 100 preferably stores parameter sets in Decoder Specific Information. In this case, receiving device 200 analyzes SampleEntry transmitted as MPU metadata or Movie Fragment metadata in an MMT packet, and obtains a parameter set referred to by an access unit.

When the parameter set is stored as sample data, receiving apparatus 200 can obtain the parameter set necessary for decoding by referring only to the sample data without referring to SampleEntry. At this time, transmitting apparatus 100 does not need to store the parameter set in SampleEntry. By doing so, transmitting apparatus 100 can use the same SampleEntry in different MPUs, so that the processing load of transmitting apparatus 100 when generating MPUs can be reduced. Furthermore, there is the advantage that receiving apparatus 200 does not need to refer to the parameter set in SampleEntry.

Alternatively, transmitting device 100 may store one default parameter set in SampleEntry and store the parameter set referred to by the access unit in sample data. In conventional MP4, it was common to store parameter sets in SampleEntry. Therefore, if there is no parameter set in SampleEntry, there is a possibility that a receiving device will stop playing. This problem can be solved by using the above method.

Alternatively, transmitting device 100 may store the parameter set in the sample data only when a parameter set different from the default parameter set is used.

In both modes, it is possible to store the parameter set in SampleEntry, so transmitting apparatus 100 may always store the parameter set in VisualSampleEntry, and receiving apparatus 200 may always acquire the parameter set from VisualSampleEntry.

In the MMT standard, MP4 header information such as Moov and Moof is transmitted as MPU metadata or movie fragment metadata, but transmitting device 100 does not necessarily transmit MPU metadata and movie fragment metadata. You don't have to. Further, the receiving device 200 determines whether SPS and PPS are stored in the sample data based on ARIB (Association of Radio Industries and Businesses) standard services, asset types, or whether or not MPU meta is transmitted. It is also possible to determine

FIG. 17 is a diagram showing an example in which the pre-slice segment data and each slice segment are set to different data units.

In the example shown in FIG. 17, the size of the data before the slice segment and the data size of slice segment 1 to slice segment 4 are Length#1 to Length#5, respectively. Field values of Fragmentation indicator, Fragment counter, and Offset included in the header of the MMT packet are shown in the figure.

Here, Offset indicates the bit length (offset) from the beginning of the coded data of the sample (access unit or picture) to which the payload data belongs to the first byte of the payload data (coded data) included in the MMT packet. This is offset information. The value of the Fragment counter is described as starting from a value obtained by subtracting 1 from the total number of fragments, but it may start from another value.

FIG. 18 is a diagram showing an example of fragmentation of Data units. In the example shown in FIG. 18, slice segment 1 is divided into three fragments, which are stored in MMT packet #2 to MMT packet #4, respectively. Also at this time, if the data size of each fragment is Length#2_1 to Length#2_3, the values of each field are as shown in the figure.

Thus, when a data unit such as a slice segment is set to Data unit, the beginning of the access unit and the beginning of the slice segment can be determined as follows based on the field values of the MMT packet header.

The beginning of the payload in a packet whose Offset value is 0 is the beginning of the access unit.

The head of the payload of a packet whose Offset value is different from 0 and whose Fragmentation indicator value is 00 or 01 is the head of the slice segment.

In addition, when data unit fragmentation does not occur and packet loss does not occur, receiving apparatus 200 stores in the MMT packet based on the number of slice segments acquired after detecting the beginning of the access unit. You can specify the index number of the slice segment.

Similarly, even when the Data unit of the pre-slice segment data is fragmented, receiving apparatus 200 can detect the beginning of the access unit and the slice segment.

Further, even when a packet loss occurs, or when the SPS, PPS, and SEI included in the pre-slice segment data are set to separate data units, the receiving apparatus 200 slices data based on the analysis result of the MMT header. By identifying the MMT packet storing the head data of the segment and then analyzing the header of the slice segment, the starting position of the slice segment or tile within the picture (access unit) can be identified. The amount of processing involved in parsing the slice header is small, and the processing load is not a problem.

In this way, each of a plurality of coded data of a plurality of slice segments is associated one-to-one with a basic data unit (data unit) that is a unit of data stored in one or more packets. Also, each of the multiple encoded data is stored in one or more MMT packets.

Header information of each MMT packet includes a Fragmentation indicator (identification information) and an Offset (offset information).

Receiving apparatus 200 determines that the beginning of payload data included in a packet having header information including a fragmentation indicator with a value of 00 or 01 is the beginning of encoded data of each slice segment. . Specifically, the beginning of the payload data contained in the packet having header information containing the Offset value not equal to 0 and the Fragmentation indicator whose value is 00 or 01 is the beginning of the encoded data of each slice segment. I judge.

Also, in the example of FIG. 17, the head of the data unit is either the head of the access unit or the head of the slice segment, and the value of the fragmentation indicator is 00 or 01. Furthermore, receiving apparatus 200 refers to the type of the NAL unit and determines whether the head of the Data Unit is an access unit delimiter or a slice segment. It is also possible to detect the beginning or the beginning of a slice segment.

In this way, transmitting apparatus 100 performs packetization so that the head of the NAL unit always starts from the head of the payload of the MMT packet, including the case where the pre-slice segment data is divided into a plurality of data units. Therefore, receiving apparatus 200 can detect the beginning of the access unit or slice segment by analyzing the fragmentation indicator and the NAL unit header. The NAL unit type is present in the first byte of the NAL unit header. Therefore, receiving device 200 can acquire the type of the NAL unit by additionally analyzing 1-byte data when analyzing the header portion of the MMT packet. In the case of audio, receiving apparatus 200 only needs to be able to detect the beginning of the access unit, and may determine based on whether the value of the fragmentation indicator is 00 or 01.

Further, as described above, when encoded data encoded so as to enable division decoding is stored in PES packets of MPEG-2 TS, transmitting apparatus 100 can use data alignment descriptors. be. An example of a method of storing encoded data in a PES packet will be described in detail below.

For example, in HEVC, the transmitting device 100 can indicate whether data stored in a PES packet is an access unit, a slice segment, or a tile by using a data alignment descriptor. Alignment types in HEVC are defined as follows.

Alignment type=8 indicates HEVC slice segments. Alignment type=9 indicates an HEVC slice segment or access unit. Alignment type=12 indicates HEVC slice segments or tiles.

Therefore, transmitting apparatus 100 can indicate that the data of the PES packet is either slice segment or pre-slice segment data by using type 9, for example. Since types indicating slices instead of slice segments are separately defined, transmitting apparatus 100 may use types indicating slices instead of slice segments.

Also, the DTS and PTS included in the header of the PES packet are set only in the PES packet including the leading data of the access unit. Therefore, if the type is 9 and the PES packet has a DTS or PTS field, the receiving device 200 assumes that the PES packet stores the entire access unit or the head division unit of the access unit. I can judge.

In addition, transmitting apparatus 100 uses a field such as transport_priority indicating the priority of a TS packet that stores a PES packet that includes the leading data of an access unit, so that receiving apparatus 200 can distinguish data included in the packet. good. Further, receiving device 200 may determine data included in the packet by analyzing whether the payload of the PES packet is an access unit delimiter. Also, the data_alignment_indicator in the PES packet header indicates whether data is stored in the PES packet according to these types. If this flag (data_alignment_indicator) is set to 1, the data stored in the PES packet is guaranteed to conform to the type indicated in the data alignment descriptor.

Also, transmitting apparatus 100 may use the data alignment descriptor only when performing PES packetization in units such as slice segments that can be divided and decoded. As a result, when the data alignment descriptor exists, receiving apparatus 200 can determine that the encoded data is PES-packetized in units that can be divided and decoded. It can be determined that the encrypted data is PES-packetized in access unit units. Note that the MPEG-2 TS standard defines that the unit of PES packetization is an access unit when the data_alignment_indicator is set to 1 and the data alignment descriptor does not exist.

If a data alignment descriptor is included in the PMT, receiving device 200 determines that PES packetization is performed in units that can be divided and decoded, and based on the packetized unit, input to each decoding unit data can be generated. In addition, if the receiving apparatus 200 determines that parallel decoding of encoded data is necessary based on program information or other descriptor information because the data alignment descriptor is not included in the PMT, , by analyzing the slice header of the slice segment, etc., to generate input data to each decoding unit. Also, when the encoded data can be decoded by a single decoding unit, receiving apparatus 200 decodes the data of the entire access unit by the decoding unit. Note that if the information indicating whether the encoded data is composed of a divisionally decodable unit such as a slice segment or a tile is separately indicated by a PMT descriptor or the like, receiving apparatus 200 uses the descriptor It may be determined whether the encoded data can be decoded in parallel based on the analysis result.

Also, since the DTS and PTS contained in the PES packet header are set only in the PES packet containing the top data of the access unit, when the access unit is divided into PES packets, the second and subsequent PES The packet does not contain information indicating the DTS and PTS of the access unit. Therefore, when decoding processing is performed in parallel, each of the decoding units 204A to 204D and the display unit 205 uses the DTS and PTS stored in the header of the PES packet containing the head data of the access unit.

(Embodiment 2)
Embodiment 2 describes a method of storing NAL size format data in an MP4 format-based MPU in MMT. In the following, a method of storing in an MPU used for MMT will be described as an example, but such a storing method can also be applied to DASH, which is based on the same MP4 format.

[How to store in MPU]
In the MP4 format, a plurality of access units are collectively stored in one MP4 file. In the MPU used for MMT, data for each medium is stored in one MP4 file, and the data can contain any number of access units. Since the MPU is a single decodable unit, for example, the MPU stores access units in GOP units.

FIG. 19 is a diagram showing the configuration of the MPU. The head of the MPU is ftyp, mmpu, and moov, which are collectively defined as MPU metadata. moov stores initialization information common to files and MMT hint tracks.

In addition, the moof includes initialization information and size for each sample and sub-sample, information (sample_duration, sample_size, sample_composition_time_offset) that can specify presentation time (PTS) and decoding time (DTS), and data_offset that indicates the position of data. Stored.

Also, a plurality of access units are each stored in mdat (mdat box) as a sample. Data other than samples among moof and mdat is defined as movie fragment metadata (hereinafter referred to as MF metadata), and mdat sample data is defined as media data.

FIG. 20 is a diagram showing the structure of MF metadata. As shown in FIG. 20, the MF metadata more specifically consists of the type, length, and data of a moof box (moof), and the type and length of an mdat box (mdat).

When storing an access unit in MP4 data, H. 264 and H.264. There are a mode in which parameter sets such as H.265 SPS and PPS can be stored as sample data and a mode in which they cannot be stored.

Here, in the mode in which data cannot be stored, the parameter set is stored in Decoder Specific Information of SampleEntry in moov. Also, in the storable mode, the parameter set is included in the sample.

MPU metadata, MF metadata, and media data are each stored in the MMT payload, and the fragment type (FT) is stored in the header of the MMT payload as an identifier capable of identifying these data. FT=0 indicates MPU metadata, FT=1 indicates MF metadata, and FT=2 indicates media data.

FIG. 19 shows an example in which MPU metadata units and MF metadata units are stored in the MMT payload as data units. It may be stored in the MMT payload on a unit-by-unit basis. Similarly, FIG. 19 illustrates an example in which sample units are stored in the MMT payload as data units. However, data units may be configured in sample units or NAL unit units, and such data units may be stored in the MMT payload in data unit units. Such data units may be further fragmented and stored in the MMT payload.

[Conventional transmission method and problems]
Conventionally, when encapsulating a plurality of access units in MP4 format, moov and moof were created when all the samples to be stored in MP4 were completed.

When the MP4 format is transmitted in real time using broadcasting, for example, if the samples stored in one MP4 file are in GOP units, moov and moof are created after the time samples in GOP units are accumulated. There is a delay associated with encapsulation. Such encapsulation on the sender side always increases the end-to-end delay by the GOP unit time. This makes it difficult to provide services in real time, and especially when live content is transmitted, it leads to deterioration of services for viewers.

FIG. 21 is a diagram for explaining the data transmission order. When MMT is applied to broadcasting, as shown in (a) of FIG. 21, MMT packets are transmitted in order of MPU configuration (MMT packets #1, #2, #3, #4, #5, #6 order), the transmission of MMT packets is delayed due to encapsulation.

In order to prevent the delay due to this encapsulation, MPU header information such as MPU metadata and MF metadata is not sent (packets #1 and #2 are not sent, packet # 3-#6 are transmitted in this order). Also, as shown in (c) of FIG. 20, the media data is transmitted first without waiting for the creation of the MPU header information, and the MPU header information is transmitted after the media data is transmitted (#3-#6, # 1 and #2) can be considered.

The receiving device decodes without using the MPU header information if the MPU header information is not transmitted, and the receiving device decodes the MPU header information if the MPU header information is post-sent with respect to the media data. Wait for information before decrypting.

However, conventional MP4-compliant receivers are not guaranteed to decode without using MPU header information. Also, when the receiving apparatus performs special processing to perform decoding without using the MPU header, if the conventional transmission method is used, the decoding processing becomes complicated, and there is a high possibility that real-time decoding will become difficult. Also, if the receiving device waits for the acquisition of the MPU header information before decoding, media data must be buffered until the receiving device acquires the header information. decryption was not guaranteed.

Therefore, as shown in (d) of FIG. 20, the transmitting apparatus according to Embodiment 2 stores only common information in MPU metadata, thereby transmitting MPU metadata before media data. Then, the transmitting apparatus according to Embodiment 2 transmits the MF metadata whose generation is delayed after the media data. This provides a transmission method or reception method that can guarantee decoding of media data.

A reception method using each of the transmission methods (a) to (d) in FIG. 21 will be described below.

In each transmission method shown in FIG. 21, first, MPU data is configured in the order of MPU metadata, MFU metadata, and media data.

After constructing MPU data, when the transmitting device transmits data in the order of MPU metadata, MF metadata, and media data as shown in FIG. ) and (A-2).

(A-1) After acquiring the MPU header information (MPU metadata and MF metadata), the receiving device decodes the media data using the MPU header information.

(A-2) The receiving device decodes the media data without using the MPU header information.

In any of these methods, there is a delay due to encapsulation on the transmitting side, but there is an advantage that the receiving device does not need to buffer media data to acquire the MPU header. Without buffering, there is no need to install memory for buffering, and no buffering delay occurs. Also, since the method (A-1) performs decoding using the MPU header information, it can also be applied to a conventional receiving apparatus.

When the transmitting device transmits only media data as shown in FIG. 21(b), the receiving device can perform decoding by the following method (B-1).

(B-1) The receiving device decodes media data without using MPU header information.

Also, although not shown, if the MPU metadata is transmitted prior to the transmission of the media data in (b) of FIG. 21, decoding can be performed by the following method (B-2).

(B-2) The receiving device decodes the media data using the MPU metadata.

Both of the above methods (B-1) and (B-2) are advantageous in that there is no delay due to encapsulation on the transmitting side and there is no need to buffer media data to obtain the MPU header. is. However, both methods (B-1) and (B-2) do not perform decoding using MPU header information, so there is a possibility that special processing will be required for decoding.

When the transmitting device transmits data in the order of media data, MPU metadata, and MF metadata as shown in (c) of FIG. Decryption can be performed by either method.

(C-1) After acquiring the MPU header information (MPU metadata and MF metadata), the receiving device decodes the media data.

(C-2) The receiving device decodes the media data without using the MPU header information.

When the above method (C-1) is used, it is necessary to buffer the media data in order to obtain the MPU header information. On the other hand, when the above method (C-2) is used, there is no need to perform buffering for obtaining MPU header information.

In both methods (C-1) and (C-2) above, no delay due to encapsulation occurs on the transmitting side. Also, since method (C-2) does not use MPU header information, there is a possibility that special processing will be required.

When the transmitting device transmits data in the order of MPU metadata, media data, and MF metadata as shown in (d) of FIG. 21, the receiving device performs the following (D-1) and (D-2) ) can be used for decoding.

(D-1) After obtaining the MPU metadata, the receiving device further obtains the MF metadata, and then decodes the media data.

(D-2) After obtaining the MPU metadata, the receiving device decodes the media data without using the MF metadata.

When the above method (D-1) is used, it is necessary to buffer the media data for MF metadata acquisition, but in the case of the above (D-2) method, for MF metadata acquisition no need to do any buffering.

Since method (D-2) does not perform decoding using MF metadata, special processing may be required.

As described above, when decoding is possible using MPU metadata and MF metadata, there is an advantage that conventional MP4 receivers can also perform decoding.

In FIG. 21, the MPU data is configured in the order of MPU metadata, MFU metadata, and media data, and in moof, position information (offset) for each sample and subsample is determined based on this configuration. ing. The MF metadata also includes data other than the media data in the mdat box (box size and type).

Therefore, when the receiving device specifies media data based on the MF metadata, the receiving device reconfigures the data in the order in which the MPU data was configured, regardless of the order in which the data was transmitted. , MPU metadata moov or MF metadata moof is used for decoding.

In FIG. 21, the MPU data is configured in the order of MPU meta data, MFU meta data, and media data. good.

For example, MPU data may be composed of MPU metadata, media data, and MF metadata in this order, and negative position information (offset) may be indicated in the MF metadata. In this case as well, regardless of the order in which the data is transmitted, the receiving device reconstructs the data in the order in which the MPU data was composed on the transmitting side, and then performs decoding using moov or moof.

The transmitting device may signal information indicating the order in which MPU data is constructed, and the receiving device may reconstruct data based on the signaled information.

As described above, the receiving apparatus converts packetized MPU metadata, packetized media data (sample data), and packetized MF metadata into this receive in order. Here, the MPU metadata is an example of first metadata, and the MF metadata is an example of second metadata.

Next, the receiving device reconstructs the MPU data (MP4 format file) including the received MPU metadata, the received MF metadata, and the received sample data. Then, the sample data included in the reconstructed MPU data is decoded using the MPU metadata and MF metadata. MF metadata is metadata that includes data (eg, length stored in mbox) that can be generated only after sample data is generated on the transmitting side.

More specifically, the operation of the receiving device is performed by each constituent element of the receiving device. For example, the receiving device includes a receiving section that receives the data, a reconstruction section that reconstructs the MPU data, and a decoding section that decodes the MPU data. Each of the receiver, generator, and decoder is implemented by a microcomputer, processor, dedicated circuit, or the like.

[Decryption method without using header information]
Next, a method of decoding without using header information will be described. Here, a method of decoding without using header information at the receiving device will be described regardless of whether or not the sending side sends header information. That is, this method is applicable to any of the transmission methods described with reference to FIG. However, some decoding methods are decoding methods that are applicable only for specific transmission methods.

FIG. 22 is a diagram showing an example of a decoding method without using header information. FIG. 22 shows only MMT payloads and MMT packets containing only media data, and does not show MMT payloads and MMT packets containing MPU metadata and MF metadata. Also, in the following description of FIG. 22, it is assumed that media data belonging to the same MPU are transmitted continuously. Also, a case where samples are stored in the payload as media data will be described as an example, but in the description of FIG. may be

In order to decode media data, the receiving device must first acquire initialization information necessary for decoding. Also, if the media is video, the receiving device must acquire initialization information for each sample, identify the start position of the MPU, which is a random access unit, and acquire the start positions of the samples and NAL units. . Also, the receiving device needs to specify the decoding time (DTS) and presentation time (PTS) of each sample.

Therefore, the receiving device can perform decoding without using the header information using, for example, the following method. If the payload stores NAL unit units or units obtained by fragmenting NAL units, "samples" in the following description should be read as "NAL units in samples".

<random access (=specify the first sample of MPU)>
The following method 1 and method 2 are available for the receiving device to identify the first sample of the MPU when the header information is not transmitted. Note that Method 3 can be used when header information is transmitted.

[Method 1] The receiving device obtains a sample included in an MMT packet with 'RAP_flag=1' in the MMT packet header.

[Method 2] The receiving device obtains a sample with 'sample number=0' in the MMT payload header.

[Method 3] If at least one of MPU metadata and/or MF metadata is transmitted before and/or after the media data, the receiving device uses the fragment type in the MMT payload header (FT) takes the samples contained in the MMT payload switched to media data.

In method 1 and method 2, when multiple samples belonging to different MPUs are mixed in one payload, it is impossible to determine which NAL unit is a random access point (RAP_flag=1 or sample number=0). For this reason, if the sample of different MPU is not mixed in one payload, or if the sample of different MPU is mixed in one payload, the last (or first) sample is a random access point, RAP_flag is set to 1, and the like.

Also, in order for the receiving device to acquire the starting position of the NAL unit, it is necessary to shift the data read pointer by the size of the NAL unit in order from the leading NAL unit of the sample.

When the data is fragmented, the receiving device can identify the data unit by referring to fragment_indicator and fragment_number.

<Determination of sample DTS>
Methods for determining the DTS of a sample include method 1 and method 2 below.

[Method 1] The receiving device determines the DTS of the first sample based on the prediction structure. However, this method requires analysis of encoded data and may be difficult to decode in real time, so the following method 2 is desirable.

[Method 2] The receiving device separately transmits the DTS of the leading sample, and acquires the DTS of the transmitted leading sample. Methods for transmitting the DTS of the first sample include, for example, a method of transmitting the DTS of the MPU first sample using MMT-SI, a method of transmitting the DTS for each sample using the MMT packet header extension area, and the like. DTS may be an absolute value or a relative value to PTS. Also, the transmitting side may signal whether or not the DTS of the first sample is included.

In both method 1 and method 2, the DTS of subsequent samples is calculated assuming that it has a fixed frame rate.

As a method of storing the DTS for each sample in the packet header, there is a method of storing the DTS of the sample included in the MMT packet in the 32-bit NTP timestamp field in the MMT packet header, in addition to using the extension area. If the DTS cannot be represented by the number of bits (32 bits) of one packet header, the DTS may be represented using multiple packet headers. Also, the DTS may be expressed by combining the NTP timestamp field of the packet header and the extension area. If DTS information is not included, a known value (for example, ALL0) is used.

<Determination of sample PTS>
The receiving device acquires the PTS of the first sample from the MPU timestamp descriptor for each asset included in the MPU. The receiving device calculates the subsequent sample PTS from a parameter such as POC indicating the display order of the samples, assuming that the frame rate is fixed. Thus, in order to calculate DTS and PTS without using header information, transmission at a fixed frame rate is essential.

Also, when MF metadata is transmitted, the receiving device receives relative time information of the DTS and PTS from the first sample indicated in the MF metadata and the timestamp of the MPU first sample indicated in the MPU timestamp descriptor. The absolute values of DTS and PTS can be calculated from the absolute values.

Note that when calculating the DTS and PTS by analyzing the encoded data, the receiving device may use the SEI information included in the access unit.

<Initialization information (parameter set)>
[For video]
For video, parameter sets are stored in the sample data. Also, when MPU metadata and MF metadata are not transmitted, it is guaranteed that a parameter set necessary for decoding can be obtained by referring only to sample data.

Also, as in (a) and (d) of FIG. 21, when the MPU metadata is transmitted before the media data, it may be specified that the parameter set is not stored in the SampleEntry. In this case, the receiving device does not refer to the parameter set of SampleEntry, but refers only to the parameter set within the sample.

Also, when MPU metadata is transmitted before media data, SampleEntry stores a common parameter set for MPUs and a default parameter set, and the receiving device stores the parameter set in SampleEntry and the parameter set in the sample. You can refer to it. By storing the parameter set in the SampleEntry, it is possible to perform decoding even with a conventional receiving apparatus that cannot reproduce unless the parameter set exists in the SampleEntry.

[For audio]
For audio, decoding requires a LATM header, and for MP4 it is mandatory that the LATM header is included in the sample entry. However, if the header information is not transmitted, it is difficult for the receiving device to acquire the LATM header, so the LATM header is separately included in control information such as SI. Note that LATM headers may be included in messages, tables, or descriptors. Note that the LATM header may also be included in the sample.

The receiving device acquires the LATM header from SI or the like before starting decoding, and starts decoding the audio. Alternatively, as shown in (a) and (d) of FIG. 21, when the MPU metadata is transmitted before the media data, the receiving device receives the LATM header before the media data. It is possible. Therefore, when the MPU meta data is transmitted before the media data, decoding can be performed even using a conventional receiving device.

<Others>
The transmission order and type of transmission order may be signaled in MMT packet headers, payload headers, or control information such as MPT or other tables, messages, descriptors. The types of transmission order here are, for example, the four types of transmission order shown in (a) to (d) of FIG. 21, and an identifier for identifying each type can be obtained before decoding starts. It should be stored in place.

Also, different transmission order types may be used for audio and video, or a common type may be used for audio and video. Specifically, for example, audio is transmitted in the order of MPU metadata, MF metadata, and media data as shown in FIG. 21(a), and video is transmitted in the order shown in FIG. 21(d). , MPU metadata, media data, and MF metadata may be transmitted in this order.

With the method described above, the receiving device can perform decoding without using header information. Also, when the MPU metadata is transmitted before the media data (FIGS. 21(a) and 21(d)), it is possible to perform decoding even with a conventional receiving device.

In particular, since the MF meta data is transmitted after the media data ((d) in FIG. 21), it is possible to perform decoding even with a conventional receiving device without causing a delay due to encapsulation.

[Structure and operation of transmitter]
Next, the configuration and operation of the transmission device will be described. 23 is a block diagram of a transmission device according to Embodiment 2, and FIG. 24 is a flowchart of a transmission method according to Embodiment 2. FIG.

As shown in FIG. 23 , transmitting device 15 includes encoding section 16 , multiplexing section 17 , and transmitting section 18 .

The encoding unit 16 converts video or audio to be encoded into, for example, H.264. The encoded data is generated by encoding according to H.265 (S10).

The multiplexer 17 multiplexes (packets) the encoded data generated by the encoder 16 (S11). Specifically, the multiplexing unit 17 packetizes each of the sample data, the MPU metadata, and the MF metadata that constitute the MP4 format file. Sample data is data obtained by encoding a video signal or an audio signal, MPU metadata is an example of first metadata, and MF metadata is an example of second metadata. Both the first metadata and the second metadata are metadata used for decoding the sample data, but the difference between them is that the second metadata is data that can be generated only after the generation of the sample data. is to include

Here, the data that can be generated only after generating the sample data is, for example, data other than the sample data stored in the mdat in the MP4 format (data in the header of the mdat. That is, type and length shown in FIG. 20). ). Here, the second metadata may include length, which is at least part of this data.

The transmission unit 18 transmits the packetized MP4 format file (S12). The transmission unit 18 transmits the MP4 format file by the method shown in (d) of FIG. 21, for example. That is, packetized MPU metadata, packetized sample data, and packetized MF metadata are transmitted in this order.

Each of the encoding unit 16, the multiplexing unit 17, and the transmitting unit 18 is implemented by a microcomputer, processor, dedicated circuit, or the like.

[Configuration of Receiving Device]
Next, the configuration and operation of the receiving device will be described. 25 is a block diagram of a receiving device according to Embodiment 2. FIG.

As shown in FIG. 25, the receiver 20 includes a packet filtering unit 21, a transmission order type determination unit 22, a random access unit 23, a control information acquisition unit 24, a data acquisition unit 25, PTS and DTS calculation It includes a unit 26 , an initialization information acquisition unit 27 , a decoding instruction unit 28 , a decoding unit 29 and a presentation unit 30 .

[Operation 1 of the receiving device]
First, the operation for specifying the MPU head position and the NAL unit position by the receiving device 20 when the media is video will be described. FIG. 26 is a flow chart of such operation of the receiver 20 . Here, it is assumed that the transmission order type of MPU data is stored in the SI information by the transmission device 15 (multiplexer 17).

First, the packet filtering unit 21 performs packet filtering on the received file. The transmission order type determination unit 22 analyzes the SI information obtained by packet filtering and acquires the transmission order type of MPU data (S21).

Next, the transmission order type determination unit 22 determines (determines) whether or not MPU header information (at least one of MPU metadata and MF metadata) is included in the data after packet filtering (S22). If the MPU header information ( is included (Yes in S22), the random access unit 23 identifies the MPU head sample by detecting that the fragment type of the MMT payload header is switched to media data (S23 ).

On the other hand, if the MPU header information is not included (No in S22), the random access unit 23 identifies the MPU head sample based on the RAP_flag of the MMT packet header or the sample number of the MMT payload header (S24).

Further, the transmission order type determination unit 22 determines whether or not the MF metadata is included in the packet-filtered data (S25). If it is determined that the MF metadata is included (Yes in S25), the data acquisition unit 25 reads the NAL unit based on the sample, subsample offset, and size information included in the MF metadata. NAL unit is acquired by this (S26). On the other hand, when it is determined that the MF metadata is not included (No in S25), the data acquisition unit 25 sequentially reads the data of the size of the NAL unit from the top NAL unit of the sample, thereby obtaining the NAL unit. (S27).

Note that even when it is determined in step S22 that the MPU header information is included, the receiving device 20 may specify the MPU head sample using the process of step S24 instead of step S23. Further, when it is determined that the MPU header information is included, the process of step S23 and the process of step S24 may be used together.

Also, even when it is determined in step S25 that the MF metadata is included, the receiving device 20 may acquire the NAL unit using the process of step S27 without using the process of step S26. Further, when it is determined that MF metadata is included, the processing of step S23 and the processing of step S24 may be used together.

Further, it is assumed that the MF data is transmitted after the media data when it is determined in step S25 that the MF metadata is included. In this case, the receiving device 20 may buffer the media data, wait until the MF metadata is acquired, and then perform the process of step S26, or the receiving device 20 may perform the process of step S26 without waiting for the acquisition of the MF metadata. It may be determined whether or not to perform the process of step S27.

For example, receiving device 20 may determine whether to wait for acquisition of MF metadata based on whether or not it has a buffer with a buffer size capable of buffering media data. Also, the receiving device 20 may determine whether to wait for acquisition of MF metadata based on whether the end-to-end delay is reduced. Further, the receiving device 20 may perform decoding processing mainly using the processing of step S26, and may use the processing of step S27 in the case of the processing mode when packet loss or the like occurs.

Note that if the transmission order type is predetermined, steps S22 and S26 may be omitted. In this case, the receiving device 20 considers the buffer size and the End-to-End delay, A method for specifying the leading sample and a method for specifying the NAL unit may be determined.

Note that if the transmission order type is known in advance, the transmission order type determination unit 22 is unnecessary in the receiving device 20 .

Further, although not described in FIG. 26 above, the decoding instruction unit 28 acquires data based on the PTS, the PTS and DTS calculated by the DTS calculation unit 26, and the initialization information acquired by the initialization information acquisition unit 27. The data acquired in the section is output to the decoding section 29 . The decoding unit 29 decodes the data, and the presentation unit 30 presents the decoded data.

[Operation 2 of the receiving device]
Next, the operation of receiving device 20 acquiring initialization information based on the transmission order type and decoding media data based on the initialization information will be described. FIG. 27 is a flow chart of such operation.

First, the packet filtering unit 21 performs packet filtering on the received file. The transmission order type determination unit 22 analyzes the SI information obtained by packet filtering and acquires the transmission order type (S301).

Next, the transmission order type determination unit 22 determines whether or not MPU metadata has been transmitted (S302). If it is determined that the MPU metadata is transmitted (Yes in S302), the transmission order type determination unit 22 determines whether the MPU metadata is transmitted before the media data as a result of the analysis in step S301. (S303). If the MPU metadata has been transmitted before the media data (Yes in S303), the initialization information acquisition unit 27 performs initialization based on the common initialization information included in the MPU metadata and the sample data initialization information. to decode the media data (S304).

On the other hand, when it is determined that the MPU metadata is transmitted after the media data (No in S303), the data acquisition unit 25 buffers the media data until the MPU metadata is acquired (S305). , the process of step S304 is performed after the MPU metadata is acquired.

Further, when it is determined in step S302 that the MPU metadata has not been transmitted (No in S302), the initialization information acquisition unit 27 decodes the media data based only on the initialization information of the sample data. (S306).

Note that if the media data decoding is guaranteed only when the transmission side is based on the initialization information of the sample data, the processing based on the determinations of steps S302 and S303 is not performed, and the processing of step S306 is used.

Also, the receiving device 20 may determine whether or not to buffer the media data before step S305. In this case, if the receiving device 20 determines to buffer the media data, it proceeds to the processing of step S305, and if it determines that the media data is not to be buffered, it proceeds to the processing of step S306. The determination of whether to buffer the media data may be made based on the buffer size and occupancy of the receiving device 20. For example, the one with the smaller End-to-End delay is selected. The decision may be made taking into account the -to-end delay.

[Operation 3 of the receiving device]
Here, the details of the transmission method and the reception method when the MF metadata is transmitted after the media data ((c) and (d) in FIG. 21) will be described. In the following, the case of (d) in FIG. 21 will be described as an example. It is assumed that only the method of (d) in FIG. 21 is used for transmission, and no transmission order type signaling is performed.

As described above, when transmitting data in the order of MPU metadata, media data, and MF metadata, as shown in (d) of FIG.
(D-1) After acquiring the MPU metadata, the receiving device 20 decodes the media data after acquiring the MF metadata.
(D-2) After acquiring the MPU metadata, the receiving device 20 decodes the media data without using the MF metadata.
Two decoding methods are possible.

Here, D-1 requires media data buffering for MF metadata acquisition, but can be decoded using the MPU header information, so it can be decoded by a conventional MP4-compliant receiving device. becomes. D-2 does not require media data buffering for MF metadata acquisition, but requires special processing for decoding because it cannot be decoded using MF metadata.

Also, the method of (d) in FIG. 21 has the advantage that the MF metadata is transmitted later than the media data, so no delay due to encapsulation occurs and the end-to-end delay can be reduced.

The receiving device 20 can select one of the above two decoding methods according to the capabilities of the receiving device 20 and the quality of service provided by the receiving device 20 .

The transmitting device 15 must ensure that the decoding operation in the receiving device 20 can be performed while reducing the occurrence of buffer overflows and underflows. For example, the following parameters can be used as elements for defining a decoder model when decoding using method D-1.

- Buffer size for reconfiguring the MPU (MPU buffer)
For example, buffer size=maximum rate×maximum MPU time×α, where maximum rate is profile and level upper limit rate of encoded data+MPU header overhead. The maximum MPU time is the maximum time length of GOP when 1 MPU=1 GOP (video).

Here, the audio may be a GOP unit common to the video, or may be another unit. α is a margin for preventing overflow, and may be multiplied or added to maximum rate×maximum MPU time. If it is multiplied, α≧1, and if it is added, α≧0.

- The upper limit of the decoding delay time from when data is input to the MPU buffer until it is decoded. (TSTD_delay in MPEG-TS STD)
For example, in consideration of the maximum MPU time and the upper limit of the decoding delay time during transmission, DTS is set such that the MPU data acquisition completion time in the receiver <= DTS.

Also, the transmitting device 15 may assign DTS and PTS according to the decoder model when decoding using method D-1. As a result, the transmitting device 15 guarantees the decoding of the receiving device that performs decoding using the method D-1, and at the same time, transmits auxiliary information necessary when decoding is performed using the method D-2. may

For example, the transmitting device 15 can signal the pre-buffering time in the decoder buffer when decoding using the D-2 method, thereby ensuring the operation of the receiving device decoding using the D-2 method.

The pre-buffering time may be included in SI control information such as messages, tables, descriptors, etc., and may be included in the headers of MMT packets, MMT payloads. Also, the SEI in the encoded data may be overwritten. The DTS and PTS for decoding using method D-1 are stored in the MPU Timestamp Descriptor, SampleEntry, and the DTS and PTS for decoding using method D-2 or the pre-buffering time is May be described in the SEI.

The receiving device 20 selects the decoding method D-1 and supports both D-1 and D-2 when the receiving device 20 supports only the MP4-compliant decoding operation using the MPU header. If so, you may choose one or the other.

The transmitting device 15 may add DTS and PTS so as to guarantee one decoding operation (D-1 in this description), and may further transmit auxiliary information for assisting one decoding operation.

Also, when method D-2 is used, compared to when method D-1 is used, the delay caused by pre-buffering of MF metadata may increase the End-to-End delay. is high. Therefore, receiving apparatus 20 may select method D-2 for decoding when it is desired to reduce the end-to-end delay. For example, receiving device 20 may always use method D-2 when always wanting to reduce the end-to-end delay. Further, the receiving device 20 may use the method D-2 only when operating in a low-delay presentation mode, such as live content, channel selection, zapping operation, etc., to be presented with a low delay.

FIG. 28 is a flow chart of such a receiving method.

First, the receiving device 20 receives an MMT packet and acquires MPU data (S401). Then, the receiving device 20 (transmission order type determination unit 22) determines whether or not to present the program in the low-delay presentation mode (S402).

If the program is not presented in the low-delay presentation mode (No in S402), the receiving device 20 (random access unit 23 and initialization information acquisition unit 27) acquires random access and initialization information using header information (S405 ). Also, the receiving device 20 (PTS/DTS calculation unit 26, decoding command unit 28, decoding unit 29, presentation unit 30) performs decoding and presentation processing based on the PTS and DTS given by the transmission side (S406).

On the other hand, when the program is presented in the low-delay presentation mode (Yes in S402), the receiving device 20 (the random access unit 23 and the initialization information acquisition unit 27) performs random access using a decoding method that does not use header information, Initialization information is acquired (S403). Further, the receiving device 20 performs decoding and presentation processing based on auxiliary information for decoding without using the PTS, DTS, and header information given on the transmitting side (S404). Note that in steps S403 and S404, processing may be performed using MPU metadata.

[Transmitting and Receiving Method Using Auxiliary Data]
The transmission/reception operation when the MF metadata is transmitted after the media data (cases (c) and (d) in FIG. 21) has been described above. Next, a method will be described in which decoding can be started earlier and the end-to-end delay can be reduced by transmitting auxiliary data having a partial function of MF metadata by the transmitting device 15 . Here, an example in which auxiliary data is further transmitted based on the transmission method shown in FIG. 21(d) will be described, but the method using auxiliary data is shown in FIGS. It is also applicable to the transmission method that is

FIG. 29(a) is a diagram showing an MMT packet transmitted using the method shown in FIG. 21(d). That is, data is transmitted in the order of MPU metadata, media data, and MF metadata.

Here, sample #1, sample #2, sample #3, and sample #4 are samples included in media data. Although an example in which media data is stored in an MMT packet in units of samples will be described here, media data may be stored in an MMT packet in units of NAL units, or may be stored in units obtained by dividing NAL units. may be In some cases, multiple NAL units are aggregated and stored in an MMT packet.

As described in D-1 above, in the case of the method shown in FIG. 21(d), that is, when data is transmitted in the order of MPU metadata, media data, and MF metadata, After acquisition, there is a method of further acquiring MF metadata and then decoding the media data. In such method D-1, media data buffering is required for MF metadata acquisition. The -1 method has the advantage of being applicable. On the other hand, the receiving device 20 has the drawback of having to wait to start decoding until MF metadata is acquired.

On the other hand, as shown in FIG. 29(b), in the method using auxiliary data, the auxiliary data is transmitted prior to the MF metadata.

MF metadata includes information indicating the DTS, PTS, offset and size of all samples included in the movie fragment. On the other hand, the auxiliary data includes information indicating the DTS, PTS, offset and size of some of the samples included in the movie fragment.

For example, MF metadata contains information for all samples (Sample #1-Sample #4), whereas auxiliary data contains information for some samples (Sample #1-#2). .

In the case shown in (b) of FIG. 29 , sample #1 and sample #2 can be decoded by using the auxiliary data. less delay. The auxiliary data may include any combination of sample information, and the auxiliary data may be repeatedly transmitted.

For example, in (c) of FIG. 29, when transmitting the auxiliary information at timing A, the transmitting device 15 includes the information of sample #1 in the auxiliary information, and when transmitting the auxiliary information at timing B, Information on sample #1 and sample #2 is included in the auxiliary information. When the transmitting device 15 transmits the auxiliary information at the timing of C, the auxiliary information includes the information of sample #1, sample #2, and sample #3.

Note that the MF metadata includes information on sample #1, sample #2, sample #3, and sample #4 (information on all samples in the movie fragment).

Auxiliary data does not necessarily have to be sent immediately after being generated.

A type indicating that auxiliary data is stored is specified in the header of the MMT packet or MMT payload.

For example, when the ancillary data is stored in the MMT payload using the MPU mode, a data type indicating that it is an ancillary data is specified as a fragment_type field value (eg, FT=3). The auxiliary data may be data based on the configuration of moof, or may be of another configuration.

When auxiliary data is stored as a control signal (descriptor, table, message) in the MMT payload, a descriptor tag, table ID, message ID, etc. indicating auxiliary data are specified.

Also, the PTS or DTS may be stored in the header of the MMT packet or MMT payload.

[Generation example of auxiliary data]
An example in which the transmitting device generates auxiliary data based on the configuration of the moof will be described below. FIG. 30 is a diagram for explaining an example in which a transmitting device generates auxiliary data based on a moof configuration.

In normal MP4, moofs are created for movie fragments as shown in FIG. The moof contains information indicating the DTS, PTS, offset and size of the sample included in the movie fragment.

Here, the transmitting device 15 forms an MP4 (MP4 file) using only some of the sample data constituting the MPU, and generates auxiliary data.

For example, as shown in (a) of FIG. 30, the transmitting device 15 generates MP4 using only sample #1 out of samples #1 to #4 constituting the MPU, and includes a moof+mdat header. Supplementary data.

Next, as shown in (b) of FIG. 30, the transmitting device 15 generates MP4 using sample #1 and sample #2 among samples #1 to #4 constituting the MPU. Let the header of moof+mdat be the next auxiliary data.

Next, as shown in (c) of FIG. 30 , transmitting device 15 uses sample #1, sample #2, and sample #3 among samples #1 to #4 that make up the MPU to generate MP4. Among them, the header of moof+mdat is used as the next auxiliary data.

Next, as shown in (d) of FIG. 30, the transmitting device 15 generates all MP4s among the samples #1 to #4 that make up the MPU. becomes.

Although the transmitting device 15 generates auxiliary data for each sample here, it may generate auxiliary data for each N samples. The value of N may be any number, for example, N=total samples/M if auxiliary data is transmitted M times when transmitting one MPU.

The information indicating the sample offset in moof may be the offset value after the sample entry area for the number of subsequent samples is secured as the NULL area.

The auxiliary data may be generated so as to fragment the MF metadata.

[Example of receiving operation using auxiliary data]
Receipt of auxiliary data generated as described in FIG. 30 will be described. FIG. 31 is a diagram for explaining reception of auxiliary data. In FIG. 31(a), the number of samples constituting the MPU is 30, and auxiliary data is generated and transmitted every 10 samples.

In (a) of FIG. 30 , samples #1 to #10 for auxiliary data #1, samples #1 to #20 for auxiliary data #2, and samples #1 to #30 for MF metadata. Each contains information.

Although samples #1 to #10, samples #11 to #20, and samples #21 to #30 are stored in one MMT payload, they may be stored in units of samples or NALs, or may be stored in fragments. or aggregated units.

The receiving device 20 receives MPU meta, sample, MF meta, and auxiliary data packets, respectively.

The receiving device 20 concatenates (backwards) the sample data in the order of reception, and updates the previous auxiliary data after receiving the latest auxiliary data. Also, the receiving device 20 can form a complete MPU by finally replacing the auxiliary data with the MF metadata.

Upon receiving the auxiliary data #1, the receiving device 20 concatenates the data as shown in the upper part of FIG. 31(b) to form MP4. As a result, the receiving device 20 can parse the samples #1 to #10 using the information of the MPU metadata and the auxiliary data #1, and the PTS, DTS, offset, and size information included in the auxiliary data. Decoding can be performed based on

Further, when the auxiliary data #2 is received, the receiving device 20 concatenates the data as shown in the middle part of FIG. 31(b) to form MP4. As a result, the receiving device 20 can parse the samples #1 to #20 using the information of the MPU metadata and the auxiliary data #2, and the information of the PTS, DTS, offset, and size contained in the auxiliary data. Decoding can be performed based on

Also, when receiving the MF metadata, the receiving device 20 concatenates the data as shown in the lower part of (b) of FIG. 31 to form MP4. As a result, the receiving device 20 can parse the samples #1 to #30 using the MPU metadata and the MF metadata, and based on the PTS, DTS, offset, and size information included in the MF metadata, decryption can be performed.

If there is no auxiliary data, the receiving device 20 can acquire sample information only after receiving the MF metadata, so it is necessary to start decoding after receiving the MF metadata. However, since the transmitting device 15 generates and transmits the auxiliary data, the receiving device 20 can obtain the sample information using the auxiliary data without waiting for the reception of the MF metadata, thereby shortening the decoding start time. be able to. Furthermore, the transmitting device 15 generates the auxiliary data based on the moof described with reference to FIG. 30, so that the receiving device 20 can parse using the conventional MP4 parser as it is.

Also, the newly generated auxiliary data and MF metadata contain sample information that overlaps with previously transmitted auxiliary data. For this reason, even if past auxiliary data cannot be acquired due to packet loss or the like, by using newly acquired auxiliary data and MF metadata, MP4 can be reconstructed and sample information (PTS, DTS, size, and offset).

Note that the auxiliary data does not necessarily need to include information on past sample data. For example, auxiliary data #1 may correspond to sample data #1-#10, and auxiliary data #2 may correspond to sample data #11-#20. For example, as shown in (c) of FIG. 31 , the transmitting device 15 may sequentially transmit complete MF metadata as data units and units obtained by fragmenting the data units as auxiliary data.

In addition, the transmitting device 15 may repeatedly transmit the auxiliary data or repeatedly transmit the MF metadata in order to prevent packet loss.

MMT packets and MMT payloads in which auxiliary data is stored contain MPU sequence numbers and asset IDs, as well as MPU metadata, MF metadata, and sample data.

The reception operation using the auxiliary data as described above will be described with reference to the flow chart of FIG. FIG. 32 is a flow chart of a reception operation using auxiliary data.

First, the receiving device 20 receives an MMT packet and analyzes the packet header and payload header (S501). Next, the receiving device 20 analyzes whether the fragment type is auxiliary data or MF metadata (S502), and if the fragment type is auxiliary data, overwrites and updates the past auxiliary data (S503). . At this time, if there is no past auxiliary data for the same MPU, the receiving device 20 uses the received auxiliary data as new auxiliary data. Then, the receiving device 20 acquires and decodes samples based on the MPU metadata, auxiliary data, and sample data (S507).

On the other hand, if the fragment type is MF metadata, the receiving device 20 overwrites the past auxiliary data with MF metadata in step S505 (S505). Then, the receiving device 20 acquires the sample in the form of a complete MPU based on the MPU metadata, the MF metadata, and the sample data, and decodes it (S506).

Although not shown in FIG. 32, in step S502, if the fragment type is MPU metadata, the receiving device 20 stores the data in the buffer, and if the fragment type is sample data, it stores the data after each sample. Store the concatenated data in a buffer.

If the auxiliary data cannot be obtained due to packet loss, the receiving device 20 can overwrite with the latest auxiliary data or decode the samples by using past auxiliary data.

Incidentally, the transmission period and the number of times of transmission of the auxiliary data may be predetermined values. Information on the transmission period and number of times (count, countdown) may be transmitted together with the data. For example, the data unit header may store the transmission cycle, the number of transmissions, and timestamps such as initial_cpb_removal_delay.

Sending the ancillary data containing the information of the first sample of the MPU one or more times prior to initial_cpb_removal_delay allows the CPB buffer model to be followed. At this time, a value based on the picture timing SEI is stored in the MPU timestamp descriptor.

The transmission method in the reception operation using such auxiliary data is not limited to the MMT method, and can be applied to streaming transmission of packets configured in the ISOBMFF file format such as MPEG-DASH.

[Transmission method when one MPU is composed of multiple movie fragments]
In the description of FIG. 19 and subsequent figures, one MPU is composed of one movie fragment, but here, a case where one MPU is composed of a plurality of movie fragments will be described. FIG. 33 is a diagram showing the configuration of an MPU made up of multiple movie fragments.

In FIG. 33, the samples (#1-#6) stored in one MPU are divided and stored in two movie fragments. A first movie fragment is generated based on samples #1-#3 and corresponding moof boxes are generated. A second movie fragment is generated based on samples #4-#6 and a corresponding moof box is generated.

The headers of the moof box and mdat box in the first movie fragment are stored in the MMT payload and MMT packet as movie fragment metadata #1. On the other hand, the headers of the moof box and mdat box in the second movie fragment are stored in the MMT payload and MMT packet as movie fragment metadata #2. In addition, in FIG. 33, the MMT payload in which the movie fragment metadata is stored is hatched.

Note that the number of samples forming the MPU and the number of samples forming the movie fragment are arbitrary. For example, the number of samples constituting the MPU may be the number of samples in GOP units, and the number of samples half the number of samples in GOP units may be the movie fragment, and two movie fragments may be configured.

Here, an example in which one MPU includes two movie fragments (moof box and mdat box) is shown, but one MPU may include three or more movie fragments instead of two. Also, the samples stored in the movie fragment may be divided into an arbitrary number of samples instead of equally divided samples.

Note that in FIG. 33, the MPU metadata unit and the MF metadata unit are stored in the MMT payload as data units. However, the transmitting device 15 may store units such as ftyp, mmpu, moov, and moof as data units in the MMT payload in units of data units, or store the data units in units of fragments in the MMT payload. good too. Also, the transmitting device 15 may store the aggregated data units in the MMT payload.

Also, in FIG. 33, the samples are stored in the MMT payload in units of samples. However, the transmitting device 15 may form data units in units of NAL units or groups of a plurality of NAL units instead of in units of samples, and store the data units in the MMT payload in units of data units. Further, the transmitting device 15 may store data units in fragmented units in the MMT payload, or may store data units in aggregated units in the MMT payload.

In FIG. 33, the MPU is configured in the order of moof#1, mdat#1, moof#2, and mdat#2, and moof#1 is given an offset assuming that the corresponding mdat#1 is behind it. there is However, mdat#1 may precede moof#1 and an offset may be added. However, in this case, the movie fragment metadata cannot be generated in the form of moof+mdat, and the moof and mdat headers are transmitted separately.

Next, the transmission order of MMT packets when the MPU having the configuration described in FIG. 33 is transmitted will be described. FIG. 34 is a diagram for explaining the transmission order of MMT packets.

FIG. 34(a) shows the transmission order when MMT packets are transmitted in the MPU configuration order shown in FIG. Specifically, (a) of FIG. 34 shows MPU meta, MF meta #1, media data #1 (samples #1 to #3), MF meta #2, media data #2 (samples #4 to #6 ) is transmitted in this order.

In (b) of FIG. 34, MPU meta, media data #1 (samples #1 to #3), MF meta #1, media data #2 (samples #4 to #6), and MF meta #2 are transmitted in this order. Give an example.

In (c) of FIG. 34, media data #1 (samples #1 to #3), MPU meta, MF meta #1, media data #2 (samples #4 to #6), and MF meta #2 are transmitted in this order. Give an example.

MF meta #1 is generated using samples #1-#3 and MF meta #2 is generated using samples #4-#6. Therefore, when the transmission method of (a) of FIG. 34 is used, a delay due to encapsulation occurs in the transmission of sample data.

On the other hand, when the transmission methods of FIGS. 34(b) and 34(c) are used, the delay due to encapsulation is does not occur and the end-to-end delay can be reduced.

Also in the transmission order shown in FIG. The amount of delay due to encapsulation can be made smaller than in the case of encapsulation.

In addition to the method shown here, for example, the transmission device 15 may concatenate MF meta#1 and MF meta#2 and transmit them collectively at the end of the MPU. In this case, the MF metas of different movie fragments may be aggregated and stored in one MMT payload. Also, MF metas of different MPUs may be aggregated together and stored in the MMT payload.

[Receiving method when one MPU is composed of multiple movie fragments]
Here, an operation example of the receiving device 20 that receives and decodes MMT packets transmitted in the transmission order described in (b) of FIG. 34 will be described. 35 and 36 are diagrams for explaining such an operation example.

Receiving device 20 receives each of MMT packets including MPU meta, sample, and MF meta transmitted in the transmission order as shown in FIG. The sample data are concatenated in the order received.

Receiving device 20 concatenates data as shown in (1) of FIG. 36 at T1, which is the time at which MF meta #1 is received, to form MP4. As a result, receiving device 20 can acquire samples #1-#3 based on the information of MPU metadata and MF meta #1, and the PTS, DTS, offset, and size information included in MF meta. Decoding can be performed based on

Further, receiving device 20 concatenates data as shown in (2) of FIG. 36 at T2, which is the time at which MF meta #2 is received, to configure MP4. As a result, receiving device 20 can acquire samples #4 to #6 based on MPU metadata and MF meta #2 information, and based on MF meta PTS, DTS, offset, and size information. decryption can be performed. Further, the receiving device 20 concatenates the data as shown in (3) of FIG. 36 to configure the MP4, thereby obtaining the samples #1 to #6 based on the information of the MF meta #1 and MF meta #2. may be obtained.

By dividing one MPU into a plurality of movie fragments, the time until the first MF meta is acquired in the MPU is shortened, so the decoding start time can be shortened. Also, it is possible to reduce the buffer size for accumulating samples before decoding.

Note that the transmission device 15 is configured such that the time from transmitting (or receiving) the first sample in the movie fragment to transmitting (or receiving) the MF meta corresponding to the movie fragment is shorter than the initial_cpb_removal_delay specified by the encoder. The division unit of the movie fragment may be set as follows. By setting in this way, the receive buffer can follow the cpb buffer, and low-delay decoding can be achieved. In this case, absolute time based on initial_cpb_removal_delay can be used for PTS and DTS.

In addition, the transmission device 15 may split movie fragments at equal intervals, or may split subsequent movie fragments at shorter intervals than previous movie fragments. As a result, the receiving device 20 can always receive the MF meta including the information of the sample before decoding the sample, and continuous decoding is possible.

The following two methods can be used to calculate the absolute time of PTS and DTS.

(1) The absolute times of PTS and DTS are determined based on the reception time (T1 or T2) of MF meta #1 or MF meta #2 and the relative time of PTS and DTS included in the MF meta.

(2) The absolute time of PTS and DTS is determined based on the absolute time signaled from the sender, such as the MPU timestamp descriptor, and the relative time of PTS and DTS contained in the MF meta.

Also, (2-A) the absolute time signaled by the transmitting device 15 may be an absolute time calculated based on initial_cpb_removal_delay specified by the encoder.

Also, (2-B) the absolute time signaled by the transmitting device 15 may be an absolute time calculated based on the predicted value of the reception time of the MF meta.

Note that MF meta #1 and MF meta #2 may be repeatedly transmitted. By repeatedly transmitting the MF meta #1 and the MF meta #2, the receiving device 20 can obtain the MF meta again even if it cannot obtain the MF meta due to packet loss or the like.

An identifier indicating the order of the movie fragments can be stored in the payload header of the MFU containing the samples that make up the movie fragments. On the other hand, the MMT payload does not include an identifier indicating the order of the MF metas that make up the movie fragment. Therefore, the receiving device 20 identifies the order of the MF meta with the packet_sequence_number. Alternatively, the transmitting device 15 stores an identifier indicating which movie fragment the MF meta belongs to in control information (message, table, descriptor), MMT header, MMT payload header, or data unit header, and performs signaling. You may

Note that the transmitting device 15 transmits the MPU meta, the MF meta, and the samples in a predetermined transmission order, and the receiving device 20 performs reception processing based on the predetermined transmission order. may Alternatively, the transmitting device 15 may signal the transmission order, and the receiving device 20 may select (determine) the receiving process based on the signaling information.

The reception method as described above will be described with reference to FIG. FIG. 37 is a flow chart of the operation of the receiving method described in FIGS.

First, receiving device 20 determines (identifies) whether data included in the payload is MPU metadata, MF metadata, or sample data (MFU), based on the fragment type indicated in the MMT payload (S601). , S602). If the data is sample data, the receiving device 20 buffers the sample and waits for reception of MF metadata corresponding to the sample and start of decoding (S603).

On the other hand, in step S602, if the data is MF metadata, the receiving device 20 acquires sample information (PTS, DTS, position information, and size) from the MF metadata, and uses the acquired sample information as Obtain samples based on the PTS and DTS, decode and present the samples based on the PTS and DTS (S604).

Although not shown, when the data is MPU metadata, the MPU metadata includes initialization information necessary for decoding. Therefore, the receiving device 20 accumulates this and uses it for decoding the sample data in step S604.

When storing the received MPU data (MPU metadata, MF metadata, and sample data) in the storage device, the receiving device 20 rearranges the MPU configuration as described in FIG. 19 or FIG. then accumulate.

On the transmitting side, a packet sequence number is assigned to MMT packets having the same packet ID. At this time, the packet sequence numbers may be assigned after the MMT packets including the MPU metadata, the MF metadata, and the sample data are rearranged in the order of transmission, or the packet sequence numbers may be assigned in the order before rearrangement. good too.

If the packet sequence numbers are given in the order before rearrangement, the receiving device 20 can rearrange the data in the configuration order of the MPU based on the packet sequence numbers, which facilitates storage.

[Method for detecting the beginning of an access unit and the beginning of a slice segment]
A method for detecting the beginning of an access unit or the beginning of a slice segment based on the information in the MMT packet header and MMT payload header will be described.

Here, non-VCL NAL units (access unit delimiters, VPS, SPS, PPS, SEI, etc.) are collectively stored in the MMT payload as data units, and non-VCL NAL units are each data units, and data units are aggregated and stored in one MMT payload.

FIG. 38 is a diagram showing a case where non-VCL NAL units are individually treated as data units and aggregated.

In the case of FIG. 38, the beginning of the access unit is the MMT packet whose fragment_type value is MFU, and is the beginning data of the MMT payload including the data unit whose aggregation_flag value is 1 and whose offset value is 0. At this time, the Fragmentation_indicator value is 0.

Also, in the case of FIG. 38, the beginning of the slice segment is an MMT packet whose fragment_type value is MFU, and is the beginning data of the MMT payload whose aggregation_flag value is 0 and fragmentation_indicator value is 00 or 01.

FIG. 39 is a diagram showing a case where non-VCL NAL units are collectively used as data units. Field values of the packet header are as shown in FIG. 17 (or FIG. 18).

In the case of FIG. 39, the head data of the payload in the packet with the offset value of 0 is the head of the access unit.

Also, in the case of FIG. 39, the beginning of the slice segment has an Offset value different from 0, and the beginning data of the payload of the packet with the fragmentation indicator value of 00 or 01 becomes the beginning of the slice segment.

[Receiving process when packet loss occurs]
Normally, when transmitting MP4 format data in an environment where packet loss occurs, the receiving device 20 restores packets by ALFEC (Application Layer FEC), packet retransmission control, or the like.

However, when packet loss occurs when AL-FEC is not used in streaming such as broadcasting, packets cannot be restored.

After data is lost due to packet loss, the receiving device 20 needs to restart video and audio decoding. For this purpose, receiving device 20 needs to detect the beginning of an access unit or NAL unit and start decoding from the beginning of the access unit or NAL unit.

However, since there is no start code at the beginning of the NAL unit in MP4 format, receiving device 20 cannot detect the beginning of the access unit or the NAL unit even if the stream is analyzed.

FIG. 40 is a flow chart of the operation of the receiver 20 when packet loss occurs.

The receiving device 20 detects packet loss using the packet sequence number, packet counter, fragment counter, etc. in the header of the MMT packet or MMT payload (S701), and determines which packet has been lost from the context (S702). .

If it is determined that no packet loss has occurred (No in S702), the receiving device 20 constructs an MP4 file and decodes the access unit or NAL unit (S703).

When it is determined that a packet loss has occurred (Yes in S702), the receiving device 20 generates a NAL unit corresponding to the NAL unit in which the packet loss occurred using dummy data, and configures an MP4 file (S704). When the receiving device 20 puts dummy data in the NAL unit, the receiving device 20 indicates that the type of the NAL unit is dummy data.

Also, the receiving device 20 detects the beginning of the next access unit or NAL unit based on the methods described in FIGS. Decryption can be resumed (S705).

Note that when a packet loss occurs, the receiving device 20 may restart decoding from the beginning of the access unit and the NAL unit based on information detected based on the packet header, or the NAL unit of dummy data. Decoding may be restarted from the beginning of the access unit and the NAL unit based on the header information of the reconstructed MP4 file, including .

When accumulating an MP4 file (MPU), the receiving device 20 may separately acquire (replace) packet data (such as NAL units) lost due to packet loss from broadcasting or communication.

At this time, when the lost packet is acquired from the communication, the receiving device 20 sends the lost packet information (packet ID, MPU sequence number, packet sequence number, IP data flow number, IP address, etc.) to the server. and get the packet. Receiving device 20 may acquire not only the lost packet but also a group of packets before and after the lost packet at the same time.

[Method of constructing a movie fragment]
This section describes in detail how movie fragments are constructed.

As explained in FIG. 33, the number of samples forming a movie fragment and the number of movie fragments forming one MPU are arbitrary. For example, the number of samples forming a movie fragment and the number of movie fragments forming one MPU may be a fixed number, or may be determined dynamically.

Here, when the transmitting side (transmitting device 15) configures the movie fragments so as to satisfy the following conditions, low-delay decoding in the receiving device 20 can be guaranteed.

The conditions are as follows.

The transmitting device 15 transmits the sample data so that the receiving device 20 can always receive the MF meta including the information of the sample (Sample(i)) before the decoding time (DTS(i)) of the sample (Sample(i)). An MF meta is generated and transmitted using the divided unit as a movie fragment.

Specifically, the transmitting device 15 constructs a movie fragment using coded samples (including the i-th sample) before DTS(i).

As a method of dynamically determining the number of samples forming a movie fragment and the number of movie fragments forming one MPU so as to guarantee low-delay decoding, for example, the following method is used.

(1) At the start of decoding, the decoding time DTS(0) of sample Sample(0) at the beginning of GOP is the time based on initial_cpb_removal_delay. The transmitting device constructs the first movie fragment using the encoded samples at a time before DTS(0). Also, the transmitting device 15 generates MF metadata corresponding to the first movie fragment and transmits it at a time before DTS(0).

(2) The transmitting device 15 constructs movie fragments so that the above conditions are satisfied in subsequent samples as well.

For example, when the leading sample of a movie fragment is the kth sample, the MF meta of the movie fragment including the kth sample is transmitted by the decoding time DTS(k) of the kth sample. When the encoding completion time of the l-th sample is before DTS(k) and the encoding completion time of the (l+1)-th sample is after DTS(k), the transmission device 15 performs the k-th A movie fragment is constructed using the l-th sample from the samples of .

Note that the transmitting device 15 may form a movie fragment using samples from the kth sample up to the lth sample.

(3) After completing the encoding of the last sample of the MPU, the transmitting device 15 constructs a movie fragment using the remaining samples, generates MF metadata corresponding to the movie fragment, and transmits it.

Note that the transmitting device 15 may form a movie fragment using some of the encoded samples instead of forming the movie fragment using all the encoded samples.

Note that the above shows an example in which the number of samples forming a movie fragment and the number of movie fragments forming one MPU are dynamically determined based on the above conditions so as to guarantee low-delay decoding. rice field. However, the method of determining the number of samples and the number of movie fragments is not limited to such methods. For example, the number of movie fragments forming one MPU may be fixed at a predetermined value, and the number of samples may be determined so as to satisfy the above conditions. Alternatively, the number of movie fragments forming one MPU and the time to divide the movie fragment (or the code amount of the movie fragment) may be fixed to predetermined values, and the number of samples may be determined so as to satisfy the above conditions.

Also, if the MPU is split into multiple movie fragments, information indicating whether the MPU is split into multiple movie fragments, attributes of the split movie fragments, or attributes of the MF meta for the split movie fragments may be sent.

Here, the movie fragment attribute is information indicating whether the movie fragment is the head movie fragment of the MPU, the last movie fragment of the MPU, or other movie fragment.

The attribute of the MF meta is whether the MF meta is the MF meta corresponding to the first movie fragment of the MPU, the MF meta corresponding to the last movie fragment of the MPU, or other movie fragments. This is information indicating whether or not the MF is meta.

The transmission device 15 may store and transmit the number of samples forming a movie fragment and the number of movie fragments forming one MPU as control information.

[Operation of Receiving Device]
The operation of the receiving device 20 based on the movie fragment configured as described above will be described.

The receiving device 20 determines the absolute time of each of the PTS and DTS based on the absolute time signaled from the transmitting side, such as the MPU timestamp descriptor, and the relative time of the PTS and DTS contained in the MF meta.

The receiving device 20 performs the following processing based on the information as to whether the MPU is divided into a plurality of movie fragments, and if the MPU is divided, based on the attributes of the divided movie fragments. .

(1) When the movie fragment is the first movie fragment of the MPU, the receiving device 20 sets the absolute time of the PTS of the first sample included in the MPU timestamp descriptor and the relative time of the PTS and DTS included in the MF meta. is used to generate the absolute time of the PTS and DTS.

(2) If the movie fragment is not the first movie fragment of the MPU, the receiving device 20 uses the relative times of the PTS and DTS contained in the MF meta without using the information of the MPU timestamp descriptor to generate the PTS and DTS. to generate the absolute time of

(3) If the movie fragment is the last movie fragment of the MPU, the receiving device 20 resets the PTS and DTS calculation processing (relative time addition processing) after calculating the absolute times of the PTS and DTS of all samples. do. Note that the reset process may be performed in the movie fragment at the beginning of the MPU.

Receiving device 20 may determine whether the movie fragment is divided as follows. Also, the receiving device 20 may acquire the attribute information of the movie fragment as follows.

For example, receiving device 20 may determine whether the movie fragment is divided based on the identifier movie_fragment_sequence_number field value indicating the order of the movie fragment indicated in the MMTP (MMT Protocol) payload header.

Specifically, when the number of movie fragments included in one MPU is 1, the movie_fragment_sequence_number field value is 1, and the field value is 2 or more, , the MPU may be determined to be divided into a plurality of movie fragments.

Further, when the number of movie fragments included in one MPU is 1, the movie_fragment_sequence_number field value is 0, and the field value has a value other than 0, the MPU may be determined to be divided into multiple movie fragments.

Movie fragment attribute information may also be determined based on the movie_fragment_sequence_number.

It should be noted that even if the movie_fragment_sequence_number is not used, whether or not the movie fragment is divided and the attribute information of the movie fragment may be determined by counting the transmission of movie fragments and MF meta contained in one MPU.

With the configurations of the transmitting device 15 and the receiving device 20 as described above, the receiving device 20 can receive the movie fragment metadata at intervals shorter than the MPU, and can start decoding with low delay. In addition, it is possible to perform decoding with low delay using decoding processing based on the MP4 parsing method.

The reception operation when the MPU is divided into a plurality of movie fragments as described above will be described using a flowchart. FIG. 41 is a flow chart of the receiving operation when the MPU is divided into multiple movie fragments. This flowchart illustrates the operation of step S604 in FIG. 37 in more detail.

First, the receiving device 20 acquires MF metadata based on the data type indicated in the MMTP payload header when the data type is MF meta (S801).

Next, the receiving device 20 determines whether the MPU is divided into a plurality of movie fragments (S802), and if the MPU is divided into a plurality of movie fragments (Yes in S802), the received MF It is determined whether the metadata is metadata at the top of the MPU (S803). If the received MF metadata is the MF metadata at the beginning of the MPU (Yes in S803), the reception device 20 receives the absolute time of the PTS indicated in the MPU timestamp descriptor, and the PTS and PTS indicated in the MF metadata. The absolute time of the PTS and DTS is calculated from the relative time of the DTS (S804), and it is determined whether or not it is the last metadata of the MPU (S805).

On the other hand, if the received MF metadata is not the MF metadata at the beginning of the MPU (No in S803), the receiving device 20 does not use the information of the MPU timestamp descriptor and uses the relative values of the PTS and DTS indicated in the MF metadata. Using the time, the absolute time of the PTS and DTS is calculated (S808), and the process proceeds to step S805.

In step S805, if it is determined that the MF metadata is the last MPU metadata (Yes in S805), the receiving device 20 calculates the absolute times of the PTS and DTS of all samples, and then resets the PTS and DTS calculation processing. do. If it is determined in step S805 that the MF metadata is not the last MPU metadata (No in S805), the receiving device 20 ends the process.

Further, when it is determined in step S802 that the MPU is not divided into a plurality of movie fragments (No in S802), the receiving device 20 obtains sample data based on the MF metadata transmitted after the MPU. and determine the PTS and DTS (S807).

Then, although not shown, receiving device 20 finally performs decoding processing and presentation processing based on the determined PTS and DTS.

[Problems that occur when movie fragments are split and their solutions]
So far, we have described how to reduce the end-to-end delay by splitting movie fragments. From now on, we will explain problems that arise when movie fragments are divided, and their solutions.

First, as a background, the picture structure in encoded data will be described. FIG. 42 is a diagram showing an example of a picture prediction structure for each TemporalId when temporal scalability is realized.

In coding schemes such as MPEG-4 AVC and HEVC (High Efficiency Video Coding), scalability in the temporal direction (temporal scalability) is achieved by using B pictures (bidirectional reference prediction pictures) that can be referenced from other pictures. can.

The TemporalId shown in (a) of FIG. 42 is an identifier of the hierarchy of the coding structure, and the larger the TemporalId value, the deeper the hierarchy. A square block indicates a picture, and Ix in the block is an I picture (intra prediction picture), Px is a P picture (forward reference prediction picture), and Bx and bx are B pictures (bidirectional reference prediction pictures). show. The x in Ix/Px/Bx indicates the display order and represents the order in which the pictures are displayed. Arrows between pictures indicate reference relationships. For example, a picture of B4 indicates that a predicted image is generated using I0 and B8 as reference images. Here, one picture is prohibited from using another picture having a TemporalId greater than its own TemporalId as a reference picture. The reason why the hierarchy is defined is to have temporal scalability. For example, decoding all the pictures in FIG. is decoded, a video of 60 fps is obtained.

FIG. 43 is a diagram showing the relationship between the decoding time (DTS) and the display time (PTS) in each picture in FIG. For example, the picture I0 shown in FIG. 43 is displayed after the decoding of B4 is completed so that no gap occurs in decoding and display.

As shown in FIG. 43, when the prediction structure includes B pictures, the decoding order and the display order are different. (reorder) processing is required.

An example of a picture prediction structure in temporal scalability has been described above. Even when temporal scalability is not used, picture delay processing and reordering may be required depending on the prediction structure. be. FIG. 44 is a diagram showing an example of a prediction structure of pictures that require picture delay processing and reorder processing. Note that the numbers in FIG. 44 indicate the order of decoding.

As shown in FIG. 44, depending on the prediction structure, the leading sample in decoding order may differ from the leading sample in presentation order. is the fourth sample. Note that FIG. 44 shows an example of the prediction structure, and the prediction structure is not limited to such a structure. In other prediction structures, the first sample in decoding order may be different from the first sample in presentation order.

FIG. 45 is a diagram showing an example in which an MPU configured in MP4 format is divided into a plurality of movie fragments and stored in MMTP payloads and MMTP packets, similar to FIG. Note that the number of samples forming the MPU and the number of samples forming the movie fragment are arbitrary. For example, the number of samples constituting the MPU may be the number of samples in GOP units, and the number of samples half the number of samples in GOP units may be the movie fragment, and two movie fragments may be configured. One sample may be one movie fragment, and the samples constituting the MPU may not be divided.

FIG. 45 shows an example in which one MPU includes two movie fragments (moof box and mdat box), but one MPU may not include two movie fragments. The number of movie fragments included in one MPU may be three or more, or the number of samples included in the MPU. Also, the samples stored in the movie fragment may be divided into an arbitrary number of samples instead of equally divided samples.

The movie fragment metadata (MF metadata) includes information on the PTS, DTS, offset, and size of the samples included in the movie fragment. The PTS and DTS are extracted from the MF meta including the information of , and the decoding timing and presentation timing are determined.

From here on, for the sake of detailed explanation, the absolute value of the decoding time of the i sample is described as DTS(i), and the absolute value of the presentation time is described as PTS(i).

Specifically, the i-th sample information among the timestamp information stored in the moof in the MF meta includes the relative values of the decoding times of the i-th sample and the (i+1)-th sample, and the i-th are the relative values of the decoding time and the presentation time of the samples of , which are hereinafter referred to as DT(i) and CT(i).

Movie fragment metadata #1 includes DT(i) and CT(i) of samples #1-#3, and movie fragment metadata #2 includes DT(i) of samples #4-#6. ) and CT(i).

Also, the absolute value of the PTS of the MPU head access unit is stored in the MPU time stamp descriptor or the like, and the receiving device 20 calculates the PTS and DTS based on the PTS_MPU of the MPU head access unit and CT and DT. calculate.

FIG. 46 is a diagram for explaining the calculation method and problems of the PTS and DTS when the MPU is composed of samples #1 to #10.

FIG. 46(a) shows an example in which the MPU is not divided into movie fragments, FIG. 46(b) shows an example in which the MPU is divided into two movie fragments of 5 sample units, and FIG. ) shows an example in which the MPU is divided into 10 movie fragments on a sample-by-sample basis.

As described in FIG. 45, when PTS and DTS are calculated using the MPU timestamp descriptor and the timestamp information (CT and DT) in MP4, the first sample in the presentation order in FIG. is fourth in decoding order. Therefore, the PTS stored in the MPU timestamp descriptor is the PTS (absolute value) of the fourth sample in decoding order. Note that this sample is hereinafter referred to as an A sample. Also, the top sample in decoding order is called a B sample.

Since the absolute time information related to the time stamp is only the information of the MPU time stamp descriptor, the receiving device 20 calculates the PTS (absolute time) and DTS (absolute time) of the other samples until the A sample arrives. Can not. The receiving device 20 cannot calculate the PTS and DTS of B samples either.

In the example of (a) of FIG. 46, the A sample is included in the same movie fragment as the B sample, and stored in one MF meta. Therefore, receiving device 20 can determine the DTS of B samples immediately after receiving the MF meta.

In the example of FIG. 46(b), the A sample is included in the same movie fragment as the B sample, and stored in one MF meta. Therefore, receiving device 20 can determine the DTS of B samples immediately after receiving the MF meta.

In the example of (c) in FIG. 46, the A sample is included in a different movie fragment from the B sample. Therefore, the receiving device 20 cannot determine the DTS of the B sample until after receiving the MF meta including CT and DT of the movie fragment including the A sample.

Therefore, in the case of the example of (c) in FIG. 46, receiving device 20 cannot start decoding immediately after arrival of B samples.

Thus, when a movie fragment including B samples does not include A samples, receiving device 20 decodes B samples only after receiving MF meta related to movie fragments including A samples. Cannot start.

When the first sample in the presentation order and the first sample in the decoding order do not match, the movie fragment is divided until the A sample and the B sample are no longer stored in the same movie fragment. do. Also, this problem occurs regardless of whether the MF meta is backward or forward.

In this way, when the first sample in the presentation order and the first sample in the decoding order do not match, if the A sample and the B sample are not stored in the same movie fragment, the B sample is immediately received. DTS cannot be determined at Therefore, the transmitting device 15 separately transmits information that enables the receiving side to calculate the DTS (absolute value) of B samples or the DTS (absolute value) of B samples. Such information may be transmitted using control information, packet headers, or the like.

Receiving device 20 calculates the DTS (absolute value) of B samples using such information. FIG. 47 is a flow chart of reception operation when DTS is calculated using such information.

The receiving device 20 receives the movie fragment at the head of the MPU (S901), and determines whether or not the A sample and the B sample are stored in the same movie fragment (S902). If they are stored in the same movie fragment (Yes in S902), the receiving device 20 does not use the DTS (absolute time) of B samples, but uses only MF meta information to calculate the DTS, and starts decoding. (S904). Note that in step S904, the receiving device 20 may determine the DTS using the DTS of B samples.

On the other hand, if the A sample and the B sample are not stored in the same movie fragment in step S902 (No in S902), the receiving device 20 obtains the DTS (absolute time) of the B sample, determines the DTS, and decodes it. is started (S903).

In the above description, the absolute value of the decoding time and the absolute value of the presentation time of each sample are calculated using the MF meta (time stamp information stored in the moof in MP4 format) in the MMT standard. Although an example has been described, it goes without saying that the MF meta may be implemented by replacing the absolute value of the decoding time of each sample and arbitrary control information that can be used to calculate the absolute value of the presentation time. An example of such control information is the relative value CT(i) between the decoding times of the i-th sample and the (i+1)-th sample, and the presentation time of the i-th sample and the (i+1)-th sample. Control information replaced with relative values, the relative value CT(i) of the decoding times of the i-th sample and the (i+1)-th sample, and the relative value of the presentation times of the i-th sample and the (i+1)-th sample There is control information that includes both.

(Embodiment 3)
[overview]
In Embodiment 3, a content transmission method and data structure when content such as video, audio, subtitles, and data broadcasting are transmitted by broadcasting will be described. That is, a content transmission method and data structure specialized for reproduction of a broadcast stream will be described.

In Embodiment 3, an example in which the MMT method (hereinafter also simply referred to as MMT) is used as the multiplexing method will be described, but other multiplexing methods such as MPEG-DASH or RTP may be used. .

First, details of a method of storing a data unit (DU: Data Unit) in a payload in MMT will be described. FIG. 48 is a diagram for explaining a method of storing a data unit in a payload in MMT.

In MMT, a transmitting device stores a part of data constituting an MPU in an MMTP payload as a data unit, attaches a header, and transmits the data. The header includes an MMTP payload header and an MMTP packet header. Note that the unit of the data unit may be the NAL unit unit or the sample unit. If the MMTP packet is scrambled, the payload is scrambled.

(a) of FIG. 48 shows an example in which a transmitting device aggregates a plurality of data units and stores them in one payload. In the example of (a) of FIG. 48, a data unit header (DUH: Data Unit Header) and a data unit length (DUL: Data Unit Length) are added to the beginning of each of a plurality of data units, and the data unit header and A plurality of data units to which data unit lengths are assigned are collectively stored in the payload.

FIG. 48(b) shows an example of storing one data unit in one payload. In the example of FIG. 48(b), a data unit header is attached to the head of the data unit and stored in the payload. FIG. 48(c) shows an example in which one data unit is divided, a data unit header is added to the divided data unit, and the divided data unit is stored in the payload.

The data unit includes timed-MFU, which is media containing information related to synchronization such as video, audio, or subtitles, non-timed-MFU, which is media which does not contain information related to synchronization such as files, MPU metadata, MF metadata, etc. and the data unit header is determined according to the type of data unit. Note that there is no data unit header in MPU metadata and MF metadata. Also, although the transmitting device cannot aggregate different types of data units in principle, it may be defined so that it can aggregate different types of data units. For example, when the size of the MF metadata is small, such as when each sample is divided into movie fragments, the number of packets can be reduced by aggregating the MF metadata and media data, and the transmission capacity can be reduced. can also

When the data unit is an MFU, some information of the MPU, such as information for configuring the MPU (MP4), is stored as a header.

For example, the timed-MFU header includes movie_fragment_sequence_number, sample_number, offset, priority, and dependency_counter, and the non-timed-MFU header includes item_iD. The meaning of each field is indicated in standards such as ISO/IEC23008-1 or ARIB STD-B60. The meaning of each field defined in such standards will be described below.

movie_fragment_sequence_number indicates the sequence number of the movie fragment to which the MFU belongs, and is also indicated in ISO/IEC14496-12.

sample_number indicates the sample number to which the MFU belongs, which is also indicated in ISO/IEC14496-12.

offset indicates, in bytes, the amount of offset of the MFU in the sample to which the MFU belongs.

Priority indicates the relative importance of the MFU in the MPU to which the MFU belongs, and an MFU with a higher priority number is more important than an MFU with a lower priority number.

dependency_counter indicates the number of MFUs whose decoding process depends on the MFU (that is, the number of MFUs whose decoding process cannot be performed unless this MFU is decoded). For example, when the MFU is HEVC and a B picture or P picture refers to an I picture, the B picture or P picture cannot be decoded unless the I picture is decoded.

Therefore, if the MFU is in sample units, the dependency_counter in the I-picture MFU indicates the number of pictures that refer to the I-picture. When the MFU is in units of NAL units, the dependency_counter in the MFU belonging to the I-picture indicates the number of NAL units belonging to the picture that references the I-picture. Furthermore, in the case of a temporally layered-encoded video signal, the MFU of the enhancement layer depends on the MFU of the base layer, so the dependency_counter in the MFU of the base layer indicates the number of MFUs of the enhancement layer. This field can only be generated after the number of dependent MFUs has been determined.

item_iD indicates an identifier that uniquely identifies an item.

The method of storing the control information in the payload in MMT is the same as the method of storing the data unit in the payload, and can be explained by replacing the data unit in FIG. 48 with the control information and the data unit length with the control information length. Note that there is no information corresponding to the data unit header.

[MP4 non-support mode]
As described with reference to FIGS. 19 and 21, methods for the transmitting device to transmit MPU in MMT include a method of transmitting MPU metadata or MF metadata before or after media data, and a method of transmitting only media data. There is a way to send Also, in the receiving device, there is a method of decoding using a receiving device and a receiving method conforming to MP4, and a method of decoding without using a header.

As a data transmission method specialized for broadcast stream reproduction, for example, there is a transmission method that does not support MP4 reconstruction in the receiving device.

A transmission method that does not support MP4 reconstruction in the receiving device is, for example, a method that does not transmit metadata (MPU metadata and MF metadata) as shown in (b) of FIG. 21 . In this case, the field value of the fragment type (information indicating the type of data unit) included in the MMTP packet is fixed to 2 (=MFU).

If metadata is not sent, as explained above, the received data cannot be decoded as MP4 by an MP4-compliant receiving device, but can be decoded without using the metadata (header). is.

Therefore, metadata is not necessarily essential information for broadcast stream decoding and reproduction. Similarly, the data unit header information in the timed-MFU explained in FIG. 48 is information for reconstructing MP4 in the receiving device. Since there is no need to reconstruct MP4 in broadcast stream reproduction, the information of the data unit header in the timed-MFU (hereinafter also referred to as timed-MFU header) is not necessarily required for broadcast stream reproduction.

The receiving device can easily reconstruct MP4 by using metadata and information for reconstructing MP4 in the data unit header (hereinafter also referred to as MP4 configuration information). However, the receiving device cannot easily reconstruct MP4 even if only one of the metadata and the MP4 configuration information in the data unit header is transmitted. There is little advantage in transmitting only one of the metadata and information for reconstructing MP4, and generating and transmitting unnecessary information causes an increase in processing and a decrease in transmission efficiency.

Therefore, the transmitting device controls the data structure and transmission of the MP4 configuration information using the following method. The sending device determines whether to indicate MP4 configuration information in the data unit header based on whether metadata is transmitted. Specifically, the transmitting device indicates the MP4 configuration information in the data unit header when the metadata is transmitted, and does not indicate the MP4 configuration information in the data unit header when the metadata is not transmitted.

As a method for not showing the MP4 configuration information in the data unit header, for example, the following method can be used.

1. The transmitting device sets the MP4 configuration information as reserved and does not operate it. This makes it possible to reduce the amount of processing on the transmission side (the amount of processing by the transmitting device) that generates the MP4 configuration information.

2. The transmitting device deletes the MP4 configuration information and compresses the header. As a result, it is possible to reduce the amount of processing on the transmission side that generates the MP4 configuration information, and to reduce the transmission capacity.

Note that, when deleting the MP4 configuration information and compressing the header, the transmitting device may indicate a flag indicating that the MP4 configuration information has been deleted (compressed). Flags are indicated in headers (MMTP packet header, MMTP payload header, data unit header), control information, or the like.

Information on whether or not to transmit metadata may be determined in advance, or may be separately signaled in the header or control information and transmitted to the receiving device.

For example, an MFU header may store information as to whether metadata corresponding to the MFU is being transmitted.

On the other hand, the receiving device can determine whether MP4 configuration information is indicated based on whether metadata is transmitted.

Here, if the data transmission order (for example, the order of MPU metadata, MF metadata, and media data) is fixed, the receiving device determines whether the metadata is received before the media data. can be determined.

If the MP4 configuration information is indicated, the receiving device can use the MP4 configuration information for MP4 reconstruction. Alternatively, the receiving device can use the MP4 configuration information for other access units, detection of the beginning of NAL units, and the like.

Note that the MP4 configuration information may be all or part of the timed-MFU header.

Similarly, in the non-timed-MFU header, the transmitting device determines whether or not metadata is transmitted in the item in the non-timed-MFU header.
id may be determined.

The transmitting device may indicate MP4 configuration information only in either timed-MFU or non-timed-MFU. If only one of them shows MP4 configuration information, the transmitting device determines whether to show MP4 configuration information based on whether metadata is transmitted and whether it is timed-MFU or non-timed-MFU. do. At the receiving device, it can be determined whether metadata is transmitted and whether MP4 configuration information is indicated based on the timed/non-timed flag.

Note that in the above description, the sending device decided whether to indicate MP4 configuration information based on whether metadata (both MPU metadata and MF metadata) was transmitted. However, the transmitting device may not indicate the MP4 configuration information if part of the metadata (either MPU metadata or MF metadata) is not transmitted.

Also, the transmitting device may determine whether to indicate MP4 configuration information based on other information than metadata.

For example, modes such as MP4 support mode/MP4 non-support mode are defined, and the transmitting device indicates MP4 configuration information in the data unit header in the case of MP4 support mode, and the data unit in the case of MP4 non-support mode. MP4 configuration information may not be indicated in the unit header. Also, in the case of the MP4 support mode, the transmission device transmits the metadata and indicates the MP4 configuration information in the data unit header, and in the case of the MP4 non-support mode, the transmission device transmits the data unit header without transmitting the metadata. may not indicate the MP4 configuration information.

[Transmitter operation flow]
Next, the operational flow of the transmitting device will be described. FIG. 49 is an operational flow of the transmitting device.

The transmitting device first determines whether to transmit metadata (S1001). When the transmitting device determines to transmit the metadata (Yes in S1002), the transmitting device moves to step S1003, generates MP4 configuration information, stores it in the header, and transmits it (S1003). In this case, the sending device also generates and transmits metadata.

On the other hand, if the transmitting device determines not to transmit the metadata (No in S1002), the transmitting device does not generate MP4 configuration information and transmits it without storing it in the header (S1004). In this case, the sending device does not generate or transmit metadata.

Whether or not to transmit metadata in step S1001 may be determined in advance. may be determined based on

[Operation Flow of Receiving Device]
Next, the operational flow of the receiving device will be described. FIG. 50 is an operation flow of the receiving device.

The receiving device first determines whether metadata has been transmitted (S1101). Whether metadata is being transmitted can be determined by monitoring the fragment type in the MMTP packet payload. Also, it may be determined in advance whether or not it is transmitted.

If the receiving device determines that metadata has been transmitted (Yes in S1102), it reconstructs the MP4 and executes decoding processing using the MP4 configuration information (S1103). On the other hand, if it is determined that the metadata has not been transmitted (No in S1102), the MP4 reconstruction process is not performed and the decoding process is executed without using the MP4 configuration information (S1104).

It should be noted that the receiving device can detect a random access point, detect the beginning of an access unit, detect the beginning of an NAL unit, etc., without using MP4 configuration information, using the methods described above. , detection of packet loss, and recovery from packet loss.

For example, as shown in FIG. 38, when non-VCL NAL units are individually treated as data units and aggregated, the head of the access unit is the head data of the MMT payload whose aggregation_flag value is 1. At this time, the Fragmentation_indicator value is 0.

Also, the beginning of the slice segment is the beginning data of the MMT payload with the aggregation_flag value of 0 and the fragmentation_indicator value of 00 or 01.

The receiving device can detect the beginning of the access unit and the slice segment based on the above information.

Note that the receiving device analyzes the NAL unit header in a packet including the head of the data unit whose fragmentation_indicator value is 00 or 01, and determines that the NAL unit type is an AU delimiter and the NAL unit type is a slice segment. It may be detected that

[Broadcast simple mode]
So far, as a method of transmitting data specialized for reproducing broadcast streams, a method that does not support MP4 configuration information in the receiving device has been described, but the method of transmitting data specialized for reproducing broadcast streams is not limited to this. do not have.

For example, the following method may be used as a data transmission method specialized for broadcast stream reproduction.

- The transmitter does not have to use AL-FEC in a fixed reception environment for broadcasting. When AL-FEC is not used, FEC_type in the MMTP packet header is always fixed to 0.

• The transmitting device may always use AL-FEC in the mobile reception environment of broadcasting and in communication UDP transmission mode. The FEC_type in the MMTP packet header is always 0 or 1 when AL-FEC is used.

• The sender may not perform bulk transmission of assets. If the bulk transfer of assets is not performed, location_infolocation, which indicates the number of transfer locations of assets within the MPT, may be fixed to 1.

• The sending device may not perform hybrid transmissions of assets, programs and messages.

Also, for example, a broadcast simple mode is specified, and if the transmission device is in the broadcast simple mode, the MP4 non-support mode is selected, or the data transmission method specialized for broadcast stream reproduction described above is used. may be Whether or not it is the broadcasting simple mode may be determined in advance, and the transmitting device may store a flag indicating the broadcasting simple mode as control information and transmit it to the receiving device.

In addition, the transmission device determines whether the MP4 non-support mode (whether metadata is transmitted) as described in FIG. A data transmission method specialized for broadcast stream reproduction shown in 1 may be used.

When the broadcasting simple mode is set, the receiving device can perform decoding processing without reconstructing MP4, considering that the mode is MP4 non-support mode.

Also, in the case of the broadcast simple mode, the receiving device can determine that the function is specialized for broadcasting, and perform reception processing specifically for broadcasting.

As a result, in the case of the broadcast simple mode, by using only functions specialized for broadcasting, it is possible not only to reduce unnecessary processing for the transmitting device and the receiving device, but also to prevent unnecessary information from being compressed and transmitted. can reduce transmission overhead.

Note that when the MP4 non-support mode is used, hint information that supports storage methods other than the MP4 configuration may be indicated.

Storage methods other than the MP4 configuration include, for example, a method of directly storing MMT packets and IP packets, and a method of converting MMT packets into MPEG-2 TS packets.

Note that in the case of the MP4 non-support mode, a format that does not conform to the MP4 configuration may be used.

For example, in the MP4 non-support mode, the data stored in the MFU may be in a format with a byte start code instead of a format in which the size of the NAL unit is attached to the beginning of the NAL unit in MP4 format. good.

In MMT, the asset type indicating the type of asset is described in 4CC registered in MP4REG (http://www.mp4ra.org), and when HEVC is used as the video signal, 'HEV1' or 'HVC1' is used. be done. 'HVC1' is a format that may include a parameter set in the sample, and 'HEV1' is a format that does not include a parameter set in the sample but includes a parameter set in the sample entry in the MPU metadata.

In the case of broadcast simple mode or MP4 non-support mode, it may be specified that the parameter set is always included in the sample if MPU metadata and MF metadata are not transmitted. Also, it may be specified that the asset type always takes the form of 'HVC1' regardless of whether 'HEV1' or 'HVC1' is indicated.

[Supplement 1: Transmitter]
As described above, when the metadata is not transmitted, the MP4 configuration information is reserved, and the non-operating transmission device can be configured as shown in FIG. FIG. 51 is a diagram showing an example of a specific configuration of a transmission device.

Transmission device 300 includes encoding section 301 , addition section 302 , and transmission section 303 . Each of the encoding unit 301, the adding unit 302, and the transmitting unit 303 is implemented by, for example, a microcomputer, a processor, or a dedicated circuit.

The encoding unit 301 encodes a video signal or an audio signal to generate sample data. The sample data are specifically data units.

The adding unit 302 adds header information including MP4 configuration information to sample data, which is data obtained by encoding a video signal or an audio signal. The MP4 configuration information is information for reconstructing the sample data as an MP4 format file on the receiving side, and is information whose contents differ depending on whether the presentation time of the sample data is determined.

As described above, the addition unit 302 adds movie_fragment_sequence_number, sample_number, offset, priority , and MP4 configuration information such as dependency_counter.

On the other hand, the adding unit 302 adds MP4 configuration information such as item_id to the header (header information) of timed-MFU, which is an example of sample data whose presentation time is not defined (sample data that does not include information about synchronization). Include.

Then, when the transmission unit 303 does not transmit the metadata corresponding to the sample data (for example, in the case of (b) in FIG. 21), the addition unit 302 determines whether the presentation time of the sample data is set. Header information that does not include MP4 configuration information is added to the sample data depending on whether the MP4 configuration information is included.

Specifically, when the presentation time of the sample data is determined, the adding unit 302 adds header information that does not include the first MP4 configuration information to the sample data, and the presentation time of the sample data is determined. If not, header information including second MP4 configuration information is added to the sample data.

For example, as shown in step S1004 in FIG. 49, when the transmission unit 303 does not transmit the metadata corresponding to the sample data, the addition unit 302 sets the MP4 configuration information to reserved (fixed value). MP4 configuration information is not substantially generated and substantially not stored in the header (header information). The metadata includes MPU metadata and movie fragment metadata.

The transmitting unit 303 transmits the sample data to which the header information is attached. More specifically, the transmitting unit 303 packetizes the sample data to which the header information is attached by the MMT method and transmits the packet.

As described above, the transmission and reception methods dedicated to reproducing broadcast streams do not require the receiving device to reconstruct the data units into MP4. If the receiving device does not need to reconfigure to MP4, the processing of the transmitting device is reduced by not generating unnecessary information such as MP4 configuration information.

On the other hand, although the transmitting device must transmit the necessary information, it is necessary to maintain consistency with the standard so that unnecessary additional information does not have to be transmitted separately.

According to the configuration of the transmitting apparatus 300, by setting the area in which the MP4 configuration information is stored to a fixed value, the MP4 configuration information is not transmitted, and only the necessary information is transmitted based on the standard. This has the effect of eliminating the need to send additional information. That is, it is possible to reduce the configuration of the transmission device and the processing amount of the transmission device. In addition, since unnecessary data is not transmitted, transmission efficiency can be improved.

[Supplement 2: Receiving device]
Also, a receiving device corresponding to the transmitting device 300 may be configured as shown in FIG. 52, for example. FIG. 52 is a diagram showing another example of the configuration of the receiver.

Receiving device 400 includes receiving section 401 and decoding section 402 . The receiving unit 401 and decoding unit 402 are implemented by, for example, a microcomputer, processor, or dedicated circuit.

The receiving unit 401 receives sample data, which is data in which a video signal or an audio signal is encoded, and is attached with header information including MP4 configuration information for reconstructing the sample data as an MP4 format file. Receive data.

The decoding unit 402 decodes the sample data without using the MP4 configuration information when the metadata corresponding to the sample data is not received by the receiving unit and the presentation time of the sample data is determined. do.

For example, the decoding unit 402 executes decoding processing without using the MP4 configuration information when the receiving unit 401 does not receive metadata corresponding to the sample data as shown in step S1104 in FIG.

Thereby, the configuration of the receiving device 400 and the amount of processing in the receiving device 400 can be reduced.

(Other embodiments)
Although the transmission method, reception method, transmission device, and reception device according to the embodiments have been described above, the present invention is not limited to these embodiments.

Further, each processing unit included in the transmitting device and the receiving device according to the above embodiments is typically implemented as an LSI, which is an integrated circuit. These may be made into one chip individually, or may be made into one chip so as to include part or all of them.

Further, circuit integration is not limited to LSIs, and may be realized by dedicated circuits or general-purpose processors. An FPGA (Field Programmable Gate Array) that can be programmed after the LSI is manufactured, or a reconfigurable processor that can reconfigure connections and settings of circuit cells inside the LSI may be used.

In each of the above embodiments, each component may be implemented by dedicated hardware or by executing a software program suitable for each component. Each component may be realized by reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory by a program execution unit such as a CPU or processor.

In other words, the transmitter and receiver comprise processing circuitry and storage electrically coupled to the processing circuitry (accessible by the control circuitry). The processing circuitry includes at least one of dedicated hardware and program execution. Also, if the processing circuit includes a program execution unit, the storage device stores a software program executed by the program execution unit. The processing circuit uses the storage device to execute the transmission method or the reception method according to the above embodiments.

Furthermore, the present invention may be the above software program or a non-transitory computer-readable recording medium on which the above program is recorded. It goes without saying that the above program can be distributed via a transmission medium such as the Internet.

In addition, the numbers used above are all examples for specifically describing the present invention, and the present invention is not limited to the illustrated numbers.

Also, the division of functional blocks in the block diagram is an example, and a plurality of functional blocks can be realized as one functional block, one functional block can be divided into a plurality of functional blocks, and some functions can be moved to other functional blocks. may Moreover, single hardware or software may process the functions of a plurality of functional blocks having similar functions in parallel or in a time-sharing manner.

Also, the order in which the steps included in the above transmission method or reception method are executed is for illustrative purposes in order to specifically describe the present invention, and orders other than the above may be used. Also, some of the above steps may be executed concurrently (in parallel) with other steps.

The transmission method, reception method, transmission device, and reception device according to one or more aspects of the present invention have been described above based on the embodiments, but the present invention is not limited to these embodiments. do not have. As long as it does not depart from the spirit of the present invention, one or more modifications of the present embodiment that can be conceived by a person skilled in the art, or a form constructed by combining the components of different embodiments may be included within the scope of the embodiments.

INDUSTRIAL APPLICABILITY The present invention can be applied to devices or devices that perform media transport such as video data and audio data.

15, 100, 300 transmitting device 16, 101, 301 encoding unit 17, 102 multiplexing unit 18, 104 transmitting unit 20, 200, 400 receiving device 21 packet filtering unit 22 transmission order type determining unit 23 random access unit 24, 212 Control information acquisition unit 25 Data acquisition unit 26 Calculation unit 27 Initialization information acquisition unit 28, 206 Decoding command unit 29, 204A, 204B, 204C, 204D, 402 Decoding unit 30 Presentation unit 201 Tuner 202 Demodulation unit 203 Demultiplexing unit 205 Display unit 211 Type determination unit 213 Slice information acquisition unit 214 Decoded data generation unit 302 Addition unit 303 Transmission unit 401 Reception unit

Claims (4)

a adding step of adding header information to the sample data, which is encoded data;
a transmitting step of transmitting the sample data to which the header information is added;
If the presentation time of the sample data is not defined,
In the adding step, adding the header information including MP4 configuration information for reconstructing the sample data as an MP4 format file on the receiving side to the sample data;
When the presentation time of the sample data is set,
transmitting control information for decoding the sample data separately from the sample data;
In the adding step, adding header information in which the MP4 configuration information is invalidated to the sample data;
In the transmitting step, packetizing and transmitting the sample data to which the header information is added according to MMT (MPEG Media Transport),
At least one of movie_fragment_sequence_number, sample_number, offset, priority, and dependency_counter in an MMTP (MMT Protocol) payload is included as the MP4 configuration information in the header information attached to the sample data whose presentation time is determined Transmission method . a receiving step of receiving the sample data, which is encoded data, to which header information has been added;
a decoding step of decoding the sample data;
If the presentation time of the sample data is not defined,
In the decoding step, using MP4 configuration information for reconstructing the sample data as an MP4 format file included in the header information, decoding the sample data,
If the presentation time of the sample data is set,
In the decoding step, performing decoding processing of the sample data using control information regarding decoding of the sample data, which is received separately from the sample data;
The sample data to which the header information has been added is packetized by the MMT method,
At least one of movie_fragment_sequence_number, sample_number, offset, priority, and dependency_counter in the MMTP payload is included as the MP4 configuration information in the header information attached to the sample data whose presentation time is determined. an adding unit that adds header information to sample data that is encoded data;
A transmitting unit that transmits the sample data to which the header information is added,
If the presentation time of the sample data is not defined,
The adding unit adds the header information including MP4 configuration information for reconstructing the sample data as an MP4 format file on the receiving side to the sample data,
If the presentation time of the sample data is set,
transmitting control information for decoding the sample data separately from the sample data;
The adding unit adds header information in which the MP4 configuration information is invalidated to the sample data,
The transmitting unit packetizes and transmits the sample data to which the header information is attached by an MMT method,
At least one of movie_fragment_sequence_number, sample_number, offset, priority, and dependency_counter in the MMTP payload is included as the MP4 configuration information in the header information added to the sample data whose presentation time is determined. a receiving unit that receives sample data that is encoded data and that is attached with header information;
a decoding unit that decodes the sample data;
If the presentation time of the sample data is not defined,
The decoding unit uses MP4 configuration information for reconstructing the sample data as an MP4 format file, which is included in the header information, and decodes the sample data,
If the presentation time of the sample data is set,
the decoding unit decodes the sample data using control information related to decoding of the sample data, which is received separately from the sample data;
The sample data to which the header information has been added is packetized by the MMT method,
At least one of movie_fragment_sequence_number, sample_number, offset, priority, and dependency_counter in the MMTP payload is included as the MP4 configuration information in the header information attached to the sample data whose presentation time is defined.

Images