JP7457930B2 - Transmission method, reception method, transmission device and reception device - Google Patents

Description

The present invention relates to a transmitting method, a receiving method, a transmitting device, and a receiving device.

As broadcasting and communication services become more sophisticated, the introduction of ultra-high-definition video content such as 8K (7680 x 4320 pixels: hereinafter also referred to as 8K4K) and 4K (3840 x 2160 pixels: hereinafter also referred to as 4K2K) is being considered. has been done. The receiving device needs to decode and display the encoded data of the received ultra-high-definition video in real time, but video with resolutions such as 8K requires a large processing load during decoding. It is difficult to decode moving images in real time with one decoder. Therefore, methods are being considered to reduce the processing load per decoder and achieve real-time processing by parallelizing the decoding process using a plurality of decoders.

The encoded data is multiplexed based on a multiplexing method such as MPEG-2 TS (Transport Stream) or MMT (MPEG Media Transport) before being transmitted. For example, Non-Patent Document 1 discloses a technique for transmitting encoded media data packet by packet according to MMT.

Information technology - High efficiency coding and media delivery in heterogeneous environments - Part 1: MPEG media transport (MMT), ISO/IEC DIS 23008-1

MMT is a system that supports hybrid distribution using broadcasting and communication, is also capable of transmitting MP4 format media, and has various functions. However, when MMT is used for data transmission when playing a broadcast stream in a receiving device, MMT has more functions than necessary, which complicates the configuration of the transmitting device and receiving device, and increases the amount of processing. It may be stored away. Furthermore, transmission bandwidth may be wasted due to unnecessary data being transmitted.

The present invention provides a transmitting device and a receiving device that can simplify the configuration of the device and reduce the processing amount of the device when transmitting data using a method such as MMT.

In order to achieve the above object, a transmission method according to one aspect of the present invention includes an assignment step of assigning header information to sample data, which is data in which a video signal or an audio signal is encoded, the header information including MP4 configuration information for reconstructing the sample data as an MP4 format file on the receiving side, the content of which varies depending on whether the presentation time of the sample data is set or not, and a transmission step of transmitting the sample data to which the header information has been assigned, and in the assignment step, if metadata corresponding to the sample data is not transmitted in the transmission step, header information that does not include the MP4 configuration information is assigned to the sample data depending on whether the presentation time of the sample data is set or not.

Further, the reception method according to one aspect of the present invention includes sample data that is data in which a video signal or an audio signal is encoded, and MP4 configuration information for reconstructing the sample data as an MP4 format file. a receiving step of receiving sample data to which header information that does not include is added; and a case where metadata corresponding to the sample data is not received in the receiving step, and a presentation time of the sample data is determined. and decoding the sample data without using the MP4 configuration information.

Note that these general or specific aspects may be realized by a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM. and a recording medium may be used in any combination.

INDUSTRIAL APPLICABILITY The present invention can simplify the configuration of a device and reduce the processing amount of the device when transmitting data using a method such as MMT.

FIG. 1 is a diagram showing an example of dividing a picture into slice segments. FIG. 2 is a diagram showing an example of a PES packet string in which picture data is stored. FIG. 3 is a diagram illustrating an example of dividing a picture according to the first embodiment. FIG. 4 is a diagram illustrating an example of picture division according to a comparative example of the first embodiment. FIG. 5 is a diagram illustrating an example of data of an access unit according to the first embodiment. FIG. 6 is a block diagram of a transmitting device according to the first embodiment. FIG. 7 is a block diagram of a receiving device according to the first embodiment. FIG. 8 is a diagram illustrating an example of an MMT packet according to the first embodiment. FIG. 9 is a diagram showing another example of the MMT packet according to the first embodiment. FIG. 10 is a diagram illustrating an example of data input to each decoding unit according to the first embodiment. FIG. 11 is a diagram illustrating an example of an MMT packet and header information according to the first embodiment. FIG. 12 is a diagram showing another example of data input to each decoding unit according to the first embodiment. FIG. 13 is a diagram showing an example of division of a picture according to the first embodiment. FIG. 14 is a flowchart of the transmission method according to the first embodiment. FIG. 15 is a block diagram of a receiving device according to the first embodiment. FIG. 16 is a flowchart of the receiving method according to the first embodiment. FIG. 17 is a diagram illustrating an example of an MMT packet and header information according to the first embodiment. FIG. 18 is a diagram showing an example of an MMT packet and header information according to the first embodiment. FIG. 19 is a diagram showing the configuration of the MPU. FIG. 20 is a diagram showing the structure of MF metadata. FIG. 21 is a diagram for explaining the data transmission order. FIG. 22 is a diagram showing an example of a method for performing decoding without using header information. FIG. 23 is a block diagram of a transmitting device according to the second embodiment. FIG. 24 is a flowchart of the transmission method according to the second embodiment. FIG. 25 is a block diagram of a receiving device according to Embodiment 2. FIG. 26 is a flowchart of the operation for specifying the MPU head position and NAL unit position. FIG. 27 is a flowchart of an operation of obtaining initialization information based on a transmission order type and decoding media data based on the initialization information. FIG. 28 is a flowchart of the operation of the receiving device when the low delay presentation mode is provided. FIG. 29 is a diagram illustrating an example of the transmission order of MMT packets when auxiliary data is transmitted. FIG. 30 is a diagram for explaining an example in which the transmitting device generates auxiliary data based on the configuration of moof. FIG. 31 is a diagram for explaining reception of auxiliary data. FIG. 32 is a flowchart of a receiving operation using auxiliary data. FIG. 33 is a diagram showing the configuration of an MPU composed of a plurality of movie fragments. FIG. 34 is a diagram for explaining the transmission order of MMT packets when the MPU having the configuration shown in FIG. 33 is transmitted. FIG. 35 is a first diagram for explaining an example of the operation of the receiving device when one MPU is composed of a plurality of movie fragments. FIG. 36 is a second diagram for explaining an example of the operation of the receiving device when one MPU is composed of a plurality of movie fragments. FIG. 37 is a flowchart showing the operation of the receiving method described with reference to FIGS. FIG. 38 is a diagram showing a case where non-VCL NAL units are individually treated as data units and aggregated. FIG. 39 is a diagram showing a case where non-VCL NAL units are combined into a data unit. FIG. 40 is a flowchart of the operation of the receiving device when packet loss occurs. Figure 41 is a flowchart of the receiving operation when an MPU is divided into multiple movie fragments. FIG. 42 is a diagram illustrating an example of a predicted structure of a picture in each TemporalId when achieving temporal scalability. FIG. 43 is a diagram showing the relationship between decoding time (DTS) and display time (PTS) in each picture in FIG. 42. FIG. 44 is a diagram illustrating an example of a predicted structure of a picture that requires picture delay processing and reorder processing. Figure 45 is a diagram showing an example in which an MPU in MP4 format is divided into multiple movie fragments and stored in an MMTP payload and an MMTP packet. FIG. 46 is a diagram for explaining the calculation method and problems of the PTS and DTS. FIG. 47 is a flowchart of the reception operation when DTS is calculated using the information for DTS calculation. FIG. 48 is a diagram for explaining a method of storing a data unit in a payload in MMT. FIG. 49 is an operation flow of the transmitting device according to the third embodiment. FIG. 50 is an operation flow of the receiving device according to the third embodiment. FIG. 51 is a diagram showing an example of a specific configuration of a transmitting device according to the third embodiment. FIG. 52 is a diagram illustrating an example of a specific configuration of a receiving device according to Embodiment 3.

A transmission method according to one aspect of the present invention includes an assignment step of assigning header information to sample data, which is data in which a video signal or an audio signal is encoded, the header information including MP4 configuration information for reconstructing the sample data as an MP4 format file on the receiving side, the content of which varies depending on whether the presentation time of the sample data is set or not, and a transmission step of transmitting the sample data to which the header information has been assigned, and in the assignment step, if metadata corresponding to the sample data is not transmitted in the transmission step, header information that does not include the MP4 configuration information is assigned to the sample data depending on whether the presentation time of the sample data is set or not.

This type of transmission method can simplify the device configuration and reduce the amount of processing required by the device when transmitting data using a method such as MMT.

Further, in the adding step, if metadata corresponding to the sample data is not transmitted in the sending step, and if the presentation time of the sample data is determined, header information that does not include the MP4 configuration information is added. If the presentation time of the sample data is not determined, header information including the MP4 configuration information is added to the sample data.

Furthermore, in the transmitting step, the sample data to which the header information has been added may be packetized and transmitted using an MMT (MPEG Media Transport) method.

In addition, the header information attached to the sample data for which a presentation time is determined may include at least one of movie_fragment_sequence_number, sample_number, offset, priority, and dependency_counter in the MMTP (MMT Protocol) payload as the MP4 configuration information, and the header information attached to the sample data for which a presentation time is not determined may include item_id in the MP4 MMTP payload as the MP4 configuration information.

Further, the sample data for which the presentation time is determined is a timed-MFU (Movie
The sample data, which is a Fragment Unit) and whose presentation time is not determined, may be a non-timed-MFU.

Further, the metadata may include MPU (Media Processing Unit) metadata and movie fragment metadata.

Further, the reception method according to one aspect of the present invention includes sample data that is data in which a video signal or an audio signal is encoded, and MP4 configuration information for reconstructing the sample data as an MP4 format file. a receiving step of receiving sample data to which header information that does not include is added; and a case where metadata corresponding to the sample data is not received in the receiving step, and a presentation time of the sample data is determined. and decoding the sample data without using the MP4 configuration information.

Such a receiving method can simplify the configuration of the device and reduce the processing amount of the device when transmitting data using a method such as MMT.

A transmitting device according to one aspect of the present invention adds MP4 configuration information to sample data, which is data in which a video signal or an audio signal is encoded, for reconstructing the sample data as an MP4 format file on a receiving side. an attaching unit that attaches header information including MP4 configuration information whose content differs depending on whether a presentation time of the sample data is determined; and a transmitting unit that transmits the sample data to which the header information is attached; If the metadata corresponding to the sample data is not transmitted by the transmitting unit, the adding unit includes the MP4 configuration information depending on whether or not a presentation time of the sample data is determined. Add header information that does not exist to the sample data.

A receiving device according to one aspect of the present invention provides sample data that is data in which a video signal or an audio signal is encoded, and does not include MP4 configuration information for reconstructing the sample data as an MP4 format file. a receiving unit that receives sample data to which header information has been added, and a case where metadata corresponding to the sample data is not received by the receiving unit, and a presentation time of the sample data is determined; , a decoding unit that decodes the sample data without using the MP4 configuration information.

Note that these comprehensive or specific aspects may be realized by a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, and the system, method, integrated circuit, It may be realized by any combination of computer programs or recording media.

Hereinafter, embodiments will be specifically described with reference to the drawings.

The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component placement and connection forms, steps, and order of steps shown in the following embodiments are merely examples and are not intended to limit the present invention. Furthermore, among the components in the following embodiments, components that are not described in an independent claim that indicates a superordinate concept are described as optional components.

(Findings that formed the basis of the present invention)
In recent years, the resolution of displays such as TVs, smartphones, and tablet terminals has been increasing. In particular, 8K4K (8K x 4K resolution) services are planned for broadcasting in Japan in 2020. Since it is difficult to decode ultra-high resolution moving images such as 8K4K in real time with a single decoder, methods of performing decoding processing in parallel using multiple decoders are being considered.

Since encoded data is multiplexed and transmitted based on a multiplexing method such as MPEG-2 TS or MMT, the receiving device needs to separate the video encoded data from the multiplexed data before decoding. . Hereinafter, the process of separating encoded data from multiplexed data will be referred to as demultiplexing.

When parallelizing decoding processing, it is necessary to distribute encoded data to be decoded to each decoder. When distributing encoded data, it is necessary to analyze the encoded data itself, and since the bit rate is particularly high in content such as 8K, the processing load associated with analysis is large. Therefore, there was a problem that the demultiplexing part became a bottleneck and reproduction could not be performed in real time.

By the way, H. 264 and H. In a video encoding system such as H.265, a transmitting device can divide a picture into a plurality of regions called slices or slice segments, and can encode each divided region so that it can be independently decoded. Thus, for example, H. In the case of H.265, a receiving device that receives broadcasting can realize parallel decoding processing by separating the data for each slice segment from the received data and outputting the data for each slice segment to separate decoders. .

FIG. 1 is a diagram showing an example of dividing one picture into four slice segments in HEVC. For example, the receiving device includes four decoders, each decoder decoding one of the four slice segments.

In conventional broadcasting, a transmitting device stores one picture (an access unit in the MPEG system standard) in one PES packet and multiplexes the PES packet into a sequence of TS packets. For this reason, a receiving device had to separate the payload of the PES packet, analyze the data of the access unit stored in the payload, separate each slice segment, and output the data of each separated slice segment to a decoder.

However, the inventors have found that there is a problem in that it is difficult to perform this processing in real time because the amount of processing required to analyze access unit data and separate slice segments is large.

FIG. 2 is a diagram illustrating an example in which data of a picture divided into slice segments is stored in the payload of a PES packet.

As shown in FIG. 2, for example, data from multiple slice segments (slice segments 1 to 4) is stored in the payload of one PES packet. In addition, the PES packet is multiplexed into a sequence of TS packets.

(Embodiment 1)
In the following, H. The following example uses H.265, but H.265 is used as an example. This embodiment can also be applied to cases where other encoding methods such as H.264 are used.

Figure 3 shows an example of dividing an access unit (picture) into division units in this embodiment. The access unit is divided into two equal parts horizontally and vertically, into a total of four tiles, by a function called tiles introduced by H.265. Also, there is a one-to-one correspondence between slice segments and tiles.

The reason for dividing the data into two equal parts horizontally and vertically will be explained below. First, when decoding, a line memory is generally required to store one horizontal line of data, but when it comes to ultra-high resolution such as 8K4K, the horizontal size becomes large, so the size of the line memory increases. When implementing a receiving device, it is desirable to be able to reduce the size of the line memory. In order to reduce the size of the line memory, vertical division is necessary. A data structure called a tile is required for vertical division. For these reasons, tiles are used.

On the other hand, since images generally have a high correlation in the horizontal direction, encoding efficiency is improved if a wider range can be referenced in the horizontal direction. Therefore, from the viewpoint of coding efficiency, it is desirable that the access unit be divided horizontally.

By dividing the access unit into two in the horizontal and vertical directions, these two characteristics can be achieved simultaneously, and both aspects of implementation and encoding efficiency can be considered. If a single decoder is capable of decoding 4K2K video images in real time, the 8K4K image is divided into four equal slices, and each slice segment is divided into 4K2K segments. can decode 8K4K images in real time.

Next, the reason why the tiles obtained by dividing the access unit in the horizontal and vertical directions and the slice segments are associated one-to-one will be explained. H. In H.265, an access unit is composed of a plurality of units called NAL (Network Adaptation Layer) units.

The payload of the NAL unit includes an access unit delimiter that indicates the start position of the access unit, an SPS (Sequence Parameter Set) that is initialization information at the time of decoding that is commonly used in sequence units, and an initialization information at the time of decoding that is commonly used within the picture. PPS (Picture Parameter Set), which is encoding information, SEI (Supplemental Enhancement Information), which is not necessary for the decoding process itself but is necessary for processing and displaying the decoding results, and encoded data of slice segments, etc. Store. The header of a NAL unit includes type information to identify the data stored in the payload.

Here, the transmitting device transmits the encoded data to MPEG-2 TS, MMT (MPEG Media Transport), MPEG DASH (Dynamic Adaptive
When multiplexing using a multiplexing format such as Streaming over HTTP (Streaming over HTTP) or RTP (Real-time Transport Protocol), the basic unit can be set to a NAL unit. In order to store one slice segment in one NAL unit, when dividing the access unit into areas, it is desirable to divide the access unit into slice segments. For this reason, the transmitting device makes a one-to-one correspondence between tiles and slice segments.

Note that, as shown in FIG. 4, the transmitting device can also set tiles 1 to 4 as one slice segment. However, in this case, all tiles are stored in one NAL unit, making it difficult for the receiving device to separate the tiles in the multiplexing layer.

Note that the slice segments include independent slice segments that can be decoded independently and reference slice segments that refer to the independent slice segments. Here, a case where independent slice segments are used will be described.

FIG. 5 is a diagram illustrating an example of data of access units divided so that the boundaries between tiles and slice segments coincide as shown in FIG. 3. The access unit data consists of the NAL unit in which the access unit delimiter placed at the beginning is stored, the SPS, PPS, and SEI NAL units placed after that, and the tiles 1 to 4 placed after that. and the data of the slice segment in which the data is stored. Note that the access unit data does not need to include some or all of the SPS, PPS, and SEI NAL units.

Next, the configuration of transmitting device 100 according to this embodiment will be explained. FIG. 6 is a block diagram showing a configuration example of transmitting device 100 according to this embodiment. This transmitting device 100 includes an encoding section 101, a multiplexing section 102, a modulating section 103, and a transmitting section 104.

The encoding unit 101 converts the input image into, for example, H. Encoded data is generated by encoding according to H.265. Further, the encoding unit 101 divides the access unit into four slice segments (tiles) and encodes each slice segment, for example, as shown in FIG. 3.

Multiplexing section 102 multiplexes the encoded data generated by encoding section 101. Modulating section 103 modulates data obtained by multiplexing. The transmitter 104 transmits the modulated data as a broadcast signal.

Next, the configuration of the receiving device 200 according to this embodiment will be described. FIG. 7 is a block diagram showing an example of the configuration of the receiving device 200 according to this embodiment. This receiving device 200 includes a tuner 201, a demodulation unit 202, a demultiplexing unit 203, a plurality of decoding units 204A to 204D, and a display unit 205.

Tuner 201 receives broadcast signals. Demodulation section 202 demodulates the received broadcast signal. The demodulated data is input to demultiplexer 203.

Demultiplexing section 203 separates the demodulated data into division units, and outputs the data for each division unit to decoding sections 204A to 204D. Here, the division unit is a division area obtained by dividing an access unit, and is, for example, a division area obtained by dividing an access unit. This is a slice segment in H.265. Further, here, an 8K4K image is divided into four 4K2K images. Therefore, there are four decoding units 204A to 204D.

The plurality of decoders 204A to 204D operate in synchronization with each other based on a predetermined reference clock. Each decoding section decodes the encoded data in units of division according to the DTS (Decoding Time Stamp) of the access unit, and outputs the decoding result to the display section 205.

The display unit 205 generates an 8K4K output image by integrating the multiple decoding results output from the multiple decoding units 204A to 204D. The display unit 205 displays the generated output image according to the PTS (Presentation Time Stamp) of the access unit that is obtained separately. Note that when integrating the decoding results, the display unit 205 performs filter processing such as a deblock filter on the boundary areas of mutually adjacent division units, such as the boundaries of tiles, so that the boundaries become visually inconspicuous. Good too.

Note that although the above description has been made using the transmitting device 100 and the receiving device 200 that transmit or receive broadcasts as an example, content may be transmitted and received via a communication network. When the receiving device 200 receives content via a communication network, the receiving device 200 separates multiplexed data from IP packets received via a network such as Ethernet.

In broadcasting, the transmission path delay from when a broadcast signal is transmitted until it reaches receiving device 200 is constant. On the other hand, in a communication network such as the Internet, the transmission path delay until data transmitted from the server reaches the receiving device 200 is not constant due to the influence of congestion. Therefore, the receiving device 200 often does not perform strictly synchronized reproduction based on a reference clock such as PCR in MPEG-2 TS for broadcasting. Therefore, the receiving device 200 may display the 8K4K output image on the display unit according to the PTS without strictly synchronizing each decoding unit.

Furthermore, due to communication network congestion, decoding processing for all division units may not be completed at the time indicated by the PTS of the access unit. In this case, the receiving device 200 skips the display of the access unit or delays the display until the decoding of at least four division units is completed and the generation of the 8K4K image is completed.

Note that content may be transmitted and received using both broadcasting and communication. Further, this method can also be applied when reproducing multiplexed data stored in a recording medium such as a hard disk or a memory.

Next, we will explain how to multiplex access units divided into slice segments when MMT is used as the multiplexing method.

FIG. 8 is a diagram illustrating an example of packetizing HEVC access unit data into MMT. Although SPS, PPS, SEI, etc. do not necessarily need to be included in the access unit, a case where they are present will be exemplified here.

NAL units that are located before the first slice segment in an access unit, such as the access unit delimiter, SPS, PPS, and SEI, are stored together in MMT packet #1. Subsequent slice segments are stored in separate MMT packets for each slice segment.

Note that, as shown in FIG. 9, a NAL unit placed before the first slice segment in the access unit may be stored in the same MMT packet as the first slice segment.

Additionally, if a NAL unit such as End-of-Sequence or End-of-Bitstream indicating the end of a sequence or stream is added after the final slice segment, these units are included in the same MMT packet as the final slice segment. Stored. However, since NAL units such as End-of-Sequence or End-of-Bitstream are inserted at the end point of decoding processing or the connection point of two streams, the receiving device 200 , may be desirable to be readily available at the multiplexing layer. In this case, these NAL units may be stored in a separate MMT packet from the slice segment. Thereby, receiving device 200 can easily separate these NAL units in the multiplexing layer.

Note that TS, DASH, RTP, or the like may be used as the multiplexing method. In these methods, the transmitting device 100 stores different slice segments in different packets. This ensures that the receiving device 200 can separate the slice segments at the multiplexing layer.

For example, when TS is used, encoded data is converted into PES packets in units of slice segments. When RTP is used, encoded data is converted into RTP packets in units of slice segments. Even in these cases, the NAL unit placed before the slice segment and the slice segment may be packetized separately, as in MMT packet #1 shown in FIG. 8.

When a TS is used, the transmitting device 100 indicates the unit of data stored in the PES packet, such as by using a data alignment descriptor. Furthermore, since DASH is a method of downloading MP4 format data units called segments using HTTP or the like, the transmitting device 100 does not packetize encoded data during transmission. Therefore, the transmitting device 100 creates sub-samples for each slice segment and stores information indicating the storage location of the sub-samples in the MP4 header so that the receiving device 200 can detect the slice segments in the multiplex layer in MP4. You may.

MMT packetization of slice segments will be described in detail below.

As shown in FIG. 8, by packetizing the encoded data, data that is commonly referenced when decoding all slice segments in an access unit, such as SPS and PPS, is stored in MMT packet #1. In this case, receiving device 200 concatenates the payload data of MMT packet #1 with the data of each slice segment, and outputs the obtained data to the decoding unit. In this way, receiving device 200 can easily generate input data for the decoding unit by concatenating the payloads of multiple MMT packets.

FIG. 10 is a diagram showing an example in which input data to the decoding units 204A to 204D is generated from the MMT packet shown in FIG. 8. The demultiplexing unit 203 generates data necessary for the decoding unit 204A to decode slice segment 1 by concatenating the payload data of MMT packet #1 and MMT packet #2. Demultiplexing section 203 similarly generates input data for decoding sections 204B to 204D. That is, the demultiplexer 203 generates input data for the decoder 204B by concatenating the payload data of MMT packet #1 and MMT packet #3. The demultiplexing unit 203 generates input data for the decoding unit 204C by concatenating the payload data of MMT packet #1 and MMT packet #4. Demultiplexing section 203 generates input data for decoding section 204D by concatenating payload data of MMT packet #1 and MMT packet #5.

Note that the demultiplexing unit 203 removes NAL units that are not necessary for decoding processing, such as the access unit delimiter and SEI, from the payload data of MMT packet #1, and extracts only the SPS and PPS NAL units that are necessary for decoding processing. may be separated and added to the slice segment data.

When the encoded data is packetized as shown in FIG. 9, the demultiplexing section 203 outputs MMT packet #1 containing the first data of the access unit in the multiplexing layer to the first decoding section 204A. . In addition, the demultiplexer 203 analyzes the MMT packet including the first data of the access unit in the multiplexing layer, separates the SPS and PPS NAL units, and slices the separated SPS and PPS NAL units into the second and subsequent slices. By adding it to each piece of data in the segment, input data for each of the second and subsequent decoders is generated.

Further, the receiving device 200 uses the information included in the header of the MMT packet to determine the type of data stored in the MMT payload and, if a slice segment is stored in the payload, the slice segment in the access unit. It is desirable to be able to identify the index number. Here, the data type refers to pre-slice segment data (NAL units placed before the first slice segment in an access unit are collectively called this) and slice segment data. Either way. When storing a unit in which an MPU is fragmented, such as a slice segment, in an MMT packet, a mode for storing an MFU (Media Fragment Unit) is used. When using this mode, the transmitting device 100 converts the Data Unit, which is the basic unit of data in the MFU, into a sample (which is a data unit in MMT and corresponds to an access unit) or a subsample (which is a data unit in MMT and corresponds to an access unit). (divided units).

At this time, the header of the MMT packet includes a field called a Fragmentation indicator and a field called a Fragment counter.

The Fragmentation indicator indicates whether the data stored in the payload of an MMT packet is a fragment of a Data unit, and if so, whether the fragment is the first or last fragment in the Data unit, or a fragment that is neither the first nor the last. In other words, the Fragmentation indicator included in the header of a packet is identification information that indicates whether (1) only the packet is included in the Data unit, which is the basic data unit, (2) the Data unit is divided and stored into multiple packets, and the packet is the first packet of the Data unit, (3) the Data unit is divided and stored into multiple packets, and the packet is a packet other than the first or last packet of the Data unit, or (4) the Data unit is divided and stored into multiple packets, and the packet is the last packet of the Data unit.

The Fragment counter is an index number indicating which fragment in the Data unit the data stored in the MMT packet corresponds to.

Therefore, by the transmitting device 100 setting samples in MMT in the Data unit, and setting the pre-slice segment data and each slice segment in fragment units of the Data unit, the receiving device 200 can read the header of the MMT packet. The information contained in the payload can be used to identify the type of data stored in the payload. In other words, the demultiplexing section 203 can generate input data to each of the decoding sections 204A to 204D by referring to the header of the MMT packet.

Figure 11 shows an example where a sample is set to a Data unit, and pre-slice segment data and slice segments are packetized as fragments of Data units.

The pre-slice segment data and the slice segment are divided into five fragments from fragment #1 to fragment #5. Each fragment is stored in a separate MMT packet. At this time, the values of the Fragmentation indicator and Fragment counter included in the header of the MMT packet are as shown in the figure.

For example, the Fragment indicator is a 2-bit binary value. The Fragment indicator of MMT packet #1 which is the beginning of the Data unit, the Fragment indicator of MMT packet #5 which is the last one, and the Fragment indicators of MMT packet #2 to MMT packet #4 which are the packets between them are each different. set to the value. Specifically, the Fragment indicator of MMT packet #1, which is the beginning of the Data unit, is set to 01, the Fragment indicator of MMT packet #5, which is the last, is set to 11, and the Fragment indicator of MMT packet #1, which is the beginning of the data unit, is set to 11. Fragment indicators up to MMT packet #4 are set to 10. Note that when the Data unit includes only one MMT packet, the Fragment indicator is set to 00.

Fragment counter is 4 in MMT packet #1, which is the total number of fragments (5) minus 1, and is decreased by 1 in each of the subsequent packets, until it is 0 in the final MMT packet #5.

Therefore, the receiving device 200 can identify the MMT packet storing the pre-slice segment data using either the Fragment indicator or the Fragment counter. Further, the receiving device 200 can identify the MMT packet storing the Nth slice segment by referring to the Fragment counter.

The header of the MMT packet separately includes a sequence number within the MPU of the Movie Fragment to which the Data unit belongs, a sequence number of the MPU itself, and a sequence number within the Movie Fragment of the sample to which the Data unit belongs. By referring to these, the demultiplexer 203 can uniquely determine the sample to which the Data unit belongs.

Further, since the demultiplexer 203 can determine the index number of a fragment in the data unit from the Fragment counter, it can uniquely identify the slice segment stored in the fragment even if a packet loss occurs. For example, even if fragment #4 shown in FIG. 11 cannot be acquired due to packet loss, the demultiplexer 203 knows that the fragment received next to fragment #3 is fragment #5, so It is possible to correctly output slice segment 4 stored in the decoding section 204D to the decoding section 204D instead of the decoding section 204C.

Note that when a transmission path that is guaranteed not to cause packet loss is used, the demultiplexer 203 refers to the header of the MMT packet to determine the type of data stored in the MMT packet or the slice segment. Arrived packets may be processed periodically without determining the index number of the packet. For example, when an access unit is transmitted using a total of five MMT packets including pre-slice data and four slice segments, the receiving device 200 determines the pre-slice data of the access unit to start decoding. , by sequentially processing the received MMT packets, the pre-slice data and the data of the four slice segments can be sequentially acquired.

Modified examples of packetization will be described below.

The slice segment does not necessarily have to be one in which the plane of the access unit is divided both horizontally and vertically, and may be one in which the access unit is divided only in the horizontal direction, as shown in FIG. However, it may be divided only in the vertical direction.

Also, if the access unit is divided only horizontally, tiles do not need to be used.

The number of divisions within an access unit is arbitrary and is not limited to four. However, the area size of slice segments and tiles must be equal to or larger than the lower limit of coding standards such as H.265.

The transmitting device 100 may store identification information indicating an in-plane division method in an access unit in an MMT message, a TS descriptor, or the like. For example, information indicating the number of divisions in the horizontal direction and vertical direction within the plane may be stored. Or identification information unique to the division method, such as divided into two equal parts horizontally and vertically as shown in Figure 3, or divided into four equal parts horizontally as shown in Figure 1. may be assigned. For example, when the access unit is divided as shown in FIG. 3, the identification information indicates mode 1, and when the access unit is divided as shown in FIG. 1, the identification information indicates mode 1. .

Further, information indicating constraints on encoding conditions related to the intra-plane division method may be included in the multiplexing layer. For example, information indicating that one slice segment is composed of one tile may be used. Or, reference blocks when performing motion compensation when decoding slice segments or tiles are limited to slice segments or tiles at the same position in the screen, or to blocks within a predetermined range in adjacent slice segments. Information indicating such things may also be used.

Further, the transmitting device 100 may switch whether to divide the access unit into a plurality of slice segments depending on the resolution of the moving image. For example, if the moving image to be processed has a resolution of 4K2K, the transmitting device 100 may not perform in-plane division, but if the moving image to be processed is 8K4K, it may divide the access unit into four. good. By predefining the division method for 8K4K video images, the receiving device 200 determines whether or not to divide within the screen and the division method by acquiring the resolution of the received video image, and performs decoding. You can switch the operation.

Furthermore, the receiving device 200 can detect whether or not there is intra-plane division by referring to the header of the MMT packet. For example, if the access unit is not divided, if the MMT Data unit is set to a sample, the Data unit is not fragmented. Therefore, the receiving device 200 can determine that the access unit is not divided if the value of the Fragmentation counter included in the header of the MMT packet is always zero. Alternatively, the receiving device 200 may detect whether the value of the Fragmentation indicator is always 01. The receiving device 200 can also determine that the access unit is not divided if the value of the Fragmentation indicator is always 01.

Furthermore, the receiving device 200 can also handle a case where the number of in-plane divisions in an access unit and the number of decoding units do not match. For example, when receiving device 200 includes two decoding sections 204A and 204B that can decode 8K2K encoded data in real time, demultiplexing section 203 sends 8K4K encoded data to decoding section 204A. Outputs two of the four slice segments that make up the slice segment.

FIG. 12 is a diagram illustrating an example of the operation when the MMT packetized data as shown in FIG. 8 is input to two decoding units 204A and 204B. Here, it is desirable that the receiving device 200 be able to integrate and output the decoding results in the decoding sections 204A and 204B as they are. Therefore, demultiplexing section 203 selects slice segments to be output to each of decoding sections 204A and 204B so that the decoding results of each of decoding sections 204A and 204B are spatially continuous.

The demultiplexer 203 may select a decoding unit to use depending on the resolution or frame rate of the encoded data of the video. For example, if the receiving device 200 has four 4K2K decoding units, and the resolution of the input image is 8K4K, the receiving device 200 performs the decoding process using all four decoding units. Also, if the resolution of the input image is 4K2K, the receiving device 200 performs the decoding process using only one decoding unit. Alternatively, even if the plane is divided into four, if 8K4K can be decoded in real time by a single decoding unit, the demultiplexer 203 integrates all the division units and outputs them to one decoding unit.

Furthermore, the receiving device 200 may determine the decoding unit to be used taking into consideration the frame rate. For example, if the receiving device 200 has two decoding units with an upper limit of 60 fps for the frame rate that can be decoded in real time when the resolution is 8K4K, there may be cases where 8K4K encoded data at 120 fps is input. In this case, if the plane is composed of four division units, slice segment 1 and slice segment 2 are input to the decoding unit 204A, and slice segment 3 and slice segment 4 are input to the decoding unit 204B, as in the example of FIG. 12. Since each of the decoding units 204A and 204B can decode up to 120 fps in real time if the resolution is 8K2K (half of 8K4K), the decoding process is performed by these two decoding units 204A and 204B.

Even if the resolution and frame rate are the same, the processing amount differs if the profile or level in the encoding method, or the encoding method itself, such as H.264 or H.265, is different. Therefore, the receiving device 200 may select the decoding unit to be used based on this information. Note that, when the receiving device 200 cannot decode all the encoded data received by broadcasting or communication, or when it cannot decode all the slice segments or tiles that constitute the area selected by the user, it may automatically determine the slice segments or tiles that can be decoded within the processing range of the decoding unit. Alternatively, the receiving device 200 may provide a user interface for the user to select the area to be decoded. At this time, the receiving device 200 may display a warning message indicating that all areas cannot be decoded, or may display information indicating the number of areas, slice segments, or tiles that can be decoded.

Furthermore, the above method can also be applied when MMT packets storing slice segments of the same encoded data are transmitted and received using multiple transmission paths such as broadcasting and communication.

In addition, the transmitting device 100 may perform coding so that the areas of each slice segment overlap in order to make the boundaries between the division units less noticeable. In the example shown in FIG. 13, an 8K4K picture is divided into four slice segments 1 to 4. Each of slice segments 1 to 3 is, for example, 8K x 1.1K, and slice segment 4 is 8K x 1K. Adjacent slice segments also overlap each other. In this way, motion compensation during coding can be efficiently performed at the boundaries when divided into four, as indicated by the dotted lines, improving the image quality of the boundary parts. In this way, image quality degradation at the boundary parts is reduced.

In this case, the display unit 205 cuts out an 8K×1K area from the 8K×1.1K area and integrates the obtained areas. Note that the transmitting device 100 includes information indicating whether the slice segments are encoded in an overlapping manner and the range of overlap in the multiplex layer or encoded data, and separately transmits the information. Good too.

Note that a similar method can be applied even when tiles are used.

The flow of operation of the transmitting device 100 will be described below. Figure 14 is a flowchart showing an example of the operation of the transmitting device 100.

First, the encoding unit 101 divides a picture (access unit) into a plurality of slice segments (tiles) that are a plurality of regions (S101). Next, the encoding unit 101 generates encoded data corresponding to each of the plurality of slice segments by encoding each of the plurality of slice segments so that each of the plurality of slice segments can be decoded independently (S102). Note that the encoding unit 101 may encode a plurality of slice segments using a single encoding unit, or may perform parallel processing using a plurality of encoding units.

Next, the multiplexing unit 102 multiplexes the plurality of encoded data by storing the plurality of encoded data generated by the encoding unit 101 in a plurality of MMT packets (S103). Specifically, as shown in FIGS. 8 and 9, the multiplexing unit 102 combines a plurality of encoded data into a plurality of encoded data so that encoded data corresponding to different slice segments is not stored in one MMT packet. Store in MMT packet. Furthermore, as shown in FIG. 8, the multiplexing unit 102 transfers control information commonly used for all decoding units in a picture to multiple MMT packets #2 to ## in which multiple encoded data are stored. 5 is stored in MMT packet #1 different from MMT packet #5. Here, the control information includes at least one of an access unit delimiter, SPS, PPS, and SEI.

Note that the multiplexing unit 102 may store the control information in the same MMT packet as any of the multiple MMT packets in which multiple pieces of encoded data are stored. For example, as shown in FIG. 9, the multiplexing unit 102 stores the control information in the first MMT packet (MMT packet #1 in FIG. 9) among the multiple MMT packets in which multiple pieces of encoded data are stored. You may.

Finally, transmitting device 100 transmits a plurality of MMT packets. Specifically, modulating section 103 modulates the data obtained by multiplexing, and transmitting section 104 transmits the modulated data (S104).

FIG. 15 is a block diagram showing an example of the configuration of the receiving device 200, and is a diagram showing in detail the configuration of the demultiplexer 203 shown in FIG. 7 and the subsequent stage. As shown in FIG. 15, receiving device 200 further includes a decoding command section 206. Further, the demultiplexing section 203 includes a type determining section 211 , a control information acquiring section 212 , a slice information acquiring section 213 , and a decoded data generating section 214 .

The flow of the operation of the receiving device 200 will be described below. FIG. 16 is a flowchart illustrating an example of the operation of the receiving device 200. Here, the operation for one access unit is shown. When decoding processing for multiple access units is executed, the processing in this flowchart is repeated.

First, the receiving device 200 receives, for example, a plurality of packets (MMT packets) generated by the transmitting device 100 (S201).

Next, the type determination unit 211 acquires the type of encoded data stored in the received packet by analyzing the header of the received packet (S202).

Next, the type determination unit 211 determines whether the data stored in the received packet is pre-slice segment data or slice segment data based on the type of the acquired encoded data (S203). .

If the data stored in the received packet is pre-slice segment data (Yes in S203), the control information acquisition unit 212 obtains the pre-slice segment data of the access unit to be processed from the payload of the received packet, and The pre-segment data is stored in memory (S204).

On the other hand, if the data stored in the received packet is slice segment data (No in S203), the receiving device 200 uses the header information of the received packet to determine whether the data stored in the received packet is multiple segment data. It is determined which region of the regions the encoded data belongs to. Specifically, the slice information acquisition unit 213 acquires the index number Idx of the slice segment stored in the received packet by analyzing the header of the received packet (S205). Specifically, the index number Idx is an index number within the Movie Fragment of the access unit (sample in MMT).

Note that the processing in step S205 may be performed all at once in step S202.

Next, the decoded data generation unit 214 determines a decoding unit that decodes the slice segment (S206). Specifically, the index number Idx and a plurality of decoding units are associated in advance, and the decoded data generation unit 214 causes the decoding unit corresponding to the index number Idx acquired in step S205 to decode the slice segment. Decide which decoder to use.

Note that, as explained in the example of FIG. The decoding unit that decodes the slice segment may be determined based on at least one of the processing capabilities of the decoding unit. For example, the decoded data generation unit 214 determines the access unit division method based on identification information in a descriptor such as an MMT message or a TS section.

Next, the decoded data generation unit 214 generates control information commonly used for all decoding units in a picture, which is included in any of the plurality of packets, and each of the plurality of encoded data of the plurality of slice segments. By combining these, a plurality of input data (combined data) to be input to a plurality of decoders is generated. Specifically, the decoded data generation unit 214 obtains slice segment data from the payload of the received packet. The decoded data generation unit 214 generates the input data to the decoding unit determined in step S206 by combining the pre-slice segment data stored in the memory in step S204 and the acquired slice segment data. (S207).

After step S204 or S207, if the data of the received packet is not the final data of the access unit (No in S208), the processes from step S201 onwards are performed again. That is, the above process is repeated until input data to the plurality of decoding units 204A to 204D corresponding to all slice segments included in the access unit is generated.

Note that the timing at which the packets are received is not limited to the timing shown in FIG. 16, and a plurality of packets may be received in advance or sequentially and stored in a memory or the like.

On the other hand, if the data of the received packet is the final data of the access unit (Yes in S208), the decoding command unit 206 outputs the multiple input data generated in step S207 to the corresponding decoding units 204A to 204D (S209).

Next, the plurality of decoding units 204A to 204D generate a plurality of decoded images by decoding the plurality of input data in parallel according to the DTS of the access unit (S210).

Finally, the display unit 205 generates a display image by combining the plurality of decoded images generated by the plurality of decoding units 204A to 204D, and displays the display image according to the PTS of the access unit (S211).

Note that the receiving device 200 obtains the DTS and PTS of the access unit by analyzing payload data of an MMT packet that stores MPU header information or Movie Fragment header information. Furthermore, when TS is used as the multiplexing method, the receiving device 200 acquires the DTS and PTS of the access unit from the header of the PES packet. If RTP is used as the multiplexing method, the receiving device 200 obtains the DTS and PTS of the access unit from the header of the RTP packet.

Further, when integrating the decoding results of the plurality of decoding units, the display unit 205 may perform filter processing such as a deblocking filter on the boundary between adjacent division units. Note that filtering is not necessary when displaying the decoding results of a single decoding unit, so the display unit 205 performs processing depending on whether or not to perform filtering on the boundaries of the decoding results of multiple decoding units. You may switch. Whether filter processing is necessary may be determined in advance depending on whether or not there is division. Alternatively, information indicating whether filter processing is necessary may be stored separately in the multiplexing layer. Additionally, information necessary for filter processing, such as filter coefficients, may be stored in the SPS, PPS, SEI, or slice segment. The decoding units 204A to 204D or the demultiplexing unit 203 acquire this information by analyzing the SEI, and output the acquired information to the display unit 205. The display unit 205 performs filter processing using this information. Note that when these pieces of information are stored in slice segments, it is desirable that the decoding units 204A to 204D acquire these pieces of information.

Note that in the above description, an example was given in which there are two types of data stored in a fragment: pre-slice segment data and slice segment data, but there may be three or more types of data. In this case, the cases are divided according to the type in step S203.

In addition, when the data size of a slice segment is large, the transmitting device 100 may fragment the slice segment and store it in the MMT packet. In other words, the transmitting device 100 may fragment the pre-slice-segment data and the slice segment. In this case, if the access unit and the data unit are set to be equal as in the packetization example shown in FIG. 11, the following problem occurs.

For example, when slice segment 1 is divided into three fragments, slice segment 1 is divided into three packets with Fragment counter values of 1 to 3 and transmitted. In addition, after the slice segment 2, the Fragment counter value becomes 4 or more, and the Fragment counter value cannot be associated with the data stored in the payload. Therefore, the receiving device 200 cannot identify the packet that stores the first data of the slice segment from the header information of the MMT packet.

In such a case, the receiving device 200 may identify the start position of the slice segment by analyzing the payload data of the MMT packet. Here, H. 264 or H. In H.265, the format for storing NAL units in the multiplex layer is a format called byte stream format in which a start code consisting of a specific bit string is added immediately before the NAL unit header, and a field indicating the size of the NAL unit is added. There are two types: a format called a NAL size format.

The byte stream format is used in MPEG-2 systems and RTP, etc. The NAL size format is used in MP4, as well as DASH and MMT, which use MP4, etc.

When the byte stream format is used, the receiving device 200 analyzes whether the leading data of the packet matches the start code. If the first data of the packet matches the start code, the receiving device 200 determines whether the data included in the packet is slice segment data by acquiring the NAL unit type from the NAL unit header that follows. can detect whether

On the other hand, in the case of the NAL size format, the receiving device 200 cannot detect the start position of the NAL unit based on the bit string. Therefore, in order to obtain the starting position of the NAL unit, the receiving device 200 needs to shift the pointer by reading data by the size of the NAL unit in order from the first NAL unit of the access unit.

However, in the header of the MPU or Movie Fragment in MMT, the size of the subsample unit is indicated, and if the subsample corresponds to pre-slice data or a slice segment, the receiving device 200 uses the subsample size information based on the size information of the subsample. The starting position of each NAL unit can be specified. Therefore, the transmitting device 100 may include information indicating whether subsample unit information exists in the MPU or Movie Fragment in the information that the receiving device 200 acquires when starting data reception, such as MPT in MMT. .

Note that the MPU data is based on and expanded from the MP4 format. In MP4, H. 264 or H. There are two modes: a mode in which parameter sets such as H.265 SPS and PPS can be stored as sample data, and a mode in which they cannot be stored. Further, information for specifying this mode is shown as an entry name of SampleEntry. If the storable mode is used and the parameter set is included in the sample, the receiving device 200 obtains the parameter set using the method described above.

On the other hand, when the non-storable mode is used, the parameter set is stored as Decoder Specific Information in SampleEntry or using a stream for parameter sets. Here, since streams for parameter sets are not generally used, it is desirable for the transmitting device 100 to store the parameter sets in Decoder Specific Information. In this case, the receiving device 200 analyzes the SampleEntry transmitted as MPU metadata or Movie Fragment metadata in the MMT packet, and acquires the parameter set referenced by the access unit.

When the parameter set is stored as sample data, the receiving device 200 can obtain the parameter set necessary for decoding by referring only to the sample data without referring to SampleEntry. At this time, the transmitting device 100 does not need to store the parameter set in SampleEntry. By doing so, the transmitting device 100 can use the same SampleEntry in different MPUs, so that the processing load on the transmitting device 100 during MPU generation can be reduced. Furthermore, there is an advantage that the receiving device 200 does not need to refer to the parameter set in SampleEntry.

Alternatively, the transmitting device 100 may store one default parameter set in SampleEntry and store the parameter set referred to by the access unit in sample data. In conventional MP4, it is common to store parameter sets in SampleEntry, so if a parameter set does not exist in SampleEntry, there is a possibility that there is a receiving device that stops playback. This problem can be solved by using the above method.

Alternatively, the transmitting device 100 may store a parameter set in the sample data only when a parameter set different from the default parameter set is used.

Note that in both modes, it is possible to store the parameter set in SampleEntry, so the transmitting device 100 may always store the parameter set in VisualSampleEntry, and the receiving device 200 may always obtain the parameter set from VisualSampleEntry.

Note that in the MMT standard, MP4 header information such as Moov and Moof is transmitted as MPU metadata or movie fragment metadata, but the transmitting device 100 does not necessarily transmit MPU metadata and movie fragment metadata. You don't have to. Furthermore, the receiving device 200 determines whether SPS and PPS are stored in the sample data based on the ARIB (Association of Radio Industries and Businesses) standard service, asset type, or whether or not MPU meta is transmitted. It is also possible to judge.

Figure 17 shows an example in which pre-slice segment data and each slice segment are set to different Data units.

In the example shown in FIG. 17, the data sizes of the data before the slice segment and slice segments 1 to 4 are Length #1 to Length #5, respectively. The field values of the Fragmentation indicator, Fragment counter, and Offset included in the header of the MMT packet are as shown in the figure.

Here, Offset indicates the bit length (offset) from the beginning of the encoded data of the sample (access unit or picture) to which the payload data belongs to the first byte of the payload data (encoded data) included in the MMT packet. This is offset information. Note that although the value of the Fragment counter will be explained as starting from the value obtained by subtracting 1 from the total number of fragments, it may start from another value.

Figure 18 is a diagram showing an example of when a Data unit is fragmented. In the example shown in Figure 18, slice segment 1 is divided into three fragments, which are stored in MMT packets #2 to #4, respectively. In this case, if the data size of each fragment is Length #2_1 to Length #2_3, respectively, the values of each field are as shown in the figure.

In this way, when a data unit such as a slice segment is set to Data unit, the beginning of the access unit and the beginning of the slice segment can be determined as follows based on the field value of the MMT packet header.

The head of the payload in a packet whose Offset value is 0 is the head of the access unit.

The beginning of the payload of a packet whose Offset value is different from 0 and whose Fragmentation index value is 00 or 01 is the beginning of a slice segment.

In addition, if fragmentation of the data unit does not occur and no packet loss occurs, the receiving device 200 can identify the index number of the slice segment to be stored in the MMT packet based on the number of slice segments obtained after detecting the beginning of the access unit.

Further, even when the Data unit of the pre-slice segment data is fragmented, the receiving device 200 can similarly detect the access unit and the beginning of the slice segment.

Furthermore, even if a packet loss occurs or if the SPS, PPS, and SEI included in the pre-slice segment data are set to separate data units, the receiving device 200 performs slice processing based on the analysis result of the MMT header. The starting position of a slice segment or tile within a picture (access unit) can be identified by identifying the MMT packet that stores the start data of the segment and then analyzing the header of the slice segment. The amount of processing involved in analyzing the slice header is small, and the processing load is not a problem.

In this way, each of the plurality of encoded data of the plurality of slice segments is in one-to-one correspondence with a basic data unit (Data unit) that is a unit of data stored in one or more packets. Furthermore, each of the plurality of encoded data is stored in one or more MMT packets.

The header information of each MMT packet includes Fragmentation indicator (identification information) and Offset (offset information).

The receiving device 200 determines that the beginning of the payload data included in the packet having the header information including the Fragmentation indicator whose value is 00 or 01 is the beginning of the encoded data of each slice segment. . Specifically, the beginning of the payload data included in a packet having header information including an Offset whose value is not 0 and a Fragmentation indicator whose value is 00 or 01 is the beginning of the encoded data of each slice segment. It is determined that

In the example of FIG. 17, the beginning of a Data unit is either the beginning of an access unit or the beginning of a slice segment, and the value of the Fragmentation indicator is 00 or 01. Furthermore, the receiving device 200 can detect the beginning of an access unit or the beginning of a slice segment without referring to the Offset by referring to the type of the NAL unit and determining whether the beginning of a Data Unit is an access unit delimiter or a slice segment.

In this way, the transmitting device 100 performs packetization so that the beginning of the NAL unit always starts from the beginning of the payload of the MMT packet, even when the pre-slice segment data is divided into multiple Data units. The receiving device 200 can detect the beginning of an access unit or a slice segment by analyzing the Fragmentation indicator and the NAL unit header. The NAL unit type is present in the first byte of the NAL unit header. Therefore, when analyzing the header portion of the MMT packet, the receiving device 200 can acquire the type of the NAL unit by additionally analyzing 1 byte of data. In the case of audio, the receiving device 200 only needs to be able to detect the beginning of the access unit, and may make a determination based on whether the value of the Fragmentation indicator is 00 or 01.

As described above, when storing encoded data that has been encoded so that it can be divided and decoded in a PES packet of MPEG-2 TS, the transmitting device 100 can use the data alignment descriptor. Below, an example of a method for storing encoded data in a PES packet is described in detail.

For example, in HEVC, the transmitting device 100 can indicate whether the data stored in the PES packet is an access unit, a slice segment, or a tile by using a data alignment descriptor. The alignment types in HEVC are defined as follows.

Alignment type=8 indicates a HEVC slice segment. Alignment type=9 indicates a HEVC slice segment or access unit. Alignment type=12 indicates a HEVC slice segment or tile.

Therefore, for example, by using type 9, the transmitting device 100 can indicate that the data of the PES packet is either a slice segment or pre-slice segment data. Since a type indicating a slice instead of a slice segment is also separately defined, the transmitting device 100 may use a type indicating a slice instead of a slice segment.

The DTS and PTS included in the header of a PES packet are set only in the PES packet that contains the first data of an access unit. Therefore, if the type is 9 and the PES packet contains a DTS or PTS field, the receiving device 200 can determine that the PES packet stores the entire access unit or the first division unit of the access unit.

Further, the transmitting device 100 may enable the receiving device 200 to distinguish the data included in the packet by using a field such as transport_priority that indicates the priority of the TS packet that stores the PES packet containing the first data of the access unit. good. Further, the receiving device 200 may determine the data included in the packet by analyzing whether the payload of the PES packet is an access unit delimiter. Additionally, data_alignment_indicator in the PES packet header indicates whether data is stored in the PES packet according to these types. If this flag (data_alignment_indicator) is set to 1, it is guaranteed that the data stored in the PES packet follows the type indicated in the data alignment descriptor.

Further, the transmitting device 100 may use the data alignment descriptor only when converting into PES packets in division-decodable units such as slice segments. Thereby, if the data alignment descriptor exists, the receiving device 200 can determine that the encoded data has been made into PES packets in units that can be divided and decodable, and if the data alignment descriptor does not exist, the receiving device 200 can determine that the encoded data is divided into PES packets. It can be determined that the encoded data is converted into PES packets in units of access units. Note that when the data_alignment_indicator is set to 1 and the data alignment descriptor does not exist, it is specified in the MPEG-2 TS standard that the unit of PES packetization is an access unit.

If the PMT contains a data alignment descriptor, the receiving device 200 can determine that the PES packetization is performed in units that can be divided and decoded, and can generate input data for each decoding unit based on the packetized units. If the PMT does not contain a data alignment descriptor and the receiving device 200 determines that parallel decoding of the encoded data is necessary based on the program information or information in other descriptors, the receiving device 200 generates input data for each decoding unit by analyzing the slice header of the slice segment. If the encoded data can be decoded by a single decoding unit, the receiving device 200 decodes the data of the entire access unit by the decoding unit. If information indicating whether the encoded data is composed of units that can be divided and decoded, such as slice segments or tiles, is separately indicated by a descriptor in the PMT, the receiving device 200 may determine whether the encoded data can be decoded in parallel based on the analysis result of the descriptor.

Furthermore, since the DTS and PTS included in the header of the PES packet are set only in the PES packet that includes the first data of the access unit, when the access unit is divided into PES packets, the second and subsequent PES The packet does not include information indicating the DTS and PTS of the access unit. Therefore, when performing decoding processing in parallel, each of the decoding units 204A to 204D and the display unit 205 use the DTS and PTS stored in the header of the PES packet containing the first data of the access unit.

(Embodiment 2)
In the second embodiment, a method of storing NAL size format data in an MP4 format-based MPU in MMT will be described. Note that, although a storage method in the MPU used for MMT will be described below as an example, such a storage method can also be applied to DASH, which is based on the same MP4 format.

[How to store in MPU]
In the MP4 format, multiple access units are stored together in one MP4 file. In the MPU used for MMT, data for each medium is stored in one MP4 file, and the data can include an arbitrary number of access units. Since the MPU is a unit that can be decoded by itself, for example, the MPU stores access units in units of GOP.

FIG. 19 is a diagram showing the configuration of the MPU. The beginning of the MPU is ftyp, mmpu, and moov, which are collectively defined as MPU metadata. moov stores initialization information common to files and an MMT hint track.

In addition, moof stores initialization information and size for each sample and subsample, information that can identify the presentation time (PTS) and decoding time (DTS) (sample_duration, sample_size, sample_composition_time_offset), as well as data_offset, which indicates the position of the data.

Further, each of the plurality of access units is stored as a sample in mdat (mdat box). Data excluding samples among moof and mdat is defined as movie fragment metadata (hereinafter referred to as MF metadata), and sample data of mdat is defined as media data.

Figure 20 is a diagram showing the structure of MF metadata. As shown in Figure 20, in more detail, MF metadata consists of type, length, and data of moof box (moof), and type and length of mdat box (mdat).

When storing access units in MP4 data, H. 264 and H. There are two modes: a mode in which parameter sets such as H.265 SPS and PPS can be stored as sample data, and a mode in which they cannot be stored.

Here, in the above storage-unable mode, the parameter set is stored in Decoder Specific Information of SampleEntry in moov. Also, in the storable mode, the parameter set is included within the sample.

MPU metadata, MF metadata, and media data are each stored in an MMT payload, and a fragment type (FT) is stored in the header of the MMT payload as an identifier that can identify these data. FT=0 indicates MPU metadata, FT=1 indicates MF metadata, and FT=2 indicates media data.

Note that although FIG. 19 shows an example in which MPU metadata units and MF metadata units are stored as data units in the MMT payload, units such as ftyp, mmpu, moov, and moof are stored as data units. It may be stored in the MMT payload in unit units. Similarly, FIG. 19 illustrates an example in which sample units are stored as data units in the MMT payload. However, a data unit may be configured in units of samples or in units of NAL units, and such data units may be stored in the MMT payload in units of data units. Such data units may be further stored in fragmented units in the MMT payload.

[Conventional transmission methods and issues]
Conventionally, when encapsulating multiple access units into the MP4 format, moovs and moofs were created when all the samples to be stored in the MP4 were completed.

When transmitting MP4 format in real time using broadcasting etc., for example, if the samples stored in one MP4 file are in units of GOP, moov and moof are created after time samples in units of GOP are accumulated. There is a delay associated with encapsulation. Such encapsulation at the transmitter always increases the end-to-end delay by a GOP unit of time. This makes it difficult to provide services in real time, leading to deterioration of services for viewers, especially when live content is transmitted.

FIG. 21 is a diagram for explaining the data transmission order. When applying MMT to broadcasting, as shown in FIG. ), there will be a delay in the transmission of the MMT packet due to encapsulation.

In order to prevent the delay caused by this encapsulation, as shown in FIG. 3-#6 in this order) has been proposed. Also, as shown in FIG. 20(c), the media data is transmitted first without waiting for the creation of the MPU header information, and the MPU header information is transmitted after the media data is transmitted (#3-#6, # A possible method is to transmit the data in the order of #1 and #2.

If the MPU header information is not transmitted, the receiving device decodes it without using the MPU header information, and if the MPU header information is sent later than the media data, the receiving device decodes the MPU header information. Wait for information to be obtained before decrypting.

However, with conventional MP4-compliant receiving devices, decoding without using MPU header information is not guaranteed. Furthermore, if a conventional transmission method is used when the receiving device performs decoding without using an MPU header through special processing, the decoding process becomes complicated and there is a high possibility that real-time decoding becomes difficult. Additionally, if the receiving device performs decoding after waiting to obtain MPU header information, it is necessary to buffer the media data until the receiving device obtains the header information, but there is no defined buffer model. decryption was not guaranteed.

Therefore, the transmitting device according to the second embodiment transmits the MPU metadata before the media data by storing only common information in the MPU metadata, as shown in FIG. 20(d). Then, the transmitting device according to the second embodiment transmits the MF metadata whose generation is delayed after the media data. This provides a transmission method or a reception method that can guarantee decoding of media data.

Below, we will explain the reception method when using each of the transmission methods (a)-(d) in Figure 21.

In each transmission method shown in FIG. 21, first, MPU data is configured in the order of MPU metadata, MFU metadata, and media data.

After configuring the MPU data, when the transmitting device transmits the data in the order of MPU metadata, MF metadata, and media data as shown in (a) of FIG. ) and (A-2).

(A-1) After acquiring the MPU header information (MPU metadata and MF metadata), the receiving device decodes the media data using the MPU header information.

(A-2) The receiving device decodes media data without using MPU header information.

In all of these methods, a delay occurs due to encapsulation on the transmitting side, but there is an advantage that the receiving device does not need to buffer the media data in order to obtain the MPU header. If buffering is not performed, there is no need to install memory for buffering, and no buffering delay occurs. Furthermore, since the method (A-1) performs decoding using MPU header information, it is also applicable to conventional receiving devices.

When the transmitting device transmits only media data as shown in FIG. 21(b), the receiving device can perform decoding using the method (B-1) below.

(B-1) The receiving device decodes media data without using MPU header information.

Furthermore, although not shown, if MPU metadata is transmitted before the media data is transmitted in FIG. 21(b), decoding can be performed using the method (B-2) below.

(B-2) The receiving device decodes the media data using the MPU metadata.

The advantages of both methods (B-1) and (B-2) above are that there is no delay due to encapsulation on the sending side, and there is no need to buffer the media data to obtain the MPU header. It is. However, since both methods (B-1) and (B-2) do not perform decoding using MPU header information, special processing may be required for decoding.

When the transmitting device transmits data in the order of media data, MPU metadata, and MF metadata as shown in (c) of FIG. 21, the receiving device performs the following (C-1) and (C-2). Decoding can be done using either method.

(C-1) After acquiring the MPU header information (MPU metadata and MF metadata), the receiving device decodes the media data.

(C-2) The receiving device decodes the media data without using the MPU header information.

When the method (C-1) above is used, it is necessary to buffer the media data in order to obtain the MPU header information. On the other hand, when the above method (C-2) is used, there is no need to perform buffering to obtain MPU header information.

Furthermore, in both methods (C-1) and (C-2), no delay due to encapsulation occurs on the transmitting side. Furthermore, since the method (C-2) does not use MPU header information, special processing may be required.

When the transmitting device transmits data in the order of MPU metadata, media data, and MF metadata as shown in (d) of FIG. 21, the receiving device transmits data in the following order (D-1) and (D-2). ) can be used to decrypt the file.

(D-1) After acquiring the MPU metadata, the receiving device further acquires the MF metadata and then decodes the media data.

(D-2) After acquiring the MPU metadata, the receiving device decodes the media data without using the MF metadata.

When the method (D-1) above is used, it is necessary to buffer the media data in order to obtain MF metadata, but in the case of method (D-2) above, it is necessary to buffer the media data in order to obtain MF metadata. There is no need to buffer.

Since the above method (D-2) does not perform decoding using MF metadata, special processing may be required.

As explained above, if decoding is possible using MPU metadata and MF metadata, there is an advantage that decoding can also be performed on conventional MP4 receiving devices.

In FIG. 21, the MPU data is structured in the order of MPU metadata, MFU metadata, and media data, and in moof, the position information (offset) for each sample and subsample is determined based on this structure. In addition, the MF metadata also includes data other than the media data in the mdat box (box size and type).

Therefore, when the receiving device identifies media data based on MF metadata, the receiving device reconstructs the data in the order in which the MPU data was configured, regardless of the order in which the data was transmitted. , decoding is performed using moov of MPU metadata or moof of MF metadata.

In addition, in FIG. 21, the MPU data is configured in the order of MPU metadata, MFU metadata, and media data, but even if the MPU data is configured in a different order from that in FIG. 21 and the position information (offset) is determined, good.

For example, the MPU data may be configured in the order of MPU metadata, media data, and MF metadata, and negative position information (offset) may be indicated in the MF metadata. In this case as well, regardless of the order in which the data is transmitted, the receiving device reconfigures the data in the order in which the MPU data was configured on the transmitting side, and then performs decoding using MOOV or MOof.

Note that the transmitting device may signal information indicating the order in which MPU data is configured, and the receiving device may reconfigure the data based on the signaled information.

As explained above, the receiving device receives packetized MPU metadata, packetized media data (sample data), and packetized MF metadata as shown in FIG. 21(d). Receive in order. Here, the MPU metadata is an example of first metadata, and the MF metadata is an example of second metadata.

Next, the receiving device reconstructs MPU data (MP4 format file) including the received MPU metadata, the received MF metadata, and the received sample data. Then, the receiving device decodes the sample data included in the reconstructed MPU data using the MPU metadata and MF metadata. The MF metadata is metadata including data that can be generated only after the sample data is generated on the transmitting side (e.g., length stored in mbox).

In more detail, the operation of the receiving device is performed by each of the components constituting the receiving device. For example, the receiving device includes a receiving unit that receives the data, a reconstruction unit that reconstructs the MPU data, and a decoding unit that decodes the MPU data. Each of the receiving unit, generating unit, and decoding unit is realized by a microcomputer, a processor, a dedicated circuit, etc.

[How to decrypt without using header information]
Next, a method of decoding without using header information will be described. Here, a method for decoding without using header information at the receiving device will be described, regardless of whether header information is sent on the transmitting side or not. That is, this method is applicable to any of the transmission methods described using FIG. 21. However, some decoding methods are applicable only to specific transmission methods.

FIG. 22 is a diagram illustrating an example of a method for decoding without using header information. In FIG. 22, only the MMT payload and MMT packet containing only media data are illustrated, and the MMT payload and MMT packet containing MPU metadata and MF metadata are not illustrated. Furthermore, in the following description of FIG. 22, it is assumed that media data belonging to the same MPU is transmitted continuously. Furthermore, an example will be explained in which a sample is stored in the payload as media data, but in the following explanation of FIG. You can leave it there.

In order to decode media data, a receiving device must first obtain initialization information necessary for decoding. Furthermore, if the media is a video, the receiving device must obtain initialization information for each sample, specify the starting position of the MPU, which is a random access unit, and obtain the starting positions of samples and NAL units. . Further, the receiving device needs to specify the decoding time (DTS) and presentation time (PTS) of each sample.

Therefore, the receiving device can perform decoding without using header information, for example, by using the method below. Note that if the payload stores NAL unit units or fragmented NAL units, "sample" in the following explanation can be read as "NAL unit in sample."

<Random access (=identifying the first sample of MPU)>
When header information is not transmitted, there are methods 1 and 2 below for the receiving device to identify the leading sample of the MPU. Note that when header information is transmitted, method 3 can be used.

[Method 1] The receiving device obtains a sample included in the MMT packet in which 'RAP_flag=1' in the MMT packet header.

[Method 2] The receiving device obtains a sample with 'sample number=0' in the MMT payload header.

[Method 3] If at least one of MPU metadata and MF metadata is transmitted before and after media data, the receiving device detects the fragment type in the MMT payload header. (FT) acquires a sample included in the MMT payload that has been switched to media data.

In Methods 1 and 2, if a single payload contains multiple samples belonging to different MPUs, it is impossible to determine which NAL unit is a random access point (RAP_flag = 1 or sample number = 0). For this reason, it is necessary to impose a constraint such as not mixing samples from different MPUs in a single payload, or, if a single payload contains samples from different MPUs, to impose a constraint such as setting RAP_flag to 1 if the last (or first) sample is a random access point.

Furthermore, in order for the receiving device to obtain the starting position of the NAL unit, it is necessary to shift the data read pointer by the size of the NAL unit sequentially from the first NAL unit of the sample.

If the data is fragmented, the receiving device can identify the data unit by referring to fragment_indicator and fragment_number.

Determining the DTS of a sample
There are two methods for determining the DTS of a sample: Method 1 and Method 2.

[Method 1] The receiving device determines the DTS of the first sample based on the prediction structure. However, this method requires analysis of encoded data and may be difficult to decode in real time, so the following method 2 is preferable.

[Method 2] The receiving device separately transmits the DTS of the first sample and acquires the transmitted DTS of the first sample. Examples of methods for transmitting the DTS of the first sample include a method of transmitting the DTS of the MPU first sample using MMT-SI, and a method of transmitting the DTS of each sample using the MMT packet header extension area. Note that DTS may be an absolute value or may be a relative value with respect to PTS. Alternatively, the transmitting side may signal whether or not the DTS of the first sample is included.

In both methods 1 and 2, the DTS of subsequent samples is calculated assuming a fixed frame rate.

As a method for storing the DTS of each sample in the packet header, in addition to using an extension area, there is a method of storing the DTS of the sample included in the MMT packet in the 32-bit NTP timestamp field in the MMT packet header. If the DTS cannot be expressed using the number of bits (32 bits) of one packet header, the DTS may be expressed using a plurality of packet headers. Further, the DTS may be expressed by combining the NTP timestamp field of the packet header and an extension field. If DTS information is not included, a known value (for example, ALL0) is assumed.

<Determination of sample PTS>
The receiving device obtains the PTS of the first sample from the MPU timestamp descriptor for each asset included in the MPU. The receiving device calculates the subsequent sample PTS based on a parameter indicating the display order of samples such as POC, assuming that the frame rate is fixed. In this way, in order to calculate DTS and PTS without using header information, transmission at a fixed frame rate is essential.

In addition, when MF metadata is transmitted, the receiving device receives the relative time information of the DTS and PTS from the first sample indicated in the MF metadata, and the timestamp of the MPU first sample indicated in the MPU timestamp descriptor. The absolute values of DTS and PTS can be calculated from the absolute values.

When analyzing the encoded data to calculate the DTS and PTS, the receiving device may use SEI information included in the access unit.

<Initialization information (parameter set)>
[For video]
For video, the parameter set is stored in the sample data. Furthermore, if MPU metadata and MF metadata are not transmitted, it is guaranteed that the parameter set necessary for decoding can be obtained by referring only to sample data.

Furthermore, as shown in FIGS. 21(a) and 21(d), when MPU metadata is transmitted before media data, it may be specified that no parameter set is stored in SampleEntry. In this case, the receiving device does not refer to the parameter set of SampleEntry, but only refers to the parameter set within the sample.

Furthermore, when MPU metadata is transmitted before media data, SampleEntry stores a parameter set common to the MPU and a default parameter set, and the receiving device stores the parameter set of SampleEntry and the parameter set in the sample. You may refer to it. By storing the parameter set in the SampleEntry, it becomes possible to perform decoding even with a conventional receiving device that cannot reproduce data unless the parameter set exists in the SampleEntry.

[For audio]
For audio, a LATM header is required for decoding, and for MP4, it is mandatory that the LATM header be included in the sample entry. However, if the header information is not transmitted, it is difficult for the receiving device to acquire the LATM header, so the LATM header is separately included in control information such as SI. Note that the LATM header may be included in a message, table, or descriptor. Note that the LATM header may be included within the sample.

The receiving device obtains the LATM header from the SI or the like before starting decoding, and starts decoding the audio. Alternatively, as shown in FIGS. 21(a) and 21(d), if the MPU metadata is transmitted before the media data, the receiving device receives the LATM header before the media data. It is possible. Therefore, if MPU metadata is transmitted before media data, it is possible to decode it even using a conventional receiving device.

<Others>
The transmission order and the type of transmission order may be notified as control information such as an MMT packet header, payload header, MPT or other table, message, or descriptor. Note that the types of transmission order here are, for example, the four types of transmission orders shown in (a) to (d) in FIG. 21, and an identifier for identifying each type can be obtained before starting decoding. It should be stored somewhere.

Furthermore, different types of transmission order may be used for audio and video, or a common type may be used for audio and video. Specifically, for example, audio is transmitted in the order of MPU metadata, MF metadata, and media data as shown in FIG. 21(a), and video is transmitted in the order of MPU metadata, MF metadata, and media data as shown in FIG. 21(d). , MPU metadata, media data, and MF metadata may be transmitted in this order.

With the method described above, the receiving device can perform decoding without using header information. Furthermore, if the MPU metadata is transmitted before the media data (FIG. 21(a) and FIG. 21(d)), it becomes possible to decode it even with a conventional receiving device.

In particular, by transmitting the MF metadata after the media data ((d) in FIG. 21), a delay due to encapsulation does not occur, and even a conventional receiving device can perform decoding.

[Configuration and operation of transmitter]
Next, the configuration and operation of the transmitting device will be explained. FIG. 23 is a block diagram of a transmitting device according to the second embodiment, and FIG. 24 is a flowchart of a transmitting method according to the second embodiment.

As shown in FIG. 23, the transmitting device 15 includes an encoding unit 16, a multiplexing unit 17, and a transmitting unit 18.

The encoding unit 16 generates encoded data by encoding the video or audio to be encoded, for example, according to H.265 (S10).

The multiplexing unit 17 multiplexes (packets) the encoded data generated by the encoding unit 16 (S11). Specifically, the multiplexing unit 17 packetizes each of sample data, MPU metadata, and MF metadata that constitute an MP4 format file. The sample data is data in which a video signal or an audio signal is encoded, MPU metadata is an example of first metadata, and MF metadata is an example of second metadata. The first metadata and the second metadata are both metadata used to decrypt sample data, but the difference between them is that the second metadata is data that can only be generated after the sample data is generated. It is to include.

Here, data that can be generated only after generation of sample data is, for example, data other than sample data stored in mdat in MP4 format (data in the header of mdat, that is, type and length shown in FIG. 20). ). Here, the second metadata only needs to include length, which is at least a part of this data.

The transmitter 18 transmits the packetized MP4 format file (S12). The transmitter 18 transmits the MP4 format file, for example, by the method shown in FIG. 21(d). That is, packetized MPU metadata, packetized sample data, and packetized MF metadata are transmitted in this order.

Note that each of the encoding section 16, the multiplexing section 17, and the transmitting section 18 is realized by a microcomputer, a processor, a dedicated circuit, or the like.

[Receiving device configuration]
Next, the configuration and operation of the receiving device will be explained. FIG. 25 is a block diagram of a receiving device according to Embodiment 2.

As shown in FIG. 25, the receiving device 20 includes a packet filtering unit 21, a transmission order type discrimination unit 22, a random access unit 23, a control information acquisition unit 24, a data acquisition unit 25, a PTS and DTS calculation unit 26, an initialization information acquisition unit 27, a decoding command unit 28, a decoding unit 29, and a presentation unit 30.

[Receiving device operation 1]
First, when the media is a video, the operation of the receiving device 20 to specify the MPU start position and the NAL unit position will be described. FIG. 26 is a flowchart of such an operation of the receiving device 20. Note that here, it is assumed that the transmission order type of MPU data is stored in the SI information by the transmitting device 15 (multiplexing unit 17).

First, the packet filtering unit 21 performs packet filtering on the received file. The transmission order type determination unit 22 analyzes the SI information obtained by packet filtering and acquires the transmission order type of MPU data (S21).

Next, the transmission order type determining unit 22 determines (determines) whether the data after packet filtering includes MPU header information (at least one of MPU metadata or MF metadata) (S22). If the MPU header information (is included (Yes in S22), the random access unit 23 identifies the MPU leading sample by detecting that the fragment type of the MMT payload header is switched to media data (S23). ).

On the other hand, if the MPU header information is not included (No in S22), the random access unit 23 identifies the MPU leading sample based on the RAP_flag of the MMT packet header or the sample number of the MMT payload header (S24).

Furthermore, the transmission order type determining unit 22 determines whether or not the packet-filtered data includes MF metadata (S25). If it is determined that MF metadata is included (Yes in S25), the data acquisition unit 25 reads the NAL unit based on the sample, subsample offset, and size information included in the MF metadata. A NAL unit is thereby obtained (S26). On the other hand, if it is determined that MF metadata is not included (No in S25), the data acquisition unit 25 reads data of the size of the NAL unit in order from the first NAL unit of the sample to Acquire (S27).

Note that even if it is determined in step S22 that MPU header information is included, the receiving device 20 may identify the MPU first sample using the process in step S24 instead of step S23. Moreover, when it is determined that MPU header information is included, the process of step S23 and the process of step S24 may be used together.

Furthermore, even if it is determined in step S25 that MF metadata is included, the receiving device 20 may acquire the NAL unit using the process in step S27 without using the process in step S26. Further, when it is determined that MF metadata is included, the processing of step S23 and the processing of step S24 may be used together.

Furthermore, a case is assumed in which it is determined in step S25 that MF metadata is included, and the MF data is transmitted after the media data. In this case, the receiving device 20 may perform the process of step S26 after buffering the media data and waiting until the MF metadata is obtained, or the receiving device 20 may buffer the media data and wait until the MF metadata is obtained before performing the process of step S26. It may be determined whether or not to perform the process of step S27.

For example, the receiving device 20 may determine whether to wait for acquisition of MF metadata based on whether it has a buffer with a buffer size that can buffer media data. Further, the receiving device 20 may determine whether to wait for acquisition of MF metadata based on whether the end-to-end delay becomes small. Further, the receiving device 20 may perform the decoding process mainly using the process of step S26, and may use the process of step S27 in a processing mode when a packet loss or the like occurs.

Note that if the transmission order type is predetermined, steps S22 and S26 may be omitted, and in this case, the receiving device 20 selects the MPU in consideration of the buffer size and End-to-End delay. The method for specifying the leading sample and the method for specifying the NAL unit may be determined.

Note that if the transmission order type is known in advance, the transmission order type determination unit 22 in the receiving device 20 is not necessary.

Although not illustrated in FIG. 26 above, the decode command unit 28 outputs the data acquired by the data acquisition unit to the decode unit 29 based on the PTS and DTS calculated by the PTS, DTS calculation unit 26 and the initialization information acquired by the initialization information acquisition unit 27. The decode unit 29 decodes the data, and the presentation unit 30 presents the decoded data.

[Receiving device operation 2]
Next, an operation in which the receiving device 20 acquires initialization information based on the transmission order type and decodes media data based on the initialization information will be described. FIG. 27 is a flowchart of such an operation.

First, the packet filtering unit 21 performs packet filtering on the received file. The transmission order type determination unit 22 analyzes the SI information obtained by packet filtering and acquires the transmission order type (S301).

Next, the transmission order type determination unit 22 determines whether MPU metadata is being transmitted (S302). If it is determined that MPU metadata has been transmitted (Yes in S302), the transmission order type determination unit 22 determines whether MPU metadata is transmitted before media data as a result of the analysis in step S301. (S303). If the MPU metadata is transmitted before the media data (Yes in S303), the initialization information acquisition unit 27 transmits the initialization information based on the common initialization information included in the MPU metadata and the initialization information of the sample data. and decrypts the media data (S304).

On the other hand, if it is determined that the MPU metadata was transmitted after the media data (No in S303), the data acquisition unit 25 buffers the media data until the MPU metadata is acquired (S305), and performs the processing of step S304 after the MPU metadata is acquired.

Further, if it is determined in step S302 that the MPU metadata has not been transmitted (No in S302), the initialization information acquisition unit 27 decodes the media data based only on the initialization information of the sample data. (S306).

Note that if the decryption of the media data is guaranteed only based on the initialization information of the sample data on the sending side, the processing based on the determinations of steps S302 and S303 is not performed, and the processing of step S306 is used.

Further, the receiving device 20 may determine whether or not to buffer the media data before step S305. In this case, if the receiving device 20 determines to buffer the media data, the process proceeds to step S305, and if it determines not to buffer the media data, the process proceeds to step S306. The determination as to whether or not to buffer media data may be made based on the buffer size and occupancy of the receiving device 20, or for example, the one with the smaller End-to-End delay may be selected. The determination may be made taking into account the -to-End delay.

[Receiving device operation 3]
Here, details of the transmission method and reception method in the case where the MF metadata is transmitted after the media data (FIGS. 21(c) and 21(d)) will be described. Below, the case of FIG. 21(d) will be explained as an example. Note that in transmission, only the method shown in FIG. 21(d) is used, and transmission order type signaling is not performed.

As mentioned above, when transmitting data in the order of MPU metadata, media data, and MF metadata as shown in FIG. 21(d),
(D-1) The receiving device 20 decodes the media data after acquiring the MPU metadata and further acquiring the MF metadata.
(D-2) After acquiring the MPU metadata, the receiving device 20 decodes the media data without using the MF metadata.
Two decoding methods are possible.

Here, D-1 requires buffering of media data to obtain MF metadata, but it can be decoded using MPU header information, so it can be decoded by a conventional MP4-compliant receiving device. becomes. Further, D-2 does not require buffering of media data to obtain MF metadata, but requires special processing for decoding because it cannot be decoded using MF metadata.

In addition, the method shown in FIG. 21(d) has the advantage that the MF metadata is sent after the media data, so no delay occurs due to encapsulation and end-to-end delay can be reduced.

The receiving device 20 can select one of the above two decoding methods depending on the capability of the receiving device 20 and the quality of service provided by the receiving device 20.

The transmitting device 15 must ensure that decoding can be performed while reducing the occurrence of buffer overflow or underflow in the decoding operation in the receiving device 20. For example, the following parameters can be used as elements for defining a decoder model when decoding is performed using method D-1.

Buffer size for reconfiguring the MPU (MPU buffer)
For example, buffer size=maximum rate×maximum MPU time×α, where the maximum rate is the upper limit rate of the profile and level of the encoded data+the overhead of the MPU header. The maximum MPU time is the maximum time length of a GOP when 1 MPU=1 GOP (video).

Here, the audio may be a GOP unit common to the video, or may be a separate unit. α is a margin to prevent overflow, and may be multiplied or added to maximum rate×maximum MPU time. When multiplied, α≧1, and when added, α≧0.

- Upper limit of decoding delay time from when data is input to the MPU buffer until it is decoded. (TSTD_delay in MPEG-TS STD)
For example, at the time of transmission, the maximum MPU time and the upper limit of the decoding delay time are taken into consideration, and the DTS is set so that the acquisition completion time of the MPU data in the receiver<=DTS.

Further, the transmitting device 15 may add DTS and PTS according to the decoder model when decoding using method D-1. Thereby, the transmitting device 15 guarantees the decoding of the receiving device that performs decoding using method D-1, and at the same time transmits the auxiliary information necessary when decoding is performed using method D-2. You can.

For example, the transmitting device 15 can guarantee the operation of the receiving device that decodes using method D-2 by signaling the pre-buffering time in the decoder buffer when decoding using method D-2.

The pre-buffering time may be included in SI control information such as a message, table, or descriptor, or may be included in the header of an MMT packet or MMT payload. Furthermore, the SEI within the encoded data may be overwritten. DTS and PTS for decoding using method D-1 are stored in the MPU timestamp descriptor and SampleEntry, and DTS and PTS for decoding using method D-2 or pre-buffering time are stored in the MPU timestamp descriptor and SampleEntry. It may be described in SEI.

If the receiving device 20 supports only MP4-compliant decoding using an MPU header, the receiving device 20 selects decoding method D-1 and supports both D-1 and D-2. If so, you may choose one or the other.

The transmitting device 15 may provide DTS and PTS to ensure the decoding operation of one side (D-1 in this description), and may also transmit supplementary information to assist the decoding operation of one side.

Also, when method D-2 is used, there is a possibility that the End-to-End delay will be larger due to the delay caused by pre-buffering of MF metadata compared to when method D-1 is used. is high. Therefore, when the receiving device 20 wants to reduce the end-to-end delay, it may select method D-2 for decoding. For example, when the receiving device 20 always wants to reduce the End-to-End delay, it may always use method D-2. Further, the receiving device 20 may use method D-2 only when operating in a low-delay presentation mode where it wants to present live content, channel selection, zapping operation, etc. with low delay.

FIG. 28 is a flowchart of such a receiving method.

First, the receiving device 20 receives an MMT packet and obtains MPU data (S401). Then, the receiving device 20 (transmission order type determining unit 22) determines whether or not to present the program in the low-delay presentation mode (S402).

If the program is not presented in the low-delay presentation mode (No in S402), the receiving device 20 (random access unit 23 and initialization information acquisition unit 27) uses the header information to acquire random access and initialization information (S405). ). Further, the receiving device 20 (PTS, DTS calculating unit 26, decoding command unit 28, decoding unit 29, presentation unit 30) performs decoding and presentation processing based on the PTS and DTS given by the transmitting side (S406).

On the other hand, when the program is presented in low-latency presentation mode (Yes in S402), the receiving device 20 (random access unit 23 and initialization information acquisition unit 27) acquires random access and initialization information using a decoding method that does not use header information (S403). The receiving device 20 also performs decoding and presentation processing based on auxiliary information for decoding without using the PTS, DTS, and header information added by the transmitting side (S404). Note that in steps S403 and S404, processing may be performed using MPU metadata.

[Transmission/reception method using auxiliary data]
The transmission and reception operations in the case where the MF metadata is transmitted after the media data (cases of (c) and (d) in FIG. 21) have been described above. Next, a method will be described in which the transmitting device 15 can start decoding earlier and reduce End-to-End delay by transmitting auxiliary data having some functions of MF metadata. Here, an example will be described in which auxiliary data is further transmitted based on the transmission method shown in FIG. 21(d), but methods using auxiliary data are shown in FIGS. 21(a) to (c). It is also applicable to other transmission methods.

FIG. 29(a) is a diagram showing an MMT packet transmitted using the method shown in FIG. 21(d). That is, data is transmitted in the order of MPU metadata, media data, and MF metadata.

Here, sample #1, sample #2, sample #3, and sample #4 are samples included in the media data. Note that, although an example in which the media data is stored in the MMT packet in sample units is described here, the media data may be stored in the MMT packet in NAL unit units, or in units obtained by dividing the NAL units. Note that multiple NAL units may be aggregated and stored in the MMT packet.

As explained above in D-1, in the case of the method shown in (d) of FIG. 21, that is, when data is transmitted in the order of MPU metadata, media data, and MF metadata, there is a method in which after acquiring the MPU metadata, the MF metadata is acquired, and then the media data is decoded. This type of D-1 method requires buffering of the media data to acquire the MF metadata, but has the advantage that the D-1 method can be applied to conventional MP4-compliant receiving devices because the decoding is performed using the MPU header information. On the other hand, it has the disadvantage that the receiving device 20 must wait to start decoding until the MF metadata is acquired.

On the other hand, as shown in FIG. 29(b), in the method using auxiliary data, the auxiliary data is transmitted before the MF metadata.

The MF metadata includes information indicating the DTS, PTS, offset, and size of all samples included in the movie fragment. On the other hand, the auxiliary data includes information indicating the DTS, PTS, offset, and size of some of the samples included in the movie fragment.

For example, MF metadata includes information about all samples (sample #1-sample #4), whereas auxiliary data includes information about some samples (sample #1-#2). .

In the case shown in FIG. 29(b), sample #1 and sample #2 can be decoded by using auxiliary data, so End-to-Ent Delay is reduced. Note that the auxiliary data may include any combination of sample information, and the auxiliary data may be transmitted repeatedly.

For example, in (c) of FIG. 29, when transmitting the auxiliary information at timing A, the transmitting device 15 includes the information of sample #1 in the auxiliary information, and when transmitting the auxiliary information at timing B, Include information on sample #1 and sample #2 in the auxiliary information. When transmitting the auxiliary information at timing C, the transmitting device 15 includes information on sample #1, sample #2, and sample #3 in the auxiliary information.

Note that the MF metadata includes information on sample #1, sample #2, sample #3, and sample #4 (information on all samples in the movie fragment).

Ancillary data does not necessarily need to be transmitted immediately after it is generated.

Note that in the header of the MMT packet or MMT payload, a type indicating that auxiliary data is stored is specified.

For example, when auxiliary data is stored in the MMT payload using MPU mode, a data type indicating that it is auxiliary data is specified as the fragment_type field value (for example, FT=3). The auxiliary data may be data based on the moof configuration, or may have other configurations.

When auxiliary data is stored in the MMT payload as a control signal (descriptor, table, message), a descriptor tag, table ID, message ID, etc. are specified to indicate that it is auxiliary data.

Furthermore, PTS or DTS may be stored in the header of the MMT packet or MMT payload.

[Example of generation of auxiliary data]
An example in which the transmitting device generates auxiliary data based on the configuration of moof will be described below. FIG. 30 is a diagram for explaining an example in which the transmitting device generates auxiliary data based on the configuration of moof.

In a normal MP4, a moof is created for a movie fragment, as shown in Figure 20. The moof contains information indicating the DTS, PTS, offset, and size of the samples contained in the movie fragment.

Here, the transmitting device 15 configures MP4 (MP4 file) using only some sample data among the sample data that configures the MPU, and generates auxiliary data.

For example, as shown in FIG. 30(a), the transmitting device 15 generates MP4 using only sample #1 among samples #1 to #4 configuring the MPU, and includes the header of moof+mdat. Use as auxiliary data.

Next, as shown in (b) of FIG. 30, the transmitting device 15 generates an MP4 using samples #1 and #2 from among samples #1-#4 that make up the MPU, and the header of moof+mdat is treated as the next auxiliary data.

Next, as shown in FIG. 30(c), the transmitting device 15 transmits MP4 using sample #1, sample #2, and sample #3 among samples #1 to #4 that constitute the MPU. The header of moof+mdat is used as the next auxiliary data.

Next, as shown in FIG. 30(d), the transmitting device 15 generates all MP4s out of samples #1 to #4 configuring the MPU, and among them, the header of moof+mdat is movie fragment metadata. becomes.

Although the transmitting device 15 generates auxiliary data for every sample here, it may also generate auxiliary data for every N samples. The value of N is an arbitrary number. For example, when transmitting auxiliary data M times when transmitting one MPU, N=total samples/M may be used.

Note that the information indicating the sample offset in moof may be an offset value after the sample entry area for the number of subsequent samples is secured as a NULL area.

Note that the auxiliary data may be generated in such a manner that the MF metadata is fragmented.

[Example of reception operation using auxiliary data]
The reception of auxiliary data generated as described in FIG. 30 will be explained. FIG. 31 is a diagram for explaining reception of auxiliary data. In FIG. 31(a), it is assumed that the number of samples forming the MPU is 30, and auxiliary data is generated and transmitted every 10 samples.

In (a) of FIG. 30, auxiliary data #1 contains samples #1 to #10, auxiliary data #2 contains samples #1 to #20, and MF metadata contains samples #1 to #30. Information is included for each.

Although samples #1 to #10, samples #11 to #20, and samples #21 to #30 are stored in one MMT payload, they may be stored in sample units or NAL units, or in fragments. It may also be stored in an aggregated unit.

The receiving device 20 receives packets of MPU meta, samples, MF meta, and auxiliary data.

The receiving device 20 concatenates the sample data in the order in which it is received (to the end), and after receiving the latest auxiliary data, updates the previous auxiliary data. In addition, the receiving device 20 can finally construct a complete MPU by replacing the auxiliary data with MF metadata.

When the receiving device 20 receives the auxiliary data #1, it concatenates the data as shown in the upper part of FIG. 31(b) to configure MP4. Thereby, the receiving device 20 can parse samples #1 to #10 using the MPU metadata and the information of the auxiliary data #1, and the information on the PTS, DTS, offset, and size included in the auxiliary data. Decoding can be performed based on

Furthermore, upon receiving the auxiliary data #2, the receiving device 20 concatenates the data as shown in the middle row of FIG. 31(b) to configure MP4. Thereby, the receiving device 20 can parse samples #1 to #20 using the MPU metadata and the information in the auxiliary data #2, and the information on the PTS, DTS, offset, and size included in the auxiliary data. decoding can be performed based on the

Furthermore, upon receiving the MF metadata, the receiving device 20 concatenates the data as shown in the lower part of FIG. 31(b) to configure MP4. Thereby, the receiving device 20 can parse samples #1 to #30 using the MPU metadata and MF metadata, and based on the PTS, DTS, offset, and size information included in the MF metadata. can be decrypted using

If there is no auxiliary data, the receiving device 20 can acquire sample information only after receiving the MF metadata, so it was necessary to start decoding after receiving the MF metadata. However, since the transmitting device 15 generates and transmits the auxiliary data, the receiving device 20 can acquire sample information using the auxiliary data without waiting for the reception of MF metadata, thereby speeding up the decoding start time. be able to. Further, since the transmitting device 15 generates the auxiliary data based on the moof described using FIG. 30, the receiving device 20 can use the conventional MP4 parser as is to perform parsing.

Furthermore, newly generated auxiliary data and MF metadata include information on samples that overlap with previously transmitted auxiliary data. Therefore, even if past auxiliary data cannot be acquired due to packet loss, etc., MP4 can be reconstructed using newly acquired auxiliary data and MF metadata, and sample information (PTS, DTS, size, and offset).

Note that the auxiliary data does not necessarily need to include information on past sample data. For example, auxiliary data #1 may correspond to sample data #1-#10, and auxiliary data #2 may correspond to sample data #11-#20. For example, as shown in FIG. 31C, the transmitting device 15 may sequentially transmit complete MF metadata as a data unit and fragments of the data unit as auxiliary data.

Furthermore, the transmitting device 15 may repeatedly transmit auxiliary data or MF metadata in order to prevent packet loss.

Note that the MMT packet and MMT payload in which auxiliary data is stored include an MPU sequence number and an asset ID, similar to the MPU metadata, MF metadata, and sample data.

The reception operation using the above-mentioned auxiliary data will be explained using the flowchart of FIG. 32. FIG. 32 is a flowchart of a receiving operation using auxiliary data.

First, the receiving device 20 receives an MMT packet and analyzes the packet header and payload header (S501). Next, the receiving device 20 analyzes whether the fragment type is auxiliary data or MF metadata (S502), and if the fragment type is auxiliary data, it updates the previous auxiliary data by overwriting it (S503). . At this time, if there is no past auxiliary data of the same MPU, the receiving device 20 uses the received auxiliary data as new auxiliary data. Then, the receiving device 20 acquires a sample and decodes it based on the MPU metadata, auxiliary data, and sample data (S507).

On the other hand, if the fragment type is MF metadata, the receiving device 20 overwrites the past auxiliary data with MF metadata in step S505 (S505). Then, the receiving device 20 acquires a sample in the form of a complete MPU based on the MPU metadata, MF metadata, and sample data, and decodes it (S506).

Although not shown in FIG. 32, in step S502, the receiving device 20 stores the data in a buffer when the fragment type is MPU metadata, and stores the data in the buffer after each sample when the fragment type is sample data. Store the concatenated data in a buffer.

If the auxiliary data cannot be acquired due to packet loss, the receiving device 20 can decode the sample by overwriting with the latest auxiliary data or by using past auxiliary data.

Note that the sending cycle and the number of sending times of the auxiliary data may be predetermined values. Information on the transmission cycle and number of times (count, countdown) may be transmitted together with the data. For example, a transmission cycle, the number of transmissions, and a timestamp such as initial_cpb_removal_delay may be stored in the data unit header.

By transmitting auxiliary data containing information of the first sample of the MPU one or more times before initial_cpb_removal_delay, it is possible to follow the CPB buffer model. At this time, a value based on the picture timing SEI is stored in the MPU timestamp descriptor.

Note that the transmission method in the reception operation using such auxiliary data is not limited to the MMT method, and can be applied to streaming transmission of packets configured in the ISOBMFF file format, such as MPEG-DASH.

[Transmission method when one MPU is composed of multiple movie fragments]
In the explanation from FIG. 19 onward, one MPU is composed of one movie fragment, but here, a case will be described where one MPU is composed of a plurality of movie fragments. FIG. 33 is a diagram showing the configuration of an MPU composed of a plurality of movie fragments.

In FIG. 33, samples (#1-#6) stored in one MPU are stored separately into two movie fragments. A first movie fragment is generated based on samples #1-#3 and a corresponding moof box is generated. A second movie fragment is generated based on samples #4-#6 and a corresponding moof box is generated.

The headers of the moof box and mdat box in the first movie fragment are stored in the MMT payload and MMT packet as movie fragment metadata #1. Meanwhile, the headers of the moof box and mdat box in the second movie fragment are stored in the MMT payload and MMT packet as movie fragment metadata #2. Note that in Figure 33, the MMT payload in which the movie fragment metadata is stored is hatched.

Note that the number of samples configuring an MPU and the number of samples configuring a movie fragment are arbitrary. For example, two movie fragments may be configured, with the number of samples configuring the MPU being the number of samples per GOP, and the number of samples being half the number of samples per GOP being the movie fragment.

Note that although an example in which one MPU includes two movie fragments (moof box and mdat box) is shown here, the number of movie fragments included in one MPU does not have to be two, but may be three or more. Furthermore, the samples stored in a movie fragment may not be divided into equal numbers of samples, but may be divided into an arbitrary number of samples.

Note that in FIG. 33, the MPU metadata unit and the MF metadata unit are each stored in the MMT payload as a data unit. However, the transmitting device 15 may store units such as ftyp, mmpu, moov, and moof as data units in the MMT payload in data unit units, or may store data units in fragmented units in the MMT payload. Good too. Further, the transmitting device 15 may store data units in the MMT payload in units of aggregated data units.

Further, in FIG. 33, samples are stored in the MMT payload in units of samples. However, the transmitting device 15 may configure a data unit not only in units of samples but also in units of NAL units or units of a plurality of NAL units, and store the data units in the MMT payload in units of data units. Further, the transmitting device 15 may store the fragmented data unit in the MMT payload, or may store the data unit in the MMT payload in the aggregated unit.

In FIG. 33, the MPU is configured in the order of moof#1, mdat#1, moof#2, mdat#2, and an offset is added to moof#1 as if the corresponding mdat#1 were attached after it. However, an offset may also be added to mdat#1 as if it were attached before moof#1. In this case, however, movie fragment metadata cannot be generated in the form of moof+mdat, and the headers of moof and mdat are transmitted separately.

Next, the transmission order of MMT packets when the MPU having the configuration described in FIG. 33 is transmitted will be described. FIG. 34 is a diagram for explaining the transmission order of MMT packets.

FIG. 34(a) shows the transmission order when MMT packets are transmitted in the MPU configuration order shown in FIG. 33. Specifically, (a) in FIG. 34 shows MPU meta, MF meta #1, media data #1 (samples #1-#3), MF meta #2, media data #2 (samples #4-#6). ) is shown below.

In (b) of FIG. 34, MPU meta, media data #1 (samples #1 to #3), MF meta #1, media data #2 (samples #4 to #6), and MF meta #2 are transmitted in this order. Give an example.

In (c) of FIG. 34, media data #1 (samples #1 to #3), MPU meta, MF meta #1, media data #2 (samples #4 to #6), and MF meta #2 are transmitted in this order. Give an example.

MF meta #1 is generated using samples #1 to #3, and MF meta #2 is generated using samples #4 to #6. Therefore, when the transmission method shown in FIG. 34A is used, a delay occurs in the transmission of sample data due to encapsulation.

On the other hand, when the transmission methods shown in FIGS. 34(b) and 34(c) are used, samples can be transmitted without waiting for MF meta to be generated, so the delay due to encapsulation is reduced. This does not occur, and the end-to-end delay can be reduced.

Also, in the transmission order (a) in FIG. 34, one MPU is divided into multiple movie fragments, and the number of samples stored in the MF meta is smaller than in the case of FIG. The amount of delay due to encapsulation can be made smaller than in the case of encapsulation.

Note that, in addition to the method shown here, for example, the transmitting device 15 may connect MF meta #1 and MF meta #2 and transmit them together at the end of the MPU. In this case, MF metas of different movie fragments may be aggregated and stored in one MMT payload. Further, MF metas of different MPUs may be aggregated together and stored in the MMT payload.

[Reception method when one MPU is composed of multiple movie fragments]
Here, an example of the operation of the receiving device 20 that receives and decodes MMT packets transmitted in the transmission order described in FIG. 34(b) will be described. 35 and 36 are diagrams for explaining such an example of operation.

The receiving device 20 receives MMT packets including an MPU meta, a sample, and an MF meta, which are transmitted in the transmission order shown in FIG. 35. Sample data is concatenated in the order received.

The receiving device 20 concatenates data as shown in (1) of FIG. 36 at T1, which is the time when MF meta #1 is received, and forms MP4. Thereby, the receiving device 20 can acquire samples #1 to #3 based on the MPU metadata and the information in the MF meta #1, and the information on the PTS, DTS, offset, and size included in the MF meta. Decoding can be performed based on

Further, the receiving device 20 concatenates data at T2, which is the time when MF meta #2 is received, as shown in (2) of FIG. 36, and configures MP4. Thereby, the receiving device 20 can acquire samples #4 to #6 based on the MPU metadata and the information on the MF meta #2, and can obtain samples #4 to #6 based on the information on the PTS, DTS, offset, and size of the MF meta. can be decrypted using In addition, the receiving device 20 concatenates the data as shown in (3) of FIG. 36 to configure MP4, thereby generating samples #1 to #6 based on the information of MF meta #1 and MF meta #2. You may obtain it.

By dividing one MPU into a plurality of movie fragments, the time it takes for the MPU to acquire the first MF meta is shortened, so the decoding start time can be accelerated. Furthermore, the buffer size for storing samples before decoding can be reduced.

Note that the transmitting device 15 determines that the time from transmitting (or receiving) the first sample in the movie fragment to transmitting (or receiving) the MF meta corresponding to the movie fragment is shorter than the initial_cpb_removal_delay specified by the encoder. The division unit of the movie fragment may be set so that By setting in this way, the reception buffer can follow the cpb buffer, and low-delay decoding can be achieved. In this case, absolute time based on initial_cpb_removal_delay can be used for PTS and DTS.

Further, the transmitting device 15 may divide the movie fragments at equal intervals, or divide subsequent movie fragments at shorter intervals than the previous movie fragment. Thereby, the receiving device 20 can always receive the MF meta including information about the sample before decoding the sample, and continuous decoding becomes possible.

The following two methods can be used to calculate the absolute time of PTS and DTS.

(1) The absolute time of PTS and DTS is determined based on the reception time (T1 or T2) of MF meta #1 and MF meta #2 and the relative time of PTS and DTS included in the MF meta.

(2) The absolute time of the PTS and DTS is determined based on the absolute time signaled from the transmitting side, such as the MPU time stamp descriptor, and the relative time of the PTS and DTS included in the MF meta.

Furthermore, (2-A) the absolute time signaled by the transmitting device 15 may be an absolute time calculated based on initial_cpb_removal_delay specified by the encoder.

Furthermore, (2-B) the absolute time signaled by the transmitting device 15 may be an absolute time calculated based on a predicted value of the MF meta reception time.

Note that MF meta #1 and MF meta #2 may be transmitted repeatedly. By repeatedly transmitting MF meta #1 and MF meta #2, the receiving device 20 can obtain the MF meta again even if it is unable to obtain the MF meta due to packet loss or the like.

An identifier indicating the order of the movie fragments can be stored in the payload header of the MFU containing the samples making up the movie fragments. On the other hand, the MMT payload does not include an identifier indicating the order of MF meta that constitutes a movie fragment. Therefore, the receiving device 20 identifies the order of the MF meta using packet_sequence_number. Alternatively, the transmitting device 15 stores an identifier indicating which movie fragment the MF meta belongs to in control information (message, table, descriptor), an MMT header, an MMT payload header, or a data unit header, and sends the signal. You may.

The transmitting device 15 may transmit the MPU meta, MF meta, and samples in a predetermined transmission order, and the receiving device 20 may perform the receiving process based on the predetermined transmission order. The transmitting device 15 may also signal the transmission order, and the receiving device 20 may select (determine) the receiving process based on the signaling information.

The above reception method will be explained using FIG. 37. FIG. 37 is a flowchart of the operation of the reception method described in FIGS. 35 and 36.

First, the receiving device 20 determines (identifies) whether the data included in the payload is MPU metadata, MF metadata, or sample data (MFU) based on the fragment type indicated in the MMT payload (S601 , S602). If the data is sample data, the receiving device 20 buffers the sample and waits for reception of MF metadata corresponding to the sample and start of decoding (S603).

On the other hand, in step S602, if the data is MF metadata, the receiving device 20 acquires sample information (PTS, DTS, position information, and size) from the MF metadata, and applies the acquired sample information to the sample information. A sample is obtained based on the PTS and the DTS, and the sample is decoded and presented based on the PTS and DTS (S604).

Although not shown, if the data is MPU metadata, the MPU metadata includes initialization information necessary for decoding. Therefore, the receiving device 20 accumulates this and uses it to decode the sample data in step S604.

Note that when the receiving device 20 stores the received MPU data (MPU metadata, MF metadata, and sample data) in the storage device, it rearranges the received MPU data into the MPU configuration described in FIG. 19 or FIG. 33. After that, it accumulates.

In addition, on the transmitting side, a packet sequence number is assigned to MMT packets with the same packet ID. At this time, the packet sequence number may be assigned after the MMT packets containing the MPU metadata, MF metadata, and sample data are rearranged in the transmission order, or the packet sequence number may be assigned in the order before rearrangement.

When packet sequence numbers are assigned in the order before rearrangement, the receiving device 20 can rearrange the data in the configuration order of the MPU based on the packet sequence numbers, making storage easier.

[Method of detecting the beginning of an access unit and the beginning of a slice segment]
A method for detecting the beginning of an access unit or the beginning of a slice segment based on the information of the MMT packet header and the MMT payload header will be described.

Here, when non-VCL NAL units (access unit delimiter, VPS, SPS, PPS, SEI, etc.) are collectively stored in the MMT payload as a data unit, and when each non-VCL NAL unit is stored as a data unit, Two examples of aggregating and storing in one MMT payload are shown below.

FIG. 38 is a diagram showing a case where non-VCL NAL units are individually treated as data units and aggregated.

In the case of FIG. 38, the start of the access unit is an MMT packet whose fragment_type value is MFU, and is the start data of an MMT payload that includes a data unit whose aggregation_flag value is 1 and whose offset value is 0. In this case, the Fragmentation_indicator value is 0.

In the case of FIG. 38, the beginning of the slice segment is an MMT packet whose fragment_type value is MFU, and is the beginning data of an MMT payload whose aggregation_flag value is 0 and fragmentation_indicator value is 00 or 01.

Figure 39 shows a case where non-VCL NAL units are grouped together into a data unit. Note that the field values of the packet header are as shown in Figure 17 (or Figure 18).

In the case of FIG. 39, the head of the access unit is the head data of the payload in the packet whose Offset value is 0.

Further, in the case of FIG. 39, the beginning of the slice segment has an Offset value different from 0, and the beginning data of the payload of the packet whose fragmentation indicator value is 00 or 01 becomes the beginning of the slice segment.

[Reception process when packet loss occurs]
Typically, when transmitting MP4 format data in an environment where packet loss occurs, the receiving device 20 restores packets using ALFEC (Application Layer FEC), packet retransmission control, or the like.

However, if AL-FEC is not used in streaming such as broadcasting and packet loss occurs, the packets cannot be restored.

After data is lost due to packet loss, the receiving device 20 needs to restart decoding of video and audio. To do this, the receiving device 20 needs to detect the beginning of the access unit or NAL unit and start decoding from the beginning of the access unit or NAL unit.

However, since there is no start code at the beginning of an MP4 format NAL unit, the receiving device 20 cannot detect the beginning of an access unit or NAL unit even if it analyzes the stream.

FIG. 40 is a flowchart of the operation of the receiving device 20 when a packet loss occurs.

The receiving device 20 detects packet loss based on the packet sequence number, packet counter, fragment counter, etc. in the header of the MMT packet or MMT payload (S701), and determines which packet has been lost from the context relationship (S702). .

If it is determined that no packet loss has occurred (No in S702), the receiving device 20 constructs an MP4 file and decodes the access unit or NAL unit (S703).

If it is determined that a packet loss has occurred (Yes in S702), the receiving device 20 generates a NAL unit corresponding to the NAL unit in which the packet loss occurred using dummy data, and configures an MP4 file (S704). When putting dummy data into a NAL unit, the receiving device 20 indicates that the data is dummy data as the type of the NAL unit.

Further, the receiving device 20 detects the beginning of the next access unit or NAL unit based on the method described in FIGS. 17, 18, 38, and 39, and inputs the data from the beginning to the decoder. Decoding can be restarted (S705).

Note that when a packet loss occurs, the receiving device 20 may restart decoding from the beginning of the access unit and the NAL unit based on information detected based on the packet header, or may resume decoding from the beginning of the access unit and the NAL unit or decode the dummy data NAL unit. Decoding may be restarted from the beginning of the access unit and NAL unit based on the header information of the reconstructed MP4 file, including the header information of the reconstructed MP4 file.

When storing an MP4 file (MPU), the receiving device 20 may separately obtain and store (replace) packet data (such as NAL units) lost due to packet loss from broadcasting or communication.

At this time, when acquiring the lost packet from the communication, the receiving device 20 sends the information of the lost packet (packet ID, MPU sequence number, packet sequence number, IP data flow number, IP address, etc.) to the server. and obtain the packet. The receiving device 20 may obtain not only the lost packet but also a group of packets before and after the lost packet at the same time.

[How to configure movie fragments]
Here, a method for configuring movie fragments will be explained in detail.

As explained with reference to FIG. 33, the number of samples making up a movie fragment and the number of movie fragments making up one MPU are arbitrary. For example, the number of samples that make up a movie fragment and the number of movie fragments that make up one MPU may be a fixed predetermined number, or may be determined dynamically.

Here, by configuring the movie fragment on the transmitting side (transmitting device 15) to satisfy the following conditions, low-latency decoding can be guaranteed on the receiving device 20.

The conditions are as follows.

The transmitting device 15 transmits the sample data so that the receiving device 20 can always receive the MF meta including information about the sample (Sample(i)) before the decoding time (DTS(i)) of the sample. MF meta is generated and transmitted using the divided units as movie fragments.

Specifically, the transmitting device 15 constructs a movie fragment using samples encoded before DTS(i) (including the i-th sample).

As a method for dynamically determining the number of samples making up a movie fragment and the number of movie fragments making up one MPU so as to guarantee low-delay decoding, for example, the following method is used.

(1) When decoding starts, the decoding time DTS(0) of the first sample Sample(0) of the GOP is a time based on initial_cpb_removal_delay. The transmitting device constructs a first movie fragment using samples that have already been encoded at a time before DTS(0). The transmitting device 15 also generates MF metadata corresponding to the first movie fragment and transmits it at a time before DTS(0).

(2) The transmitting device 15 configures movie fragments in subsequent samples as well so as to satisfy the above conditions.

For example, when the first sample of a movie fragment is the k-th sample, the MF meta of the movie fragment including the k-th sample is transmitted by the decoding time DTS(k) of the k-th sample. If the encoding completion time of the l-th sample is before DTS(k) and the encoding completion time of the (l+1)th sample is after DTS(k), the transmitting device 15 transmits the A movie fragment is constructed using the lth sample from the samples in .

Note that the transmitting device 15 may configure a movie fragment using samples from the k-th sample to samples less than the l-th sample.

(3) After the encoding of the last MPU sample is completed, the transmitting device 15 configures a movie fragment using the remaining samples, generates MF metadata corresponding to the movie fragment, and transmits it.

Note that the transmitting device 15 may not configure a movie fragment using all encoded samples, but may configure a movie fragment using some encoded samples.

Note that the above example shows an example in which the number of samples forming a movie fragment and the number of movie fragments forming one MPU are dynamically determined based on the above conditions to ensure low-delay decoding. Ta. However, the method for determining the number of samples and the number of movie fragments is not limited to this method. For example, the number of movie fragments constituting one MPU may be fixed to a predetermined value, and the number of samples may be determined to satisfy the above conditions. Furthermore, the number of movie fragments constituting one MPU and the time at which the movie fragments are divided (or the code amount of the movie fragments) may be fixed to predetermined values, and the number of samples may be determined so as to satisfy the above conditions.

In addition, if the MPU is divided into multiple movie fragments, information indicating whether the MPU is divided into multiple movie fragments, attributes of the divided movie fragments, or attributes of the MF meta for the divided movie fragments. may be sent.

Here, the attribute of a movie fragment is information indicating whether the movie fragment is the first movie fragment of the MPU, the last movie fragment of the MPU, or any other movie fragment.

In addition, the attributes of the MF meta include whether the MF meta corresponds to the first movie fragment of the MPU, the MF meta that corresponds to the last movie fragment of the MPU, or the MF meta corresponds to other movie fragments. This is information indicating whether the MF meta is used or not.

Note that the transmitting device 15 may store and transmit the number of samples making up a movie fragment and the number of movie fragments making up one MPU as control information.

[Operation of receiving device]
The operation of the receiving device 20 based on the movie fragment configured as described above will be explained.

The receiving device 20 determines the absolute time of each PTS and DTS based on the absolute time signaled from the transmitting side, such as the MPU timestamp descriptor, and the relative time of the PTS and DTS included in the MF meta.

Based on information on whether the MPU is divided into multiple movie fragments, the receiving device 20 performs the following processing based on the attributes of the divided movie fragments if the MPU is divided.

(1) When the movie fragment is the first movie fragment of the MPU, the receiving device 20 determines the absolute time of the PTS of the first sample included in the MPU timestamp descriptor and the relative time of the PTS and DTS included in the MF meta. to generate the absolute time of PTS and DTS.

(2) If the movie fragment is not the first movie fragment of the MPU, the receiving device 20 uses the relative time of the PTS and DTS included in the MF meta, without using the information of the MPU timestamp descriptor, to generates an absolute time.

(3) If the movie fragment is the last movie fragment of the MPU, the receiving device 20 resets the PTS and DTS calculation process (relative time addition process) after calculating the absolute time of the PTS and DTS of all samples. do. Note that the reset process may be performed in the movie fragment at the beginning of the MPU.

The receiving device 20 may determine whether the movie fragment is divided as follows. The receiving device 20 may also obtain attribute information of the movie fragment as follows.

For example, the receiving device 20 may determine whether the movie fragments are divided based on the identifier movie_fragment_sequence_number field value indicating the order of movie fragments indicated in the MMTP (MMT Protocol) payload header.

Specifically, when the number of movie fragments included in one MPU is 1, the movie_fragment_sequence_number field value is 1, and the field value has a value of 2 or more, , it may be determined that the MPU is divided into multiple movie fragments.

In addition, the receiving device 20 may determine that an MPU is divided into multiple movie fragments if the number of movie fragments contained in one MPU is 1, the movie_fragment_sequence_number field value is 0, and the field value is a value other than 0.

The attribute information of the movie fragment may also be determined based on the movie_fragment_sequence_number.

In addition, without using movie_fragment_sequence_number, it is also possible to determine whether a movie fragment is divided and the attribute information of the movie fragment by counting the transmission of movie fragments and MF meta contained in one MPU.

With the configuration of the transmitting device 15 and receiving device 20 as described above, the receiving device 20 can receive movie fragment metadata at shorter intervals than the MPU, and can start decoding with low delay. Further, by using decoding processing based on the MP4 parsing method, it is possible to perform decoding with low delay.

The receiving operation when the MPU is divided into multiple movie fragments as described above will be explained using a flowchart. Figure 41 is a flowchart of the receiving operation when the MPU is divided into multiple movie fragments. Note that this flowchart illustrates the operation of step S604 in Figure 37 in more detail.

First, the receiving device 20 acquires MF metadata if the data type is MF meta based on the data type indicated in the MMTP payload header (S801).

Next, the receiving device 20 determines whether the MPU is divided into a plurality of movie fragments (S802), and if the MPU is divided into a plurality of movie fragments (Yes in S802), the received MF It is determined whether the metadata is the first metadata of the MPU (S803). If the received MF metadata is the MF metadata at the beginning of the MPU (Yes in S803), the receiving device 20 sets the absolute time of the PTS indicated in the MPU timestamp descriptor, and the PTS and MF metadata indicated in the MF metadata. The absolute time of PTS and DTS is calculated from the relative time of DTS (S804), and it is determined whether it is the last metadata of the MPU (S805).

On the other hand, if the received MF metadata is not the MF metadata at the beginning of an MPU (No in S803), the receiving device 20 does not use the information in the MPU timestamp descriptor, but calculates the absolute times of the PTS and DTS using the relative times of the PTS and DTS indicated in the MF metadata (S808), and proceeds to processing in step S805.

If it is determined in step S805 that it is the last MF metadata of the MPU (Yes in S805), the receiving device 20 resets the PTS and DTS calculation process after calculating the absolute time of the PTS and DTS of all samples. do. If it is determined in step S805 that the MF metadata is not the last MF metadata of the MPU (No in S805), the receiving device 20 ends the process.

Further, if it is determined in step S802 that the MPU is not divided into multiple movie fragments (No in S802), the receiving device 20 acquires sample data based on the MF metadata transmitted after the MPU. Then, PTS and DTS are determined (S807).

Then, although not shown, the receiving device 20 finally performs decoding and presentation processing based on the determined PTS and DTS.

[Issues that occur when dividing movie fragments and their solutions]
So far, we have described how to reduce end-to-end delay by dividing movie fragments. From now on, we will explain new problems that arise when movie fragments are divided, and solutions to them.

First, as background, the picture structure in encoded data will be explained. FIG. 42 is a diagram illustrating an example of a predicted structure of a picture in each TemporalId when achieving temporal scalability.

In coding methods such as MPEG-4 AVC and HEVC (High Efficiency Video Coding), temporal scalability is achieved by using B pictures (bidirectional reference predicted pictures) that can be referenced from other pictures. can.

TemporalId shown in (a) of FIG. 42 is an identifier of the hierarchy of the encoding structure, and the larger the value of TemporalId, the deeper the hierarchy. A square block indicates a picture, Ix in the block is an I picture (intra-screen predicted picture), Px is a P picture (forward reference predicted picture), and Bx and bx are B pictures (bidirectional reference predicted picture). show. The x in Ix/Px/Bx indicates a display order and indicates the order in which pictures are displayed. Arrows between pictures indicate reference relationships; for example, picture B4 indicates that a predicted image is generated using I0 and B8 as reference images. Here, one picture is prohibited from using another picture whose TemporalId is larger than its own TemporalId as a reference image. The reason why the layers are specified is to provide temporal scalability. For example, in FIG. 42, when all pictures are decoded, a video of 120 fps (frame per second) is obtained, but only the layers with TemporalId from 0 to 3 When decoded, a 60fps video is obtained.

FIG. 43 is a diagram showing the relationship between decoding time (DTS) and display time (PTS) in each picture in FIG. 42. For example, picture I0 shown in FIG. 43 is displayed after the decoding of B4 is completed so that no gap occurs in decoding and display.

As shown in FIG. 43, when the prediction structure includes a B picture, the decoding order and the display order are different, so the receiving device 20 performs picture delay processing and rearranges the pictures after decoding them. (Reorder) processing is required.

Examples of picture prediction structures with temporal scalability have been described above, but even when temporal scalability is not used, picture delay processing and reorder processing may be necessary depending on the prediction structure. be. FIG. 44 is a diagram illustrating an example of a predicted structure of a picture that requires picture delay processing and reorder processing. Note that the numbers in FIG. 44 indicate the decoding order.

As shown in FIG. 44, depending on the prediction structure, the first sample in decoding order may be different from the first sample in presentation order. In FIG. 44, the first sample in presentation order is different from the first sample in decoding order. This is the fourth sample. Note that FIG. 44 shows an example of a prediction structure, and the prediction structure is not limited to such a structure. In other prediction structures, the first sample in decoding order and the first sample in presentation order may be different.

Similar to FIG. 33, FIG. 45 is a diagram showing an example in which an MPU configured in MP4 format is divided into a plurality of movie fragments and stored in MMTP payloads and MMTP packets. Note that the number of samples configuring an MPU and the number of samples configuring a movie fragment are arbitrary. For example, two movie fragments may be configured, with the number of samples configuring the MPU being the number of samples per GOP, and the number of samples being half the number of samples per GOP being the movie fragment. One sample may be one movie fragment, and the samples making up the MPU may not be divided.

Although FIG. 45 shows an example in which one MPU includes two movie fragments (moof box and mdat box), one MPU does not need to include two movie fragments. The number of movie fragments included in one MPU may be three or more, or may be the number of samples included in the MPU. Further, the samples stored in a movie fragment may not be divided into equal numbers of samples, but may be divided into an arbitrary number of samples.

The movie fragment metadata (MF metadata) includes information on the PTS, DTS, offset, and size of the sample included in the movie fragment, and when the receiving device 20 decodes the sample, the receiving device 20 The PTS and DTS are extracted from the MF meta that includes information, and the decoding timing and presentation timing are determined.

From here on, for detailed explanation, the absolute value of the decoding time of the i sample will be written as DTS(i), and the absolute value of the presentation time will be written as PTS(i).

Specifically, the information of the i-th sample among the timestamp information stored in the moof in MF meta is the relative value of the decoding time of the i-th sample and the (i+1)-th sample, and the i-th These are the relative values of the decoding time and presentation time of the sample, and these are hereinafter referred to as DT(i) and CT(i).

Movie fragment metadata #1 includes DT(i) and CT(i) of samples #1-#3, and movie fragment metadata #2 includes DT(i) of samples #4-#6. ) and CT(i).

Further, the absolute value of PTS of the access unit at the beginning of the MPU is stored in the MPU time stamp descriptor, etc., and the receiving device 20 calculates the PTS and DTS based on the PTS_MPU of the access unit at the beginning of the MPU, and the CT and DT. calculate.

FIG. 46 is a diagram for explaining a method of calculating PTS and DTS and problems when an MPU is configured by samples #1 to #10.

(a) of FIG. 46 shows an example in which the MPU is not divided into movie fragments, (b) in FIG. 46 shows an example in which the MPU is divided into two movie fragments of 5 samples each, and (c) in FIG. ) shows an example in which the MPU is divided into 10 movie fragments in units of samples.

As described in Figure 45, when the PTS and DTS are calculated using the MPU timestamp descriptor and the timestamp information (CT and DT) in the MP4, the first sample in the presentation order in Figure 44 is the fourth in the decoding order. Therefore, the PTS stored in the MPU timestamp descriptor is the PTS (absolute value) of the fourth sample in the decoding order. Note that hereafter, this sample will be referred to as sample A. Also, the first sample in the decoding order will be referred to as sample B.

Since the absolute time information related to the time stamp is only the information of the MPU time stamp descriptor, the receiving device 20 calculates the PTS (absolute time) and DTS (absolute time) of the other samples until the A sample arrives. Can not. The receiving device 20 also cannot calculate the PTS and DTS of the B sample.

In the example of FIG. 46(a), sample A is included in the same movie fragment as sample B and is stored in one MF meta. Therefore, the receiving device 20 can determine the DTS of sample B immediately after receiving the MF meta.

In the example of FIG. 46(b), the A sample is included in the same movie fragment as the B sample, and is stored in one MF meta. Therefore, the receiving device 20 can determine the DTS of the B sample immediately after receiving the MF meta.

In the example of (c) in FIG. 46, sample A is included in a different movie fragment from sample B. Therefore, the receiving device 20 cannot determine the DTS of sample B until it has received the MF meta including the CT and DT of the movie fragment that includes sample A.

Therefore, in the example of FIG. 46(c), the receiving device 20 cannot start decoding immediately after the B sample arrives.

In this way, if a movie fragment including a B sample does not include an A sample, the receiving device 20 decodes the B sample only after receiving the MF meta related to the movie fragment including the A sample. Unable to start.

This problem occurs when the first sample in the presentation order and the first sample in the decoding order do not match, and the movie fragment is divided until the A sample and B sample are no longer stored in the same movie fragment. do. Moreover, this problem occurs regardless of whether the MF meta is postponed or postponed.

In this way, if the first sample in the presentation order and the first sample in the decoding order do not match, and if the A sample and the B sample are not stored in the same movie fragment, the DTS cannot be determined. Therefore, the transmitting device 15 separately transmits the DTS (absolute value) of the B samples or information that allows the receiving side to calculate the DTS (absolute value) of the B samples. Such information may be transmitted using control information, a packet header, or the like.

The receiving device 20 uses such information to calculate the DTS (absolute value) of the B samples. FIG. 47 is a flowchart of the reception operation when the DTS is calculated using such information.

The receiving device 20 receives the movie fragment at the beginning of the MPU (S901), and determines whether the A sample and the B sample are stored in the same movie fragment (S902). If they are stored in the same movie fragment (Yes in S902), the receiving device 20 calculates the DTS using only the MF meta information without using the DTS (absolute time) of the B sample, and starts decoding. (S904). Note that in step S904, the receiving device 20 may determine the DTS using the DTS of B samples.

On the other hand, if the A sample and the B sample are not stored in the same movie fragment in step S902 (No in S902), the receiving device 20 obtains the DTS (absolute time) of the B sample, determines the DTS, and decodes the sample. (S903).

Note that in the above explanation, the absolute value of the decoding time and the absolute value of the presentation time of each sample is calculated using MF meta (time stamp information stored in MP4 format moof) in the MMT standard. Although an example has been described, it goes without saying that the MF meta may be replaced with any control information that can be used to calculate the absolute value of the decoding time and the absolute value of the presentation time of each sample. An example of such control information is the relative value CT(i) of the decoding times of the i-th sample and the (i+1)-th sample mentioned above, and the relative value CT(i) of the presentation times of the i-th sample and the (i+1)-th sample. The control information replaced with relative values, the relative value CT(i) of the decoding time of the i-th sample and the (i+1)-th sample, and the relative value of the presentation time of the i-th sample and the (i+1)-th sample. There is control information that includes both.

(Embodiment 3)
[overview]
In Embodiment 3, a content transmission method and data structure when transmitting content such as video, audio, subtitles, and data broadcasting by broadcasting will be described. That is, a content transmission method and data structure specialized for playing a broadcast stream will be explained.

Note that in Embodiment 3, an example will be described in which the MMT method (hereinafter also simply referred to as MMT) is used as the multiplexing method, but other multiplexing methods such as MPEG-DASH or RTP may also be used. .

First, details of a method for storing a data unit (DU: Data Unit) in a payload in MMT will be explained. FIG. 48 is a diagram for explaining a method of storing a data unit in a payload in MMT.

In MMT, a transmitting device stores a part of data that constitutes an MPU in an MMTP payload as a data unit, attaches a header, and transmits the data. The header includes an MMTP payload header and an MMTP packet header. Note that the data unit may be a NAL unit or a sample. When MMTP packets are scrambled, the payload is scrambled.

FIG. 48(a) shows an example in which a transmitting device aggregates a plurality of data units and stores them in one payload. In the example of FIG. 48(a), a data unit header (DUH) and a data unit length (DUL) are added to the beginning of each of a plurality of data units, and the data unit header and A plurality of data units to which a data unit length is assigned are collectively stored in the payload.

FIG. 48(b) shows an example in which one data unit is stored in one payload. In the example of FIG. 48(b), a data unit header is added to the beginning of the data unit and stored in the payload. FIG. 48C shows an example in which one data unit is divided, a data unit header is added to the divided data unit, and the data unit header is stored in the payload.

Data units include timed-MFU, which is media that includes information related to synchronization such as video, audio, or subtitles; non-timed-MFU, which is media that does not include information related to synchronization such as files; MPU metadata; and MF metadata, and a data unit header is determined according to the type of data unit. Note that MPU metadata and MF metadata do not have data unit headers. In addition, a transmitting device cannot aggregate different types of data units in principle, but may be specified to be able to aggregate different types of data units. For example, when the size of the MF metadata is small, such as when it is divided into movie fragments for each sample, the number of packets can be reduced by aggregating the MF metadata and media data, and the transmission capacity can also be reduced.

When the data unit is an MFU, some information of the MPU, such as information for configuring the MPU (MP4), is stored as a header.

For example, the header of a timed-MFU includes movie_fragment_sequence_number, sample_number, offset, priority, and dependency_counter, and the header of a non-timed-MFU includes i tem_iD is included. The meaning of each field is shown in standards such as ISO/IEC23008-1 or ARIB STD-B60. The meaning of each field defined in such standards will be explained below.

movie_fragment_sequence_number indicates the sequence number of the movie fragment to which the MFU belongs, and is also indicated in ISO/IEC14496-12.

sample_number indicates the sample number to which the MFU belongs, and is also specified in ISO/IEC14496-12.

offset indicates the offset amount of the MFU in the sample to which the MFU belongs, in bytes.

Priority indicates the relative importance of the MFU in the MPU to which it belongs, with an MFU with a higher priority number being more important than an MFU with a lower priority number.

dependency_counter indicates the number of MFUs on which the decoding process depends on the MFU (that is, the number of MFUs for which the decoding process cannot be performed unless this MFU is decoded). For example, when the MFU is HEVC and a B picture or a P picture references an I picture, the B picture or P picture cannot be decoded unless the I picture is decoded.

Therefore, when the MFU is in sample units, the dependency_counter in the MFU of an I picture indicates the number of pictures that reference the I picture. When the MFU is in NAL unit units, the dependency_counter in the MFU belonging to the I picture indicates the number of NAL units that belong to the picture that references the I picture. Furthermore, in the case of a video signal that has been hierarchically coded in the temporal direction, since the MFU of the enhancement layer depends on the MFU of the base layer, the dependency_counter in the MFU of the base layer indicates the number of MFUs of the enhancement layer. This field can only be generated after the number of dependent MFUs has been determined.

item_id indicates an identifier that uniquely identifies an item.

The method of storing control information in a payload in MMT is similar to the method of storing a data unit in a payload, and can be explained by replacing the data unit in FIG. 48 with control information and replacing the data unit length with the control information length. Note that there is no information equivalent to the data unit header.

[MP4 non-support mode]
As explained in FIGS. 19 and 21, methods for the transmitting device to transmit MPU in MMT include transmitting MPU metadata or MF metadata before or after media data, and transmitting only media data. There is a way to send it. Further, in the receiving device, there are a method of decoding using a receiving device and a receiving method compliant with MP4, and a method of decoding without using a header.

As a data transmission method specialized for broadcast stream playback, there is, for example, a transmission method that does not support MP4 reconfiguration in a receiving device.

A transmission method that does not support MP4 reconfiguration in the receiving device is, for example, a method in which metadata (MPU metadata and MF metadata) is not transmitted, as shown in FIG. 21(b). In this case, the field value of the fragment type (information indicating the type of data unit) included in the MMTP packet is fixed to 2 (=MFU).

As explained above, if metadata is not transmitted, an MP4-compliant receiving device cannot decode the received data as MP4, but it is possible to decode it without using metadata (header).

Therefore, metadata is not necessarily essential information for broadcast stream decoding and playback. Similarly, the information in the data unit header in timed-MFU, described in FIG. 48, is information for reconstructing MP4 in the receiving device. Since there is no need to reconstruct MP4 for broadcast stream playback, the information in the data unit header in timed-MFU (hereinafter also referred to as timed-MFU header) is not necessarily information required for broadcast stream playback.

The receiving device can easily reconstruct MP4 by using metadata and information for reconstructing MP4 in the data unit header (hereinafter also referred to as MP4 configuration information). However, the receiving device cannot easily reconstruct the MP4 even if only one of the metadata and the MP4 configuration information in the data unit header is transmitted. There is little merit in transmitting only either the metadata or the information for reconstructing the MP4, and generating and transmitting unnecessary information results in an increase in processing and a decrease in transmission efficiency.

Therefore, the transmitter uses the method described below to control the data structure and transmission of the MP4 configuration information. The transmitting device determines whether to indicate MP4 configuration information in the data unit header based on whether metadata is transmitted. Specifically, the transmitting device indicates MP4 configuration information in the data unit header when metadata is transmitted, and does not indicate MP4 configuration information in the data unit header when metadata is not transmitted.

As a method of not indicating MP4 configuration information in the data unit header, for example, the following method can be used.

1. The transmitting device sets the MP4 configuration information to reserved and does not operate it. This makes it possible to reduce the amount of processing on the sending side that generates MP4 configuration information (the amount of processing on the transmitting device).

2. The transmitting device deletes the MP4 configuration information and compresses the header. This reduces the amount of processing on the sending side that generates the MP4 configuration information, and also reduces transmission capacity.

Note that, when deleting MP4 configuration information and compressing the header, the transmitting device may display a flag indicating that the MP4 configuration information has been deleted (compressed). The flag is indicated in a header (MMTP packet header, MMTP payload header, data unit header), control information, or the like.

Further, information as to whether or not metadata is to be transmitted may be determined in advance, or may be separately signaled in a header or control information and transmitted to the receiving device.

For example, information as to whether metadata corresponding to the MFU is being transmitted may be stored in the MFU header.

On the other hand, the receiving device can determine whether MP4 configuration information is indicated based on whether metadata is transmitted.

Here, if the data transmission order is fixed (e.g., MPU metadata, MF metadata, media data, etc.), the receiving device can transmit the data based on whether the metadata is received before the media data. It may be determined by

If MP4 configuration information is indicated, the receiving device can use the MP4 configuration information for MP4 reconfiguration. Alternatively, the receiving device can use the MP4 configuration information to detect the beginning of other access units or NAL units.

Note that the MP4 configuration information may be all or a part of the timed-MFU header.

Similarly, the transmitting device selects an item in the non-timed-MFU header based on whether or not metadata is transmitted in the non-timed-MFU header.
It may also be determined whether to indicate the id.

The transmitting device may indicate MP4 configuration information only in either the timed-MFU or the non-timed-MFU. If MP4 configuration information is to be shown to only one of the MFUs, the transmitting device determines whether to show the MP4 configuration information based on whether the MFU is a timed-MFU or a non-timed-MFU, in addition to whether metadata is transmitted. do. The receiving device can determine whether metadata is transmitted and whether MP4 configuration information is indicated based on the timed/non-timed flag.

Note that in the above description, the transmitting device determines whether to indicate MP4 configuration information based on whether metadata (both MPU metadata and MF metadata) is transmitted. However, the transmitting device may not indicate the MP4 configuration information when part of the metadata (either MPU metadata or MF metadata) is not transmitted.

Furthermore, the transmitting device may determine whether to indicate MP4 configuration information based on information other than metadata.

For example, modes such as MP4 support mode/MP4 non-support mode are defined, and the transmitting device indicates the MP4 configuration information in the data unit header in case of MP4 support mode and the data unit header in case of MP4 non-support mode. MP4 configuration information may not be indicated in the unit header. Further, in the case of MP4 support mode, the transmitting device transmits metadata and indicates MP4 configuration information in the data unit header, and in the case of MP4 non-support mode, the transmitting device transmits metadata and indicates MP4 configuration information in the data unit header without transmitting metadata. Also, the MP4 configuration information may not be shown.

[Operation flow of transmitter]
Next, the operational flow of the transmitter will be explained. FIG. 49 is an operation flow of the transmitting device.

The transmitting device first determines whether to transmit metadata (S1001). When the transmitting device determines to transmit metadata (Yes in S1002), the transmitting device moves to step S1003, generates MP4 configuration information, stores it in a header, and transmits it (S1003). In this case, the transmitting device also generates and transmits metadata.

On the other hand, if the transmitting device determines not to transmit metadata (No in S1002), it transmits the MP4 configuration information without generating it or storing it in the header (S1004). In this case, the transmitting device does not generate or transmit metadata.

Whether or not to transmit metadata in step S1001 may be determined in advance, or may be determined based on whether metadata has been generated within the transmitting device, or whether metadata has been transmitted within the transmitting device.

[Operation flow of receiving device]
Next, the operation flow of the receiving device will be explained. FIG. 50 is an operation flow of the receiving device.

First, the receiving device determines whether metadata is being transmitted (S1101). Whether metadata is being transmitted can be determined by monitoring the fragment type in the MMTP packet payload. Also, whether metadata is being transmitted may be determined in advance.

If the receiving device determines that metadata is being transmitted (Yes in S1102), it reconfigures the MP4 and executes decoding processing using the MP4 configuration information (S1103). On the other hand, if it is determined that metadata has not been transmitted (No in S1102), the MP4 reconstruction process is not performed and the decoding process is executed without using the MP4 configuration information (S1104).

Note that the receiving device can detect a random access point, detect the beginning of an access unit, detect the beginning of a NAL unit, etc. without using the MP4 configuration information using the method described above, and can perform decoding processing. , detecting packet loss, and processing recovery from packet loss.

For example, when non-VCL NAL units are individually treated as data units and aggregated as shown in FIG. 38, the beginning of the access unit is the beginning data of the MMT payload whose aggregation_flag value is 1. At this time, the Fragmentation_indicator value is 0.

Further, the beginning of the slice segment is the beginning data of the MMT payload whose aggregation_flag value is 0 and the fragmentation_indicator value is 00 or 01.

The receiving device can detect the beginning of the access unit and the slice segment based on the above information.

In addition, the receiving device may analyze the NAL unit header in a packet that includes the beginning of a data unit whose fragmentation_indicator value is 00 or 01, and detect that the type of the NAL unit is an AU delimiter and that the type of the NAL unit is a slice segment.

[Broadcast simple mode]
So far, we have explained a method that does not support MP4 configuration information in the receiving device as a data transmission method specialized for broadcast stream playback, but this is not the only method for transmitting data specialized for broadcast stream playback. do not have.

For example, the following method may be used as a data transmission method specialized for broadcast stream playback.

- In a fixed broadcast reception environment, the transmitting device does not need to use AL-FEC. If AL-FEC is not used, FEC_type in the MMTP packet header is always fixed to 0.

- The transmitting device may always use AL-FEC in a mobile broadcast reception environment and in a communication UDP transmission mode. When AL-FEC is used, FEC_type in the MMTP packet header is always 0 or 1.

- The transmitting device does not need to perform bulk transmission of assets. If bulk transmission of assets is not performed, location_infolocation indicating the number of transmission locations of assets within the MPT may be fixed to 1.

- The sending device does not have to perform hybrid transmission of assets, programs, and messages.

Further, for example, if the broadcast simple mode is defined and the transmitter is in the broadcast simple mode, the transmitting device may set the MP4 non-support mode, or use the data transmission method specialized for broadcast stream playback as described above. You can also use it as Whether or not it is broadcast simple mode may be determined in advance, or the transmitting device may store a flag indicating that it is broadcast simple mode as control information and transmit it to the receiving device.

The transmitting device may also determine whether the device is in the MP4 non-support mode (whether metadata is transmitted) as described in FIG. 49, and if the device is in the MP4 non-support mode, the transmitting device may use the data transmission method specialized for broadcast stream playback as shown above as the broadcast simple mode.

When the receiving device is in the broadcast simple mode, it is assumed that the receiving device is in an MP4 non-support mode and can perform decoding processing without reconstructing the MP4.

Furthermore, when the receiving device is in the broadcast simple mode, it can determine that the function is specialized for broadcasting, and perform reception processing specialized for broadcasting.

As a result, in the case of broadcast simple mode, by using only broadcast-specific functions, it is possible to not only reduce unnecessary processing for the transmitting device and receiving device, but also to avoid compressing and transmitting unnecessary information. Therefore, transmission overhead can be reduced.

When the MP4 non-support mode is used, hint information supporting storage methods other than the MP4 format may be indicated.

Examples of storage methods other than the MP4 configuration include a method of directly storing MMT packets and IP packets, and a method of converting MMT packets into MPEG-2 TS packets.

Note that in the case of the MP4 non-support mode, a format that does not follow the MP4 configuration may be used.

For example, in the case of an MP4 non-support mode, the data stored in the MFU may be in a format with a byte start code at the beginning of the NAL unit, rather than the MP4 format format with the size of the NAL unit at the beginning of the NAL unit.

In MMT, the asset type indicating the type of asset is described in 4CC registered in MP4REG (http://www.mp4ra.org), and when HEVC is used as the video signal, 'HEV1' or 'HVC1' is used. It will be done. 'HVC1' is a format that may include a parameter set in the sample, and 'HEV1' is a format that does not include a parameter set in the sample but includes a parameter set in the sample entry in the MPU metadata.

In the case of broadcast simple mode or MP4 non-support mode, if MPU metadata and MF metadata are not transmitted, it may be specified that the parameter set is always included in the sample. Further, it may be specified that the format is always 'HVC1' regardless of whether 'HEV1' or 'HVC1' is indicated in the asset type.

[Supplement 1: Transmitting device]
As described above, when metadata is not transmitted, the MP4 configuration information is set to reserved, and the transmitting device that is not operated can be configured as shown in FIG. 51. FIG. 51 is a diagram showing an example of a specific configuration of a transmitting device.

The transmitting device 300 includes an encoding section 301, an adding section 302, and a transmitting section 303. Each of the encoding section 301, the adding section 302, and the transmitting section 303 is realized by, for example, a microcomputer, a processor, a dedicated circuit, or the like.

The encoding unit 301 encodes a video signal or an audio signal to generate sample data. Specifically, the sample data is a data unit.

The adding unit 302 adds header information including MP4 configuration information to sample data, which is data obtained by encoding a video signal or an audio signal. The MP4 configuration information is information for reconstructing the sample data as an MP4 format file on the receiving side, and the content differs depending on whether or not the presentation time of the sample data is determined.

As described above, the adding unit 302 adds movie_fragment_sequence_number, sample_number, offset, priority to the header (header information) of timed-MFU, which is an example of sample data (sample data including information regarding synchronization) for which the presentation time is determined. , and MP4 configuration information such as dependency_counter.

On the other hand, the adding unit 302 adds MP4 configuration information such as item_id to the header (header information) of timed-MFU, which is an example of sample data for which the presentation time is not determined (sample data that does not include information regarding synchronization). Include.

Then, if the transmitting unit 303 does not transmit metadata corresponding to the sample data (for example, in a case like (b) in FIG. 21), the assigning unit 302 determines whether the presentation time of the sample data has been determined Depending on the situation, header information that does not include MP4 configuration information is added to the sample data.

Specifically, when the presentation time of the sample data is determined, the adding unit 302 adds header information that does not include the first MP4 configuration information to the sample data, and when the presentation time of the sample data is determined. If not, header information including second MP4 configuration information is added to the sample data.

For example, as shown in step S1004 in FIG. 49, when the transmitting unit 303 does not transmit the metadata corresponding to the sample data, the providing unit 302 sets the MP4 configuration information to reserved (fixed value). MP4 configuration information is not substantially generated and is not substantially stored in the header (header information). Note that the metadata includes MPU metadata and movie fragment metadata.

The transmitter 303 transmits sample data to which header information is added. More specifically, the transmitter 303 packetizes the sample data to which header information has been added using the MMT method and transmits the packet.

As described above, in the transmission method and reception method specialized for playing back a broadcast stream, there is no need for the receiving device to reconstruct data units into MP4. If the receiving device does not need to reconstruct data into MP4, the processing load of the transmitting device is reduced by not generating unnecessary information such as MP4 configuration information.

On the other hand, the transmitting device must send the necessary information, but must maintain compatibility with the standard so that it does not have to send unnecessary additional information separately.

According to the configuration of the transmitting device 300, by setting the area where MP4 configuration information is stored to a fixed value, the MP4 configuration information is not transmitted, and only necessary information is transmitted based on the standard, and unnecessary information is transmitted. This has the effect of not having to send additional information. In other words, the configuration of the transmitting device and the amount of processing of the transmitting device can be reduced. Furthermore, since unnecessary data is not transmitted, transmission efficiency can be improved.

[Supplement 2: Receiving device]
Further, the receiving device corresponding to the transmitting device 300 may be configured as shown in FIG. 52, for example. FIG. 52 is a diagram showing another example of the configuration of the receiving device.

Receiving device 400 includes a receiving section 401 and a decoding section 402. The receiving section 401 and the decoding section 402 are realized by, for example, a microcomputer, a processor, a dedicated circuit, or the like.

The receiving unit 401 receives sample data, which is data obtained by encoding a video signal or an audio signal, to which header information including MP4 configuration information for reconstructing the sample data as an MP4 format file is added. Receive data.

The decoding unit 402 decodes the sample data without using MP4 configuration information when the metadata corresponding to the sample data is not received by the receiving unit and the presentation time of the sample data is determined. do.

For example, as shown in step S1104 in FIG. 50, if metadata corresponding to the sample data is not received by the receiving unit 401, the decoding unit 402 executes the decoding process without using the MP4 configuration information.

Thereby, the configuration of the receiving device 400 and the amount of processing in the receiving device 400 can be reduced.

(Other embodiments)
Although the transmitting method, receiving method, transmitting device, and receiving device according to the embodiment have been described above, the present invention is not limited to this embodiment.

Furthermore, each processing section included in the transmitting device and receiving device according to the above embodiments is typically realized as an LSI, which is an integrated circuit. These may be integrated into one chip individually, or may be integrated into one chip including some or all of them.

Further, circuit integration is not limited to LSI, and may be realized using a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after the LSI is manufactured, or a reconfigurable processor that can reconfigure the connections and settings of circuit cells inside the LSI may be used.

In each of the above embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

In other words, the transmitting device and the receiving device include a processing circuit and a storage electrically connected to the processing circuit (accessible from the control circuit). The processing circuit includes at least one of dedicated hardware and a program execution unit. Further, when the processing circuit includes a program execution section, the storage device stores a software program executed by the program execution section. The processing circuit uses the storage device to execute the transmission method or reception method according to the above embodiment.

Furthermore, the present invention may be the above-mentioned software program, or may be a non-transitory computer-readable recording medium on which the above-mentioned program is recorded. Furthermore, it goes without saying that the above program can be distributed via a transmission medium such as the Internet.

Moreover, all the numbers used above are exemplified to specifically explain the present invention, and the present invention is not limited to the illustrated numbers.

Furthermore, the division of functional blocks in the block diagram is just an example; multiple functional blocks may be realized as one functional block, one functional block may be divided into multiple functional blocks, or some functions may be moved to other functional blocks. You can. Further, functions of a plurality of functional blocks having similar functions may be processed in parallel or in a time-sharing manner by a single piece of hardware or software.

Furthermore, the order in which the steps included in the above transmission method or reception method are executed is for illustrative purposes in order to concretely explain the present invention, and an order other than the above may be used. Further, some of the above steps may be executed simultaneously (in parallel) with other steps.

The transmission method, reception method, transmission device, and reception device according to one or more aspects of the present invention have been described above based on the embodiments, but the present invention is not limited to these embodiments. do not have. Unless departing from the spirit of the present invention, various modifications that can be thought of by those skilled in the art to this embodiment, and embodiments constructed by combining components of different embodiments may also be applied to one or more of the present invention. may be included within the scope of the embodiment.

INDUSTRIAL APPLICATION This invention is applicable to the apparatus or apparatus which performs media transport, such as video data and audio data.

15, 100, 300 Transmitting device 16, 101, 301 Encoding section 17, 102 Multiplexing section 18, 104 Transmitting section 20, 200, 400 Receiving device 21 Packet filtering section 22 Transmission order type determining section 23 Random access section 24, 212 Control information acquisition section 25 Data acquisition section 26 Calculation section 27 Initialization information acquisition section 28, 206 Decoding instruction section 29, 204A, 204B, 204C, 204D, 402 Decoding section 30 Presentation section 201 Tuner 202 Demodulation section 203 Demultiplexing section 205 Display section 211 Type discrimination section 213 Slice information acquisition section 214 Decoded data generation section 302 Addition section 303 Transmission section 401 Receiving section

Claims (4)

an adding step of adding header information to sample data that is encoded data;
a transmitting step of transmitting the sample data to which the header information is attached,
If the presentation time of the sample data is not determined,
In the adding step, the header information including MP4 configuration information for reconstructing the sample data as an MP4 format file on the receiving side is added to the sample data,
If the presentation time of the sample data is determined,
transmitting control information regarding decoding of the sample data separately from the sample data;
In the adding step, header information that invalidates the MP4 configuration information is added to the sample data,
In the transmitting step, the sample data to which the header information has been added is packetized and transmitted using an MMT (MPEG Media Transport) method;
The header information added to the sample data for which the presentation time is not determined includes item_id as the MP4 configuration information in the MMTP (MMT Protocol) payload. a receiving step of receiving sample data that is encoded data and to which header information is attached;
a decoding step of decoding the sample data;
If the presentation time of the sample data is not determined,
In the decoding step, decoding the sample data using MP4 configuration information included in the header information for reconstructing the sample data as an MP4 format file;
If the presentation time of the sample data is determined,
In the decoding step, decoding the sample data using control information regarding decoding of the sample data, which is received separately from the sample data,
The sample data to which the header information has been added is packetized using the MMT method,
The header information added to the sample data for which the presentation time is not determined includes item_id in the MMTP payload as the MP4 configuration information. an adding unit that adds header information to sample data that is encoded data;
a transmitter that transmits the sample data to which the header information is added,
If the presentation time of the sample data is not determined,
The adding unit adds the header information including MP4 configuration information for reconstructing the sample data as an MP4 format file on the receiving side to the sample data,
If the presentation time of the sample data is determined,
transmitting control information regarding decoding of the sample data separately from the sample data;
The adding unit adds header information that invalidates the MP4 configuration information to the sample data,
The transmitting unit packetizes the sample data to which the header information is attached using an MMT method, and transmits the packetized data.
The header information added to the sample data for which the presentation time is not determined includes item_id in the MMTP payload as the MP4 configuration information. The transmitting device. a receiving unit that receives sample data that is encoded data and to which header information is added;
and a decoding unit that decodes the sample data,
If the presentation time of the sample data is not determined,
The decoding unit performs a decoding process on the sample data using MP4 configuration information included in the header information for reconstructing the sample data as an MP4 format file,
If the presentation time of the sample data is determined,
The decoding unit performs a decoding process on the sample data using control information regarding decoding of the sample data, which is received separately from the sample data;
The sample data to which the header information has been added is packetized using the MMT method,
Header information added to the sample data for which the presentation time is not determined includes item_id in the MMTP payload as the MP4 configuration information. Receiving device.

Images