JP7361287B2 - Transmission method and transmission device - Google Patents

Description

The present invention relates to a transmitting method, a receiving method, a transmitting device, and a receiving device.

As broadcasting and communication services become more sophisticated, the introduction of ultra-high-definition video content such as 8K (7680 x 4320 pixels: hereinafter also referred to as 8K4K) and 4K (3840 x 2160 pixels: hereinafter also referred to as 4K2K) is being considered. has been done. The receiving device needs to decode and display the encoded data of the received ultra-high-definition video in real time, but video with resolutions such as 8K requires a large processing load during decoding. It is difficult to decode moving images in real time with one decoder. Therefore, methods are being considered to reduce the processing load per decoder and achieve real-time processing by parallelizing the decoding process using a plurality of decoders.

Further, the encoded data is multiplexed based on a multiplexing method such as MPEG-2 TS (Transport Stream) or MMT (MPEG Media Transport) before being transmitted. For example, Non-Patent Document 1 discloses a technique for transmitting encoded media data packet by packet according to MMT.

Information technology - High efficiency coding and media delivery in heterogeneous environment - Part1: MPEG media transp ort(MMT), ISO/IEC DIS 23008-1

By the way, as broadcasting and communication services become more sophisticated, the introduction of ultra-high definition video content such as 8K and 4K (3840 x 2160 pixels) is being considered. In multiplexing methods such as MMT (MPEG Media Transport) and MPEG-DASH, media data such as video, audio, and files are multiplexed into ISOBMFF (ISO based media file format) (MP4 format) files and converted into MMT packets. Later, it is transmitted using a broadcast transmission path or a communication transmission path.

However, in MMT transmission, there is a possibility that the buffer operation of the receiving device cannot be guaranteed.

The present invention provides a transmission method that can guarantee buffer operation of a receiving device when transmitting data using a method such as MMT.

In order to achieve the above object, a transmission method according to one aspect of the present invention is a transmission method that transmits a plurality of transmission packets, wherein the transmission packet is composed of a variable-length packet header and a variable-length payload. , a reception buffer model for receiving the plurality of transmission packets to be transmitted includes a first reception buffer model that receives the transmission packets and includes a variable-length packet header and a variable-length payload stored in the received transmission packets. The buffer includes a buffer that converts a packet into a second packet having a header-expanded fixed-length packet header and outputs the second packet obtained by the conversion at a predetermined extraction rate, and Storing the plurality of transmission packets in a predetermined data unit, and transmitting the predetermined data unit in each of a plurality of continuous unit transmission periods, which are periods of unit time that do not overlap with each other. The plurality of transmission packets, the number of which is equal to or less than the predetermined number of packets, are transmitted per unit time, and the plurality of transmission packets include a first transmission packet storing a real packet different from an invalid packet; and a second transmission packet storing an invalid packet, and when storing the first transmission packet in the predetermined data unit, the number of the first transmission packet is counted as the number of first packets. However, when storing the second transmission packet in the predetermined data unit, 1 is added to the quotient obtained by dividing the packet size of the second transmission packet by the maximum value of the packet size of the first transmission packet. The added integer value is counted as the second number of packets, and the plurality of transmission packets are transferred to The first transmission packet is stored in a predetermined data unit, and the maximum value of the packet size of the first transmission packet is 1500B.

Further, a receiving method according to one aspect of the present invention is a receiving method in a receiving device having a buffer, in which a plurality of transmission packets each including a variable-length packet header and a variable-length payload are received, A first packet consisting of a variable length packet header and a variable length payload stored in the plurality of transmitted packets is transferred to a second packet having a header-expanded fixed length packet header using the buffer. The second packet obtained by the conversion is outputted from the buffer at a predetermined extraction rate, and the buffer size of the buffer is equal to the transmission rate of the plurality of transmission packets, the average packet length of the transmission packets , and the plurality of transmission packets are in unit time periods that do not overlap with each other, and each of the plurality of consecutive unit transmission periods is larger than the maximum buffer occupancy calculated using the extraction rate. The first transmission is a transmission packet that is stored in a predetermined data unit and transmitted in a predetermined data unit, and is a transmission packet that is less than or equal to the predetermined number of packets transmitted per unit time, and stores real packets that are different from invalid packets. packet and a second transmission packet storing the invalid packet, the plurality of transmission packets stored in the predetermined data unit are such that the first transmission packet is stored in the predetermined data unit. If the number of the first transmission packets is counted as the first packet number, and the second transmission packet is stored in the predetermined data unit, the packet size of the second transmission packet is counted as the number of first transmission packets. The integer value obtained by adding 1 to the quotient obtained by dividing by the maximum packet size of one transmission packet is counted as the second packet number, and the first and second packet numbers obtained by counting The packets are stored in the predetermined data unit such that the sum total of the packets is equal to or less than the predetermined number of packets, and the maximum value of the packet size of the first transmission packet is 1500B.

In order to achieve the above object, a transmission method according to one aspect of the present invention is a transmission method that transmits a plurality of transmission packets, wherein the transmission packet is composed of a variable-length packet header and a variable-length payload. , a reception buffer model for receiving the plurality of transmission packets to be transmitted includes a first reception buffer model that receives the transmission packets and includes a variable-length packet header and a variable-length payload stored in the received transmission packets. The buffer includes a buffer that converts a packet into a second packet having a header-expanded fixed-length packet header and outputs the second packet obtained by the conversion at a predetermined extraction rate, and Storing the plurality of transmission packets in a predetermined data unit, and transmitting the predetermined data unit in each of a plurality of continuous unit transmission periods, which are periods of unit time that do not overlap with each other. The plurality of transmission packets, the number of which is equal to or less than the predetermined number of packets, are transmitted per unit time, and the plurality of transmission packets include a first transmission packet storing a real packet different from an invalid packet; and a second transmission packet storing an invalid packet, and when storing the first transmission packet in the predetermined data unit, the number of the first transmission packet is counted as the number of first packets. However, when storing the second transmission packet in the predetermined data unit, 1 is added to the quotient obtained by dividing the packet size of the second transmission packet by the maximum value of the packet size of the first transmission packet. The added integer value is counted as the second number of packets, and the plurality of transmission packets are transferred to The invalid packet is stored in a predetermined data unit and is a NULL packet.

Further, a receiving method according to one aspect of the present invention is a receiving method in a receiving device having a buffer, in which a plurality of transmission packets each including a variable-length packet header and a variable-length payload are received, A first packet consisting of a variable length packet header and a variable length payload stored in the plurality of transmitted packets is transferred to a second packet having a header-expanded fixed length packet header using the buffer. The second packet obtained by the conversion is outputted from the buffer at a predetermined extraction rate, and the buffer size of the buffer is equal to the transmission rate of the plurality of transmission packets, the average packet length of the transmission packets , and the plurality of transmission packets are in a unit time period that does not overlap with each other, and the unit transmission is larger than the maximum buffer occupancy calculated using the extraction rate, and the plurality of transmission packets are in a unit time period that does not overlap with each other, A first transmission packet that is stored and transmitted in a predetermined data unit in a period, and is a transmission packet that is less than or equal to the predetermined number of packets transmitted per unit time, and that stores real packets that are different from invalid packets. The plurality of transmission packets, which include a transmission packet and a second transmission packet storing the invalid packet, and which are stored in the predetermined data unit are such that the first transmission packet is stored in the predetermined data unit. If stored, the number of the first transmission packets is counted as the number of first packets, and if the second transmission packet is stored in the predetermined data unit, the packet size of the second transmission packet is counted as the number of first transmission packets. The integer value obtained by adding 1 to the quotient obtained by dividing by the maximum packet size of the first transmission packet is counted as the number of second packets, and the number of first packets and second packets obtained by counting The packets are stored in the predetermined data unit such that the sum of the numbers is equal to or less than the predetermined number of packets, and the invalid packet is a NULL packet.

In order to achieve the above object, a transmission method according to one aspect of the present invention transmits a plurality of transmission packets while satisfying regulations according to a predetermined reception buffer model in order to guarantee buffer operation of a reception device. In the transmission method, the transmission packet includes a variable-length packet header and a variable-length payload, and the reception buffer model receives the transmission packet and generates a variable-length data stored in the received transmission packet. A first packet consisting of a packet header and a variable length payload is converted into a second packet having a fixed length packet header with the header expanded, and the second packet obtained by the conversion is extracted in a predetermined manner. It includes a buffer that outputs at a rate, and transmits the plurality of transmission packets in a number not more than a predetermined number of packets per unit time.

Further, a receiving method according to one aspect of the present invention is a receiving method in a receiving device having a buffer, in which a plurality of transmission packets each including a variable-length packet header and a variable-length payload are received, A first packet consisting of a variable length packet header and a variable length payload stored in the plurality of transmitted packets is transferred to a second packet having a header-expanded fixed length packet header using the buffer. The second packet obtained by the conversion is outputted from the buffer at a predetermined extraction rate, and the buffer size of the buffer is equal to the transmission rate of the plurality of transmission packets, the average packet length of the transmission packets , and the maximum buffer occupancy determined using the extraction rate.

Note that these general or specific aspects may be realized by a system, a device, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM. and a recording medium may be used in any combination.

The present invention can guarantee buffer operation of a receiving device when transmitting data using a method such as MMT.

FIG. 1 is a diagram showing an example of dividing a picture into slice segments. FIG. 2 is a diagram showing an example of a PES packet string in which picture data is stored. FIG. 3 is a diagram illustrating an example of dividing a picture according to the first embodiment. FIG. 4 is a diagram illustrating an example of picture division according to a comparative example of the first embodiment. FIG. 5 is a diagram illustrating an example of data of an access unit according to the first embodiment. FIG. 6 is a block diagram of a transmitting device according to the first embodiment. FIG. 7 is a block diagram of the receiving device according to the first embodiment. FIG. 8 is a diagram illustrating an example of an MMT packet according to the first embodiment. FIG. 9 is a diagram showing another example of the MMT packet according to the first embodiment. FIG. 10 is a diagram illustrating an example of data input to each decoding unit according to the first embodiment. FIG. 11 is a diagram illustrating an example of an MMT packet and header information according to the first embodiment. FIG. 12 is a diagram illustrating another example of data input to each decoding unit according to the first embodiment. FIG. 13 is a diagram illustrating an example of dividing a picture according to the first embodiment. FIG. 14 is a flowchart of the transmission method according to the first embodiment. FIG. 15 is a block diagram of a receiving device according to the first embodiment. FIG. 16 is a flowchart of the receiving method according to the first embodiment. FIG. 17 is a diagram illustrating an example of an MMT packet and header information according to the first embodiment. FIG. 18 is a diagram illustrating an example of an MMT packet and header information according to the first embodiment. FIG. 19 is a diagram showing the configuration of the MPU. FIG. 20 is a diagram showing the structure of MF metadata. FIG. 21 is a diagram for explaining the data transmission order. FIG. 22 is a diagram illustrating an example of a method for decoding without using header information. FIG. 23 is a block diagram of a transmitting device according to the second embodiment. FIG. 24 is a flowchart of the transmission method according to the second embodiment. FIG. 25 is a block diagram of a receiving device according to Embodiment 2. FIG. 26 is a flowchart of the operation for specifying the MPU head position and NAL unit position. FIG. 27 is a flowchart of operations for obtaining initialization information based on the transmission order type and decoding media data based on the initialization information. FIG. 28 is a flowchart of the operation of the receiving device when the low delay presentation mode is provided. FIG. 29 is a diagram illustrating an example of the transmission order of MMT packets when auxiliary data is transmitted. FIG. 30 is a diagram for explaining an example in which the transmitting device generates auxiliary data based on the configuration of moof. FIG. 31 is a diagram for explaining reception of auxiliary data. FIG. 32 is a flowchart of a receiving operation using auxiliary data. FIG. 33 is a diagram showing the configuration of an MPU composed of a plurality of movie fragments. FIG. 34 is a diagram for explaining the transmission order of MMT packets when the MPU having the configuration shown in FIG. 33 is transmitted. FIG. 35 is a first diagram for explaining an example of the operation of the receiving device when one MPU is composed of a plurality of movie fragments. FIG. 36 is a second diagram for explaining an example of the operation of the receiving device when one MPU is composed of a plurality of movie fragments. FIG. 37 is a flowchart of the operation of the reception method described in FIGS. 35 and 36. FIG. 38 is a diagram showing a case where non-VCL NAL units are individually treated as data units and aggregated. FIG. 39 is a diagram showing a case in which non-VCL NAL units are combined into a data unit. FIG. 40 is a flowchart of the operation of the receiving device when packet loss occurs. FIG. 41 is a flowchart of the reception operation when the MPU is divided into a plurality of movie fragments. FIG. 42 is a diagram illustrating an example of a predicted structure of a picture in each TemporalId when achieving temporal scalability. FIG. 43 is a diagram showing the relationship between decoding time (DTS) and display time (PTS) in each picture in FIG. 42. FIG. 44 is a diagram illustrating an example of a predicted structure of a picture that requires picture delay processing and reorder processing. FIG. 45 is a diagram showing an example in which an MPU configured in MP4 format is divided into a plurality of movie fragments and stored in MMTP payloads and MMTP packets. FIG. 46 is a diagram for explaining the calculation method and problems of PTS and DTS. FIG. 47 is a flowchart of the reception operation when DTS is calculated using the information for DTS calculation. FIG. 48 is a diagram for explaining a method of storing a data unit in a payload in MMT. FIG. 49 is an operation flow of the transmitting device according to the third embodiment. FIG. 50 is an operation flow of the receiving device according to the third embodiment. FIG. 51 is a diagram illustrating an example of a specific configuration of a transmitting device according to Embodiment 3. FIG. 52 is a diagram illustrating an example of a specific configuration of a receiving device according to Embodiment 3. FIG. 53 is a diagram showing a method of storing non-timed media in the MPU and a method of transmitting it using MMTP packets. FIG. 54 is a diagram showing an example in which a plurality of pieces of divided data obtained by dividing a file are packetized and transmitted. FIG. 55 is a diagram showing another example in which a plurality of pieces of divided data obtained by dividing a file are packetized and transmitted. FIG. 56 is a diagram showing the syntax of a loop for each file in the asset management table. FIG. 57 is an operation flow for specifying a divided data number in a receiving device. FIG. 58 is an operation flow for specifying the number of divided data in the receiving device. FIG. 59 is an operational flow for determining whether to operate a fragment counter in a transmitting device. FIG. 60 is a diagram for explaining a method for specifying the number of divided data and the divided data number (when using a fragment counter). FIG. 61 is an operation flow of the transmitting device when utilizing a fragment counter. FIG. 62 is an operation flow of the receiving device when utilizing a fragment counter. FIG. 63 is a diagram showing a service configuration when the same program is transmitted using multiple IP data flows. FIG. 64 is a diagram showing an example of a specific configuration of a transmitting device. FIG. 65 is a diagram showing an example of a specific configuration of a receiving device. FIG. 66 is an operation flow of the transmitting device. FIG. 67 is an operation flow of the receiving device. FIG. 68 shows a receive buffer model based on the receive buffer model specified in ARIB STD B-60, particularly when only a broadcast transmission path is used. FIG. 69 is a diagram illustrating an example of aggregating a plurality of data units and storing them in one payload. FIG. 70 is a diagram showing an example in which a plurality of data units are aggregated and stored in one payload, and a video signal in the NAL size format is used as one data unit. FIG. 71 is a diagram showing the structure of the payload of an MMTP packet in which the data unit length is not indicated. FIG. 72 is an example of an extend area added to each packet. FIG. 73 shows an operation flow by the receiving device. FIG. 74 is a diagram illustrating an example of a specific configuration of a transmitting device. FIG. 75 is a diagram showing an example of a specific configuration of a receiving device. FIG. 76 is an operational flowchart of the transmitting device. FIG. 77 is an operation flow of the receiving device. FIG. 78 is a diagram showing a protocol stack of the MMT/TLV method defined in ARIB STD-B60. FIG. 79 is a diagram showing the structure of a TLV packet. FIG. 80 is a diagram illustrating an example of a block diagram of a receiving device. FIG. 81 is a diagram showing the configuration of a transmission slot. FIG. 82 is a diagram showing a transmission frame and one TLV stream (TLV packet sequence) having a specific tlv_stream_id stored in the transmission frame. FIG. 83 is a flowchart of a sending method when the buffer size of the TLV packet buffer and the maximum number of packets per transmission frame are defined. FIG. 84 is a diagram showing an example of a specific configuration of a transmitting device. FIG. 85 is a diagram showing an example of a specific configuration of a receiving device. FIG. 86 is an operation flow of the transmitting device. FIG. 87 is an operation flow of the receiving device.

A transmission method according to one aspect of the present invention is a transmission method for transmitting a plurality of transmission packets while satisfying regulations according to a predetermined reception buffer model in order to guarantee buffer operation of a reception device, the transmission method comprising: A packet is composed of a variable-length packet header and a variable-length payload, and the reception buffer model receives the transmission packet and processes the variable-length packet header and variable-length payload stored in the received transmission packet. a buffer that converts a first packet consisting of the following into a second packet having a header-expanded fixed-length packet header, and outputs the second packet obtained by the conversion at a predetermined extraction rate; The plurality of transmission packets, the number of which is equal to or less than a predetermined number of packets, are transmitted per unit time.

Such a transmitting device can guarantee buffer operation of the receiving device when transmitting data using a method such as MMT.

For example, further, the plurality of transmission packets having a number equal to or less than the predetermined number of packets are stored in a predetermined data unit, and in the transmission, each of the plurality of consecutive unit transmission periods is a period of the unit time that does not overlap with each other. In this case, by transmitting the predetermined data unit in the unit transmission period, the plurality of transmission packets equal to or less than the predetermined number of packets may be transmitted per unit time.

For example, the predetermined number of packets includes a plurality of packet numbers having different values set according to the transmission rate of the predetermined data unit; The plurality of transmission packets, the number of which is equal to or less than the corresponding number of packets, may be stored in the predetermined data unit.

For example, the plurality of transmission packets include a first transmission packet storing a real packet different from an invalid packet, and a second transmission packet storing the invalid packet, and in the storage, the When storing the first transmission packet in the predetermined data unit, the number of the first transmission packet is counted as the first packet number, and when storing the second transmission packet in the predetermined data unit, the second transmission packet is counted as the first packet number. The integer value obtained by dividing the packet size of the transmission packet by the maximum value of the packet size of the first transmission packet is counted as the second packet number, and the first packet number and the second packet number obtained by counting are counted. The plurality of transmission packets may be stored in the predetermined data unit so that the total number of packets is equal to or less than the predetermined number of packets.

For example, the invalid packet may be a NULL packet.

For example, the maximum value of the packet size of the first transmission packet may be 1500B.

For example, the predetermined data unit may be a transmission frame, and the unit transmission period may be a transmission frame length corresponding to the transmission frame.

For example, the predetermined number of packets may be a value set so that the transmission packets of the minimum packet size are not consecutive.

A receiving method according to one aspect of the present invention is a receiving method in a receiving device having a buffer, in which a plurality of transmission packets each including a variable-length packet header and a variable-length payload are received, and the received converting a first packet consisting of a variable-length packet header and a variable-length payload stored in the plurality of transmission packets into a second packet having a header-expanded fixed-length packet header using the buffer; and outputs the second packet obtained by the conversion from the buffer at a predetermined extraction rate, and the buffer size of the buffer is determined by the transmission rate of the plurality of transmission packets, the average packet length of the transmission packets, and This is larger than the maximum buffer occupancy determined using the extraction rate.

Note that these comprehensive or specific aspects may be realized by a system, a device, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, and the system, device, integrated circuit, It may be realized by any combination of computer programs or recording media.

Hereinafter, embodiments will be specifically described with reference to the drawings.

Note that the embodiments described below are all inclusive or specific examples. The numerical values, shapes, materials, components, arrangement positions and connection forms of the components, steps, order of steps, etc. shown in the following embodiments are merely examples, and do not limit the present invention. Further, among the constituent elements in the following embodiments, constituent elements that are not described in the independent claims indicating the most significant concept will be described as arbitrary constituent elements.

(Findings that formed the basis of the present invention)
In recent years, the resolution of displays such as TVs, smartphones, and tablet terminals has been increasing. In particular, 8K4K (8K x 4K resolution) services are planned for broadcasting in Japan in 2020. Since it is difficult to decode ultra-high resolution moving images such as 8K4K in real time with a single decoder, methods of performing decoding processing in parallel using multiple decoders are being considered.

Since encoded data is multiplexed and transmitted based on a multiplexing method such as MPEG-2 TS or MMT, the receiving device needs to separate the video encoded data from the multiplexed data before decoding. . Hereinafter, the process of separating encoded data from multiplexed data will be referred to as demultiplexing.

When parallelizing decoding processing, it is necessary to distribute encoded data to be decoded to each decoder. When distributing encoded data, it is necessary to analyze the encoded data itself, and since the bit rate is particularly high in content such as 8K, the processing load associated with analysis is large. Therefore, there was a problem that the demultiplexing part became a bottleneck and reproduction could not be performed in real time.

By the way, H. 264 and H. In a video encoding system such as H.265, a transmitting device can divide a picture into a plurality of regions called slices or slice segments, and can encode each divided region so that it can be independently decoded. Thus, for example, H. In the case of H.265, a receiving device that receives broadcasting can realize parallel decoding processing by separating the data for each slice segment from the received data and outputting the data for each slice segment to separate decoders. .

FIG. 1 is a diagram showing an example of dividing one picture into four slice segments in HEVC. For example, the receiving device includes four decoders, each decoder decoding one of the four slice segments.

In conventional broadcasting, a transmitting device stores one picture (an access unit in the MPEG system standard) in one PES packet, and multiplexes the PES packet into a TS packet string. Therefore, the receiving device separates the payload of the PES packet, analyzes the access unit data stored in the payload, separates each slice segment, and decodes the data of each separated slice segment. It was necessary to output to .

However, the inventors have found that there is a problem in that it is difficult to perform this processing in real time because the amount of processing required to analyze access unit data and separate slice segments is large.

FIG. 2 is a diagram illustrating an example in which data of a picture divided into slice segments is stored in the payload of a PES packet.

As shown in FIG. 2, for example, data of a plurality of slice segments (slice segments 1 to 4) are stored in the payload of one PES packet. Furthermore, the PES packets are multiplexed into a TS packet sequence.

(Embodiment 1)
In the following, H. The following will explain the case using H.265 as an example. This embodiment can also be applied when other encoding methods such as H.264 are used.

FIG. 3 is a diagram showing an example in which the access unit (picture) in this embodiment is divided into division units. The access unit is H. A function called tile introduced by H.265 divides the screen into two equal parts horizontally and vertically, for a total of four tiles. Furthermore, slice segments and tiles have a one-to-one correspondence.

The reason why it is divided into two in the horizontal and vertical directions will be explained. First, during decoding, a line memory that stores one horizontal line of data is generally required, but when it comes to ultra-high resolution such as 8K4K, the size of the line memory increases because the horizontal size increases. In implementing a receiving device, it is desirable to be able to reduce the size of line memory. Vertical partitioning is required to reduce the size of the line memory. Vertical partitioning requires data structures called tiles. For these reasons, tiles are used.

On the other hand, since images generally have a high correlation in the horizontal direction, encoding efficiency is improved if a wider range can be referenced in the horizontal direction. Therefore, from the viewpoint of coding efficiency, it is desirable that the access unit be divided horizontally.

By dividing the access unit into two in the horizontal and vertical directions, these two characteristics can be achieved simultaneously, and both aspects of implementation and encoding efficiency can be considered. If a single decoder is capable of decoding 4K2K video images in real time, the 8K4K image is divided into four equal slices, and each slice segment is divided into 4K2K segments. can decode 8K4K images in real time.

Next, the reason why the tiles obtained by dividing the access unit in the horizontal and vertical directions and the slice segments are associated one-to-one will be explained. H. In H.265, an access unit is composed of a plurality of units called NAL (Network Adaptation Layer) units.

The payload of the NAL unit includes an access unit delimiter that indicates the start position of the access unit, an SPS (Sequence Parameter Set) that is initialization information at the time of decoding that is commonly used in sequence units, and an initialization information at the time of decoding that is commonly used within the picture. PPS (Picture Parameter Set), which is encoding information, SEI (Supplemental Enhancement Information), which is not necessary for the decoding process itself but is necessary for processing and displaying the decoding results, and encoded data of slice segments, etc. Store. The header of a NAL unit includes type information to identify the data stored in the payload.

Here, the transmitting device transmits the encoded data to MPEG-2 TS, MMT (MPEG Media Transport), MPEG DASH (Dynamic Adaptive
When multiplexing using a multiplexing format such as Streaming over HTTP (Streaming over HTTP) or RTP (Real-time Transport Protocol), the basic unit can be set to a NAL unit. In order to store one slice segment in one NAL unit, when dividing the access unit into areas, it is desirable to divide the access unit into slice segments. For this reason, the transmitting device makes a one-to-one correspondence between tiles and slice segments.

Note that, as shown in FIG. 4, the transmitting device can also set tiles 1 to 4 as one slice segment. However, in this case, all tiles are stored in one NAL unit, making it difficult for the receiving device to separate the tiles in the multiplexing layer.

Note that the slice segments include independent slice segments that can be decoded independently and reference slice segments that refer to the independent slice segments. Here, a case where independent slice segments are used will be described.

FIG. 5 is a diagram illustrating an example of data of access units divided so that the boundaries between tiles and slice segments coincide as shown in FIG. 3. The access unit data consists of the NAL unit in which the access unit delimiter placed at the beginning is stored, the SPS, PPS, and SEI NAL units placed after that, and the tiles 1 to 4 placed after that. and the data of the slice segment in which the data is stored. Note that the access unit data does not need to include some or all of the SPS, PPS, and SEI NAL units.

Next, the configuration of transmitting device 100 according to this embodiment will be described. FIG. 6 is a block diagram showing a configuration example of transmitting device 100 according to this embodiment. This transmitting device 100 includes an encoding section 101, a multiplexing section 102, a modulating section 103, and a transmitting section 104.

The encoding unit 101 converts the input image into, for example, H. Encoded data is generated by encoding according to H.265. Further, the encoding unit 101 divides the access unit into four slice segments (tiles) and encodes each slice segment, for example, as shown in FIG. 3.

Multiplexing section 102 multiplexes the encoded data generated by encoding section 101. Modulating section 103 modulates data obtained by multiplexing. The transmitter 104 transmits the modulated data as a broadcast signal.

Next, the configuration of receiving device 200 according to this embodiment will be explained. FIG. 7 is a block diagram showing a configuration example of receiving device 200 according to this embodiment. This receiving device 200 includes a tuner 201, a demodulating section 202, a demultiplexing section 203, a plurality of decoding sections 204A to 204D, and a display section 205.

Tuner 201 receives broadcast signals. Demodulation section 202 demodulates the received broadcast signal. The demodulated data is input to demultiplexer 203.

Demultiplexing section 203 separates the demodulated data into division units, and outputs the data for each division unit to decoding sections 204A to 204D. Here, the division unit is a division area obtained by dividing an access unit, and is, for example, a division area obtained by dividing an access unit. This is a slice segment in H.265. Further, here, an 8K4K image is divided into four 4K2K images. Therefore, there are four decoding units 204A to 204D.

The plurality of decoders 204A to 204D operate in synchronization with each other based on a predetermined reference clock. Each decoding section decodes the encoded data in units of division according to the DTS (Decoding Time Stamp) of the access unit, and outputs the decoding result to the display section 205.

The display unit 205 generates an 8K4K output image by integrating the multiple decoding results output from the multiple decoding units 204A to 204D. The display unit 205 displays the generated output image according to the PTS (Presentation Time Stamp) of the access unit that is obtained separately. Note that when integrating the decoding results, the display unit 205 performs filter processing such as a deblock filter on the boundary areas of mutually adjacent division units, such as the boundaries of tiles, so that the boundaries become visually inconspicuous. Good too.

Note that although the above description has been made using the transmitting device 100 and the receiving device 200 that transmit or receive broadcasts as an example, content may be transmitted and received via a communication network. When the receiving device 200 receives content via a communication network, the receiving device 200 separates multiplexed data from IP packets received via a network such as Ethernet.

In broadcasting, the transmission path delay from when a broadcast signal is transmitted until it reaches receiving device 200 is constant. On the other hand, in a communication network such as the Internet, the transmission path delay until data transmitted from the server reaches the receiving device 200 is not constant due to the influence of congestion. Therefore, the receiving device 200 often does not perform strictly synchronized reproduction based on a reference clock such as PCR in MPEG-2 TS for broadcasting. Therefore, the receiving device 200 may display the 8K4K output image on the display unit according to the PTS without strictly synchronizing each decoding unit.

Furthermore, due to communication network congestion, decoding processing for all division units may not be completed at the time indicated by the PTS of the access unit. In this case, the receiving device 200 skips the display of the access unit or delays the display until the decoding of at least four division units is completed and the generation of the 8K4K image is completed.

Note that content may be transmitted and received using both broadcasting and communication. Further, this method can also be applied when reproducing multiplexed data stored in a recording medium such as a hard disk or a memory.

Next, a method for multiplexing access units divided into slice segments when MMT is used as the multiplexing method will be described.

FIG. 8 is a diagram illustrating an example of packetizing HEVC access unit data into MMT. Although SPS, PPS, SEI, etc. do not necessarily need to be included in the access unit, a case where they are present will be exemplified here.

NAL units arranged before the first slice segment in the access unit, such as the access unit delimiter, SPS, PPS, and SEI, are collectively stored in MMT packet #1. Subsequent slice segments are stored in separate MMT packets for each slice segment.

Note that, as shown in FIG. 9, a NAL unit placed before the first slice segment in the access unit may be stored in the same MMT packet as the first slice segment.

Additionally, if a NAL unit such as End-of-Sequence or End-of-Bitstream indicating the end of a sequence or stream is added after the final slice segment, these units are included in the same MMT packet as the final slice segment. Stored. However, since NAL units such as End-of-Sequence or End-of-Bitstream are inserted at the end point of decoding processing or the connection point of two streams, the receiving device 200 , may be desirable to be readily available at the multiplexing layer. In this case, these NAL units may be stored in a separate MMT packet from the slice segment. Thereby, receiving device 200 can easily separate these NAL units in the multiplexing layer.

Note that TS, DASH, RTP, or the like may be used as the multiplexing method. In these methods as well, the transmitting device 100 stores different slice segments in different packets. This ensures that the receiving device 200 can separate slice segments in the multiplex layer.

For example, when TS is used, encoded data is converted into PES packets in units of slice segments. When RTP is used, encoded data is converted into RTP packets in units of slice segments. Even in these cases, the NAL unit placed before the slice segment and the slice segment may be packetized separately, as in MMT packet #1 shown in FIG. 8.

When a TS is used, the transmitting device 100 indicates the unit of data stored in the PES packet, such as by using a data alignment descriptor. Furthermore, since DASH is a method of downloading MP4 format data units called segments using HTTP or the like, the transmitting device 100 does not packetize encoded data during transmission. Therefore, the transmitting device 100 creates sub-samples for each slice segment and stores information indicating the storage location of the sub-samples in the MP4 header so that the receiving device 200 can detect the slice segments in the multiplex layer in MP4. You may.

MMT packetization of slice segments will be described in detail below.

As shown in FIG. 8, by packetizing encoded data, data that is commonly referred to when decoding all slice segments in an access unit such as SPS and PPS is stored in MMT packet #1. In this case, receiving device 200 concatenates the payload data of MMT packet #1 and the data of each slice segment, and outputs the obtained data to the decoding unit. In this way, the receiving device 200 can easily generate input data to the decoding unit by concatenating payloads of multiple MMT packets.

FIG. 10 is a diagram showing an example in which input data to the decoding units 204A to 204D is generated from the MMT packet shown in FIG. 8. The demultiplexing unit 203 generates data necessary for the decoding unit 204A to decode slice segment 1 by concatenating the payload data of MMT packet #1 and MMT packet #2. Demultiplexing section 203 similarly generates input data for decoding sections 204B to 204D. That is, the demultiplexer 203 generates input data for the decoder 204B by concatenating the payload data of MMT packet #1 and MMT packet #3. The demultiplexing unit 203 generates input data for the decoding unit 204C by concatenating the payload data of MMT packet #1 and MMT packet #4. Demultiplexing section 203 generates input data for decoding section 204D by concatenating payload data of MMT packet #1 and MMT packet #5.

Note that the demultiplexing unit 203 removes NAL units that are not necessary for decoding processing, such as the access unit delimiter and SEI, from the payload data of MMT packet #1, and extracts only the SPS and PPS NAL units that are necessary for decoding processing. may be separated and added to the slice segment data.

When the encoded data is packetized as shown in FIG. 9, the demultiplexing section 203 outputs MMT packet #1 containing the first data of the access unit in the multiplexing layer to the first decoding section 204A. . In addition, the demultiplexer 203 analyzes the MMT packet including the first data of the access unit in the multiplexing layer, separates the SPS and PPS NAL units, and slices the separated SPS and PPS NAL units into the second and subsequent slices. By adding it to each piece of data in the segment, input data for each of the second and subsequent decoding units is generated.

Further, the receiving device 200 uses the information included in the header of the MMT packet to determine the type of data stored in the MMT payload and, if a slice segment is stored in the payload, the slice segment in the access unit. It is desirable to be able to identify the index number. Here, the data type refers to pre-slice segment data (NAL units placed before the first slice segment in an access unit are collectively called this) and slice segment data. Either way. When storing a unit in which an MPU is fragmented, such as a slice segment, in an MMT packet, a mode for storing an MFU (Media Fragment Unit) is used. When using this mode, the transmitting device 100 converts the Data Unit, which is the basic unit of data in the MFU, into a sample (which is a data unit in MMT and corresponds to an access unit) or a subsample (which is a data unit in MMT and corresponds to an access unit). (divided units).

At this time, the header of the MMT packet includes a field called a Fragmentation indicator and a field called a Fragment counter.

The Fragmentation indicator indicates whether the data stored in the payload of the MMT packet is a fragmented Data unit, and if so, whether the fragment is the first or last fragment in the Data unit, or , indicates whether the fragment is neither the beginning nor the end. In other words, the Fragmentation indicator included in the header of a certain packet indicates that (1) only the packet is included in the Data unit, which is a basic data unit, (2) the Data unit is stored divided into multiple packets, and The packet is the first packet of the Data unit, (3) the Data unit is stored divided into multiple packets, and the packet is a packet other than the first and last packet of the Data unit, and (4) This is identification information indicating whether a Data unit is stored divided into a plurality of packets and whether the packet is the last packet of the Data unit.

The Fragment counter is an index number indicating which fragment in the Data unit the data stored in the MMT packet corresponds to.

Therefore, by the transmitting device 100 setting samples in MMT in the Data unit, and setting the pre-slice segment data and each slice segment in fragment units of the Data unit, the receiving device 200 can read the header of the MMT packet. The information contained in the payload can be used to identify the type of data stored in the payload. In other words, the demultiplexing section 203 can generate input data to each of the decoding sections 204A to 204D by referring to the header of the MMT packet.

FIG. 11 is a diagram illustrating an example in which a sample is set in a Data unit, and pre-slice segment data and slice segments are packetized as a fragment of the Data unit.

The pre-slice segment data and the slice segment are divided into five fragments from fragment #1 to fragment #5. Each fragment is stored in a separate MMT packet. At this time, the values of the Fragmentation indicator and Fragment counter included in the header of the MMT packet are as shown in the figure.

For example, the Fragment indicator is a 2-bit binary value. The Fragment indicator of MMT packet #1 which is the beginning of the Data unit, the Fragment indicator of MMT packet #5 which is the last one, and the Fragment indicators of MMT packet #2 to MMT packet #4 which are the packets between them are each different. set to the value. Specifically, the Fragment indicator of MMT packet #1, which is the beginning of the Data unit, is set to 01, the Fragment indicator of MMT packet #5, which is the last, is set to 11, and the Fragment indicator of MMT packet #1, which is the beginning of the data unit, is set to 11. Fragment indicators up to MMT packet #4 are set to 10. Note that when the Data unit includes only one MMT packet, the Fragment indicator is set to 00.

In addition, the Fragment counter is 4, which is the value obtained by subtracting 1 from the total number of fragments, 5, in MMT packet #1, and decreases by 1 in subsequent packets, and is 0 in the final MMT packet #5. be.

Therefore, the receiving device 200 can identify the MMT packet storing the pre-slice segment data using either the Fragment indicator or the Fragment counter. Further, the receiving device 200 can identify the MMT packet storing the Nth slice segment by referring to the Fragment counter.

The header of the MMT packet separately includes a sequence number within the MPU of the Movie Fragment to which the Data unit belongs, a sequence number of the MPU itself, and a sequence number within the Movie Fragment of the sample to which the Data unit belongs. By referring to these, the demultiplexer 203 can uniquely determine the sample to which the Data unit belongs.

Further, since the demultiplexer 203 can determine the index number of a fragment in the data unit from the Fragment counter, it can uniquely identify the slice segment stored in the fragment even if a packet loss occurs. For example, even if fragment #4 shown in FIG. 11 cannot be acquired due to packet loss, the demultiplexer 203 knows that the fragment received next to fragment #3 is fragment #5, so It is possible to correctly output slice segment 4 stored in the decoding section 204D to the decoding section 204D instead of the decoding section 204C.

Note that when a transmission path that is guaranteed not to cause packet loss is used, the demultiplexer 203 refers to the header of the MMT packet to determine the type of data stored in the MMT packet or the slice segment. Arrived packets may be processed periodically without determining the index number of the packet. For example, when an access unit is transmitted using a total of five MMT packets including pre-slice data and four slice segments, the receiving device 200 determines the pre-slice data of the access unit to start decoding. , by sequentially processing the received MMT packets, the pre-slice data and the data of the four slice segments can be sequentially acquired.

Modified examples of packetization will be described below.

The slice segment does not necessarily have to be one in which the plane of the access unit is divided both horizontally and vertically, and may be one in which the access unit is divided only in the horizontal direction, as shown in FIG. However, it may be divided only in the vertical direction.

Furthermore, if the access unit is divided only in the horizontal direction, there is no need to use tiles.

Furthermore, the number of in-plane divisions in the access unit is arbitrary and is not limited to four. However, the area size of slice segments and tiles is H. The value must be at least the lower limit of the encoding standard such as H.265.

The transmitting device 100 may store identification information indicating an in-plane division method in an access unit in an MMT message, a TS descriptor, or the like. For example, information indicating the number of divisions in the horizontal direction and vertical direction within the plane may be stored. Or identification information unique to the division method, such as divided into two equal parts horizontally and vertically as shown in Figure 3, or divided into four equal parts horizontally as shown in Figure 1. may be assigned. For example, when the access unit is divided as shown in FIG. 3, the identification information indicates mode 1, and when the access unit is divided as shown in FIG. 1, the identification information indicates mode 1. .

Further, information indicating constraints on encoding conditions related to the intra-plane division method may be included in the multiplexing layer. For example, information indicating that one slice segment is composed of one tile may be used. Or, reference blocks when performing motion compensation when decoding slice segments or tiles are limited to slice segments or tiles at the same position in the screen, or to blocks within a predetermined range in adjacent slice segments. Information indicating such things may also be used.

Further, the transmitting device 100 may switch whether to divide the access unit into a plurality of slice segments depending on the resolution of the moving image. For example, if the moving image to be processed has a resolution of 4K2K, the transmitting device 100 may not perform in-plane division, but if the moving image to be processed is 8K4K, it may divide the access unit into four. good. By predefining the division method for 8K4K video images, the receiving device 200 determines whether or not to divide within the screen and the division method by acquiring the resolution of the received video image, and performs decoding. You can switch the operation.

Furthermore, the receiving device 200 can detect whether or not there is division within a plane by referring to the header of the MMT packet. For example, if the access unit is not divided and the MMT Data unit is set to sample, the Data unit will not be fragmented. Therefore, the receiving device 200 can determine that the access unit is not divided if the value of the Fragment counter included in the header of the MMT packet is always zero. Alternatively, the receiving device 200 may detect whether the value of the Fragmentation indicator is always 01. The receiving device 200 can determine that the access unit is not divided even if the value of the Fragmentation indicator is always 01.

Furthermore, the receiving device 200 can also handle a case where the number of in-plane divisions in an access unit and the number of decoding units do not match. For example, when receiving device 200 includes two decoding sections 204A and 204B that can decode 8K2K encoded data in real time, demultiplexing section 203 sends 8K4K encoded data to decoding section 204A. Outputs two of the four slice segments that make up the segment.

FIG. 12 is a diagram illustrating an example of the operation when the MMT packetized data as shown in FIG. 8 is input to two decoding units 204A and 204B. Here, it is desirable that the receiving device 200 be able to integrate and output the decoding results in the decoding sections 204A and 204B as they are. Therefore, demultiplexing section 203 selects slice segments to be output to each of decoding sections 204A and 204B so that the decoding results of each of decoding sections 204A and 204B are spatially continuous.

Further, the demultiplexing unit 203 may select a decoding unit to be used depending on the resolution or frame rate of the encoded data of the moving image. For example, in the case where the receiving device 200 includes four 4K2K decoding units, if the resolution of the input image is 8K4K, the receiving device 200 performs the decoding process using all four decoding units. Furthermore, if the resolution of the input image is 4K2K, the receiving device 200 performs the decoding process using only one decoding unit. Alternatively, even if the frame is divided into four, if 8K4K can be decoded in real time by a single decoding unit, the demultiplexing unit 203 integrates all division units into one decoding unit. Output to.

Furthermore, receiving device 200 may decide which decoder to use in consideration of the frame rate. For example, if the receiving device 200 is equipped with two decoders whose upper limit of the frame rate that can be decoded in real time is 60 fps when the resolution is 8K4K, there is a case where encoded data of 120 fps at 8K4K is input. be. At this time, assuming that the in-plane is composed of four division units, slice segment 1 and slice segment 2 are input to the decoding unit 204A, and slice segment 3 and slice segment 4 are input to the decoding unit 204A, as in the example of FIG. The data is input to the decoding unit 204B. Since each of the decoding units 204A and 204B can decode in real time up to 120 fps for 8K2K (resolution is half of 8K4K), decoding processing is performed by these two decoding units 204A and 204B.

Furthermore, even if the resolution and frame rate are the same, the profile or level of the encoding method, or the H. 264 or H. If the encoding method itself is different, such as H.265, the processing amount will be different. Therefore, receiving device 200 may select a decoding unit to use based on this information. Note that if the receiving device 200 cannot decode all the encoded data received through broadcasting or communication, or if it cannot decode all the slice segments or tiles that make up the area selected by the user, the receiving device 200 may Slice segments or tiles that can be decoded within the processing range of the decoder may be automatically determined. Alternatively, the receiving device 200 may provide a user interface for the user to select an area to decode. At this time, the receiving device 200 may display a warning message indicating that all regions cannot be decoded, or may display information indicating the number of decodable regions, slice segments, or tiles.

Furthermore, the above method can also be applied when MMT packets storing slice segments of the same encoded data are transmitted and received using multiple transmission paths such as broadcasting and communication.

Further, the transmitting device 100 may perform encoding so that the regions of each slice segment overlap in order to make the boundaries between division units less noticeable. In the example shown in FIG. 13, an 8K4K picture is divided into four slice segments 1 to 4. Each of slice segments 1 to 3 is, for example, 8K×1.1K, and slice segment 4 is 8K×1K. Also, adjacent slice segments overlap each other. By doing so, motion compensation during encoding can be efficiently executed at the boundaries between the four divisions shown by the dotted lines, so that the image quality at the boundaries is improved. In this way, image quality deterioration at the boundary portion is reduced.

In this case, the display unit 205 cuts out an 8K×1K area from the 8K×1.1K area and integrates the obtained areas. Note that the transmitting device 100 includes information indicating whether the slice segments are encoded in an overlapping manner and the range of overlap in the multiplex layer or encoded data, and separately transmits the information. Good too.

Note that a similar method can be applied even when tiles are used.

The flow of the operation of the transmitting device 100 will be described below. FIG. 14 is a flowchart illustrating an example of the operation of the transmitting device 100.

First, the encoding unit 101 divides a picture (access unit) into a plurality of slice segments (tiles) that are a plurality of regions (S101). Next, the encoding unit 101 generates encoded data corresponding to each of the plurality of slice segments by encoding each of the plurality of slice segments so that each of the plurality of slice segments can be decoded independently (S102). Note that the encoding unit 101 may encode a plurality of slice segments using a single encoding unit, or may perform parallel processing using a plurality of encoding units.

Next, the multiplexing unit 102 multiplexes the plurality of encoded data by storing the plurality of encoded data generated by the encoding unit 101 in a plurality of MMT packets (S103). Specifically, as shown in FIGS. 8 and 9, the multiplexing unit 102 combines a plurality of encoded data into a plurality of encoded data so that encoded data corresponding to different slice segments is not stored in one MMT packet. Store in MMT packet. Furthermore, as shown in FIG. 8, the multiplexing unit 102 transfers control information commonly used for all decoding units in a picture to multiple MMT packets #2 to ## in which multiple encoded data are stored. 5 is stored in MMT packet #1 different from MMT packet #5. Here, the control information includes at least one of an access unit delimiter, SPS, PPS, and SEI.

Note that the multiplexing unit 102 may store the control information in the same MMT packet as any of the multiple MMT packets in which multiple pieces of encoded data are stored. For example, as shown in FIG. 9, the multiplexing unit 102 stores the control information in the first MMT packet (MMT packet #1 in FIG. 9) among the multiple MMT packets in which multiple pieces of encoded data are stored. You may.

Finally, transmitting device 100 transmits a plurality of MMT packets. Specifically, modulating section 103 modulates the data obtained by multiplexing, and transmitting section 104 transmits the modulated data (S104).

FIG. 15 is a block diagram showing an example of the configuration of the receiving device 200, and is a diagram showing in detail the configuration of the demultiplexer 203 shown in FIG. 7 and the subsequent stage. As shown in FIG. 15, receiving device 200 further includes a decoding command section 206. Further, the demultiplexing section 203 includes a type determining section 211 , a control information acquiring section 212 , a slice information acquiring section 213 , and a decoded data generating section 214 .

The flow of the operation of the receiving device 200 will be described below. FIG. 16 is a flowchart illustrating an example of the operation of the receiving device 200. Here, the operation for one access unit is shown. When decoding processing for multiple access units is executed, the processing in this flowchart is repeated.

First, the receiving device 200 receives, for example, a plurality of packets (MMT packets) generated by the transmitting device 100 (S201).

Next, the type determination unit 211 acquires the type of encoded data stored in the received packet by analyzing the header of the received packet (S202).

Next, the type determination unit 211 determines whether the data stored in the received packet is pre-slice segment data or slice segment data based on the type of the acquired encoded data (S203). .

If the data stored in the received packet is pre-slice segment data (Yes in S203), the control information acquisition unit 212 obtains the pre-slice segment data of the access unit to be processed from the payload of the received packet, and The pre-segment data is stored in memory (S204).

On the other hand, if the data stored in the received packet is slice segment data (No in S203), the receiving device 200 uses the header information of the received packet to determine whether the data stored in the received packet is multiple segment data. It is determined which region of the regions the encoded data belongs to. Specifically, the slice information acquisition unit 213 acquires the index number Idx of the slice segment stored in the received packet by analyzing the header of the received packet (S205). Specifically, the index number Idx is an index number within the Movie Fragment of the access unit (sample in MMT).

Note that the processing in step S205 may be performed all at once in step S202.

Next, the decoded data generation unit 214 determines a decoding unit that decodes the slice segment (S206). Specifically, the index number Idx and a plurality of decoding units are associated in advance, and the decoded data generation unit 214 causes the decoding unit corresponding to the index number Idx acquired in step S205 to decode the slice segment. Decide which decoder to use.

Note that, as explained in the example of FIG. The decoding unit that decodes the slice segment may be determined based on at least one of the processing capabilities of the decoding unit. For example, the decoded data generation unit 214 determines the access unit division method based on identification information in a descriptor such as an MMT message or a TS section.

Next, the decoded data generation unit 214 generates control information commonly used for all decoding units in a picture, which is included in any of the plurality of packets, and each of the plurality of encoded data of the plurality of slice segments. By combining these, a plurality of input data (combined data) to be input to a plurality of decoders is generated. Specifically, the decoded data generation unit 214 obtains slice segment data from the payload of the received packet. The decoded data generation unit 214 generates the input data to the decoding unit determined in step S206 by combining the pre-slice segment data stored in the memory in step S204 and the acquired slice segment data. (S207).

After step S204 or S207, if the data of the received packet is not the final data of the access unit (No in S208), the processes from step S201 onwards are performed again. That is, the above process is repeated until input data to the plurality of decoding units 204A to 204D corresponding to all slice segments included in the access unit is generated.

Note that the timing at which the packets are received is not limited to the timing shown in FIG. 16, and a plurality of packets may be received in advance or sequentially and stored in a memory or the like.

On the other hand, if the data of the received packet is the final data of the access unit (Yes in S208), the decoding instruction unit 206 outputs the plurality of input data generated in step S207 to the corresponding decoding units 204A to 204D. (S209).

Next, the plurality of decoding units 204A to 204D generate a plurality of decoded images by decoding the plurality of input data in parallel according to the DTS of the access unit (S210).

Finally, the display unit 205 generates a display image by combining the plurality of decoded images generated by the plurality of decoding units 204A to 204D, and displays the display image according to the PTS of the access unit (S211).

Note that the receiving device 200 obtains the DTS and PTS of the access unit by analyzing payload data of an MMT packet that stores MPU header information or Movie Fragment header information. Furthermore, when TS is used as the multiplexing method, the receiving device 200 acquires the DTS and PTS of the access unit from the header of the PES packet. If RTP is used as the multiplexing method, the receiving device 200 obtains the DTS and PTS of the access unit from the header of the RTP packet.

Further, when integrating the decoding results of the plurality of decoding units, the display unit 205 may perform filter processing such as a deblocking filter on the boundary between adjacent division units. Note that filtering is not necessary when displaying the decoding results of a single decoding unit, so the display unit 205 performs processing depending on whether or not to perform filtering on the boundaries of the decoding results of multiple decoding units. You may switch. Whether filter processing is necessary may be determined in advance depending on whether or not there is division. Alternatively, information indicating whether filter processing is necessary may be stored separately in the multiplexing layer. Additionally, information necessary for filter processing, such as filter coefficients, may be stored in the SPS, PPS, SEI, or slice segment. The decoding units 204A to 204D or the demultiplexing unit 203 acquire this information by analyzing the SEI, and output the acquired information to the display unit 205. The display unit 205 performs filter processing using this information. Note that when these pieces of information are stored in slice segments, it is desirable that the decoding units 204A to 204D acquire these pieces of information.

Note that in the above description, an example was given in which there are two types of data stored in a fragment: pre-slice segment data and slice segment data, but there may be three or more types of data. In this case, the cases are divided according to the type in step S203.

Furthermore, when the data size of the slice segment is large, the transmitting device 100 may fragment the slice segment and store it in an MMT packet. That is, the transmitting device 100 may fragment the pre-slice segment data and the slice segment. In this case, if the access unit and data unit are set equally as in the packetization example shown in FIG. 11, the following problem will occur.

For example, when slice segment 1 is divided into three fragments, slice segment 1 is divided into three packets with Fragment counter values of 1 to 3 and transmitted. In addition, after the slice segment 2, the Fragment counter value becomes 4 or more, and the Fragment counter value cannot be associated with the data stored in the payload. Therefore, the receiving device 200 cannot identify the packet that stores the first data of the slice segment from the header information of the MMT packet.

In such a case, the receiving device 200 may identify the start position of the slice segment by analyzing the payload data of the MMT packet. Here, H. 264 or H. In H.265, the format for storing NAL units in the multiplex layer is a format called byte stream format in which a start code consisting of a specific bit string is added immediately before the NAL unit header, and a field indicating the size of the NAL unit is added. There are two types: a format called a NAL size format.

The byte stream format is used in MPEG-2 systems, RTP, etc. The NAL size format is used in MP4, DASH and MMT, etc. that use MP4.

When the byte stream format is used, the receiving device 200 analyzes whether the leading data of the packet matches the start code. If the first data of the packet matches the start code, the receiving device 200 determines whether the data included in the packet is slice segment data by acquiring the NAL unit type from the NAL unit header that follows. can detect whether

On the other hand, in the case of the NAL size format, the receiving device 200 cannot detect the start position of the NAL unit based on the bit string. Therefore, in order to obtain the starting position of the NAL unit, the receiving device 200 needs to shift the pointer by reading data by the size of the NAL unit in order from the first NAL unit of the access unit.

However, in the header of the MPU or Movie Fragment in MMT, the size of the subsample unit is indicated, and if the subsample corresponds to pre-slice data or a slice segment, the receiving device 200 uses the subsample size information based on the size information of the subsample. The starting position of each NAL unit can be specified. Therefore, the transmitting device 100 may include information indicating whether subsample unit information exists in the MPU or Movie Fragment in the information that the receiving device 200 acquires when starting data reception, such as MPT in MMT. .

Note that the MPU data is based on and expanded from the MP4 format. In MP4, H. 264 or H. There are two modes: a mode in which parameter sets such as H.265 SPS and PPS can be stored as sample data, and a mode in which they cannot be stored. Further, information for specifying this mode is shown as an entry name of SampleEntry. If the storable mode is used and the parameter set is included in the sample, the receiving device 200 obtains the parameter set using the method described above.

On the other hand, when the non-storable mode is used, the parameter set is stored as Decoder Specific Information in SampleEntry or using a stream for parameter sets. Here, since streams for parameter sets are not generally used, it is desirable for the transmitting device 100 to store the parameter sets in Decoder Specific Information. In this case, the receiving device 200 analyzes the SampleEntry transmitted as MPU metadata or Movie Fragment metadata in the MMT packet, and obtains the parameter set referenced by the access unit.

When the parameter set is stored as sample data, the receiving device 200 can obtain the parameter set necessary for decoding by referring only to the sample data without referring to SampleEntry. At this time, the transmitting device 100 does not need to store the parameter set in SampleEntry. By doing so, the transmitting device 100 can use the same SampleEntry in different MPUs, thereby reducing the processing load on the transmitting device 100 when generating the MPU. Furthermore, there is an advantage that the receiving device 200 does not need to refer to the parameter set in SampleEntry.

Alternatively, the transmitting device 100 may store one default parameter set in SampleEntry and store the parameter set referred to by the access unit in sample data. In conventional MP4, it is common to store parameter sets in SampleEntry, so if a parameter set does not exist in SampleEntry, there is a possibility that there is a receiving device that stops playback. This problem can be solved by using the above method.

Alternatively, the transmitting device 100 may store a parameter set in the sample data only when a parameter set different from the default parameter set is used.

Note that in both modes, it is possible to store the parameter set in SampleEntry, so the transmitting device 100 may always store the parameter set in VisualSampleEntry, and the receiving device 200 may always obtain the parameter set from VisualSampleEntry.

Note that in the MMT standard, MP4 header information such as Moov and Moof is transmitted as MPU metadata or movie fragment metadata, but the transmitting device 100 does not necessarily transmit MPU metadata and movie fragment metadata. You don't have to. Furthermore, the receiving device 200 determines whether SPS and PPS are stored in the sample data based on the ARIB (Association of Radio Industries and Businesses) standard service, asset type, or whether or not MPU meta is transmitted. It is also possible to judge.

FIG. 17 is a diagram illustrating an example in which pre-slice segment data and each slice segment are set to different data units.

In the example shown in FIG. 17, the data sizes of the pre-slice segment data and slice segments 1 to 4 are Length #1 to Length #5, respectively. The field values of Fragmentation indicator, Fragment counter, and Offset included in the header of the MMT packet are as shown in the figure.

Here, Offset indicates the bit length (offset) from the beginning of the encoded data of the sample (access unit or picture) to which the payload data belongs to the first byte of the payload data (encoded data) included in the MMT packet. This is offset information. Note that although the value of the Fragment counter will be explained as starting from the value obtained by subtracting 1 from the total number of fragments, it may start from another value.

FIG. 18 is a diagram illustrating an example in which a Data unit is fragmented. In the example shown in FIG. 18, slice segment 1 is divided into three fragments, and each fragment is stored in MMT packet #2 to MMT packet #4. Also at this time, assuming that the data size of each fragment is Length #2_1 to Length #2_3, the values of each field are as shown in the figure.

In this way, when a data unit such as a slice segment is set to Data unit, the beginning of the access unit and the beginning of the slice segment can be determined as follows based on the field value of the MMT packet header.

The head of the payload in a packet whose Offset value is 0 is the head of the access unit.

The beginning of the payload of a packet whose Offset value is different from 0 and whose Fragmentation index value is 00 or 01 is the beginning of a slice segment.

Furthermore, when fragmentation of the Data unit does not occur and no packet loss occurs, the receiving device 200 determines the number of slice segments that are stored in the MMT packet based on the number of slice segments acquired after detecting the beginning of the access unit. The index number of a slice segment can be determined.

Further, even when the Data unit of the pre-slice segment data is fragmented, the receiving device 200 can similarly detect the access unit and the beginning of the slice segment.

Furthermore, even if a packet loss occurs or if the SPS, PPS, and SEI included in the pre-slice segment data are set to separate data units, the receiving device 200 performs slice processing based on the analysis result of the MMT header. The starting position of a slice segment or tile within a picture (access unit) can be identified by identifying the MMT packet that stores the start data of the segment and then analyzing the header of the slice segment. The amount of processing involved in analyzing the slice header is small, and the processing load is not a problem.

In this way, each of the plurality of encoded data of the plurality of slice segments is in one-to-one correspondence with a basic data unit (Data unit) that is a unit of data stored in one or more packets. Moreover, each of the plurality of encoded data is stored in one or more MMT packets.

The header information of each MMT packet includes Fragmentation indicator (identification information) and Offset (offset information).

The receiving device 200 determines that the beginning of the payload data included in the packet having the header information including the Fragmentation indicator whose value is 00 or 01 is the beginning of the encoded data of each slice segment. . Specifically, the beginning of the payload data included in a packet having header information including an Offset whose value is not 0 and a Fragmentation indicator whose value is 00 or 01 is the beginning of the encoded data of each slice segment. It is determined that

Further, in the example of FIG. 17, the beginning of the Data unit is either the beginning of the access unit or the beginning of the slice segment, and the value of the Fragmentation indicator is 00 or 01. Furthermore, the receiving device 200 refers to the type of the NAL unit and determines whether the beginning of the Data Unit is an access unit delimiter or a slice segment, thereby determining whether the access unit is It is also possible to detect the beginning or the beginning of a slice segment.

In this way, the transmitting device 100 performs packetization so that the beginning of the NAL unit always starts from the beginning of the payload of the MMT packet, even when the pre-slice segment data is divided into multiple Data units. The receiving device 200 can detect the beginning of an access unit or a slice segment by analyzing the Fragmentation indicator and the NAL unit header. The NAL unit type is present in the first byte of the NAL unit header. Therefore, when analyzing the header portion of the MMT packet, the receiving device 200 can acquire the type of the NAL unit by additionally analyzing 1 byte of data. In the case of audio, the receiving device 200 only needs to be able to detect the beginning of the access unit, and may make a determination based on whether the value of the Fragmentation indicator is 00 or 01.

Furthermore, as described above, when storing encoded data that is encoded so that it can be dividedly decoded in a PES packet of an MPEG-2 TS, the transmitting device 100 can use a data alignment descriptor. be. Hereinafter, an example of a method of storing encoded data in a PES packet will be described in detail.

For example, in HEVC, the transmitting device 100 can indicate whether the data stored in the PES packet is an access unit, a slice segment, or a tile by using a data alignment descriptor. The alignment types in HEVC are defined as follows.

Alignment type=8 indicates a HEVC slice segment. Alignment type=9 indicates HEVC slice segment or access unit. Alignment type=12 indicates HEVC slice segment or tile.

Therefore, for example, by using type 9, the transmitting device 100 can indicate that the data of the PES packet is either a slice segment or pre-slice segment data. Since a type indicating a slice instead of a slice segment is also separately defined, the transmitting device 100 may use a type indicating a slice instead of a slice segment.

Furthermore, the DTS and PTS included in the header of the PES packet are set only in the PES packet that includes the first data of the access unit. Therefore, if the type is 9 and the DTS or PTS field exists in the PES packet, the receiving device 200 assumes that the entire access unit or the first division unit in the access unit is stored in the PES packet. Can be judged.

Further, the transmitting device 100 may enable the receiving device 200 to distinguish the data included in the packet by using a field such as transport_priority that indicates the priority of the TS packet that stores the PES packet containing the first data of the access unit. good. Further, the receiving device 200 may determine the data included in the packet by analyzing whether the payload of the PES packet is an access unit delimiter. Additionally, data_alignment_indicator in the PES packet header indicates whether data is stored in the PES packet according to these types. If this flag (data_alignment_indicator) is set to 1, it is guaranteed that the data stored in the PES packet follows the type indicated in the data alignment descriptor.

Further, the transmitting device 100 may use the data alignment descriptor only when converting into PES packets in division-decodable units such as slice segments. Thereby, if the data alignment descriptor exists, the receiving device 200 can determine that the encoded data has been made into PES packets in units that can be divided and decodable, and if the data alignment descriptor does not exist, the receiving device 200 can determine that the encoded data is divided into PES packets. It can be determined that the encoded data is converted into PES packets in units of access units. Note that when the data_alignment_indicator is set to 1 and the data alignment descriptor does not exist, it is specified in the MPEG-2 TS standard that the unit of PES packetization is an access unit.

If the data alignment descriptor is included in the PMT, the receiving device 200 determines that the PES packet has been made into units that can be divided and decoded, and inputs the data to each decoding unit based on the packetized unit. Data can be generated. Furthermore, if the PMT does not include a data alignment descriptor and it is determined that parallel decoding of encoded data is necessary based on program information or other descriptor information, the receiving device 200 , the slice header of the slice segment, etc. to generate input data to each decoding unit. Furthermore, if the encoded data can be decoded by a single decoding unit, the receiving device 200 decodes the data of the entire access unit using the decoding unit. Note that if information indicating whether the encoded data is composed of units that can be divided and decoded such as slice segments or tiles is separately indicated by a PMT descriptor, the receiving device 200 It may be determined whether the encoded data can be decoded in parallel based on the analysis result.

Furthermore, since the DTS and PTS included in the header of the PES packet are set only in the PES packet that includes the first data of the access unit, when the access unit is divided into PES packets, the second and subsequent PES The packet does not include information indicating the DTS and PTS of the access unit. Therefore, when performing decoding processing in parallel, each decoding section 204A to 204D and display section 205 use the DTS and PTS stored in the header of the PES packet containing the first data of the access unit.

(Embodiment 2)
In the second embodiment, a method of storing NAL size format data in an MP4 format-based MPU in MMT will be described. Note that, although a storage method in the MPU used for MMT will be described below as an example, such a storage method can also be applied to DASH, which is based on the same MP4 format.

[How to store in MPU]
In the MP4 format, multiple access units are stored together in one MP4 file. In the MPU used for MMT, data for each medium is stored in one MP4 file, and the data can include an arbitrary number of access units. Since the MPU is a unit that can be decoded by itself, for example, the MPU stores access units in units of GOP.

FIG. 19 is a diagram showing the configuration of the MPU. The beginning of the MPU is ftyp, mmpu, and moov, which are collectively defined as MPU metadata. moov stores initialization information common to files and an MMT hint track.

Moof also includes initialization information and size for each sample or subsample, information that can specify presentation time (PTS) and decoding time (DTS) (sample_duration, sample_size, sample_composition_time_offset), data_offset that indicates the position of data, etc. But Stored.

Further, each of the plurality of access units is stored as a sample in mdat (mdat box). Data excluding samples among moof and mdat is defined as movie fragment metadata (hereinafter referred to as MF metadata), and sample data of mdat is defined as media data.

FIG. 20 is a diagram showing the structure of MF metadata. As shown in FIG. 20, the MF metadata includes, in more detail, the type, length, and data of the moof box (moof), and the type, length, and length of the mdat box (mdat).

When storing access units in MP4 data, H. 264 and H. There are two modes: a mode in which parameter sets such as H.265 SPS and PPS can be stored as sample data, and a mode in which they cannot be stored.

Here, in the above storage-unable mode, the parameter set is stored in Decoder Specific Information of SampleEntry in moov. Also, in the storable mode, the parameter set is included within the sample.

MPU metadata, MF metadata, and media data are each stored in an MMT payload, and a fragment type (FT) is stored in the header of the MMT payload as an identifier that can identify these data. FT=0 indicates MPU metadata, FT=1 indicates MF metadata, and FT=2 indicates media data.

Note that although FIG. 19 shows an example in which MPU metadata units and MF metadata units are stored as data units in the MMT payload, units such as ftyp, mmpu, moov, and moof are stored as data units. It may be stored in the MMT payload in unit units. Similarly, FIG. 19 illustrates an example in which sample units are stored as data units in the MMT payload. However, a data unit may be configured in units of samples or in units of NAL units, and such data units may be stored in the MMT payload in units of data units. Such data units may be further stored in fragmented units in the MMT payload.

[Conventional transmission methods and issues]
Conventionally, when encapsulating multiple access units into the MP4 format, moovs and moofs were created when all the samples to be stored in the MP4 were completed.

When transmitting MP4 format in real time using broadcasting etc., for example, if the samples stored in one MP4 file are in units of GOP, moov and moof are created after time samples in units of GOP are accumulated. There is a delay associated with encapsulation. Such encapsulation at the transmitter always increases the end-to-end delay by a GOP unit of time. This makes it difficult to provide services in real time, leading to deterioration of services for viewers, especially when live content is transmitted.

FIG. 21 is a diagram for explaining the data transmission order. When applying MMT to broadcasting, as shown in FIG. ), there will be a delay in the transmission of the MMT packet due to encapsulation.

In order to prevent the delay caused by this encapsulation, as shown in FIG. 3-#6 in this order) has been proposed. Also, as shown in FIG. 20(c), the media data is transmitted first without waiting for the creation of the MPU header information, and the MPU header information is transmitted after the media data is transmitted (#3-#6, # A possible method is to transmit the data in the order of #1 and #2.

If the MPU header information is not transmitted, the receiving device decodes it without using the MPU header information, and if the MPU header information is sent later than the media data, the receiving device decodes the MPU header information. Wait for information to be obtained before decrypting.

However, with conventional MP4-compliant receiving devices, decoding without using MPU header information is not guaranteed. Furthermore, if a conventional transmission method is used when the receiving device performs decoding without using an MPU header through special processing, the decoding process becomes complicated and there is a high possibility that real-time decoding becomes difficult. Additionally, if the receiving device performs decoding after waiting to obtain MPU header information, it is necessary to buffer the media data until the receiving device obtains the header information, but there is no defined buffer model. decryption was not guaranteed.

Therefore, the transmitting device according to the second embodiment transmits the MPU metadata before the media data by storing only common information in the MPU metadata, as shown in FIG. 20(d). Then, the transmitting device according to the second embodiment transmits the MF metadata whose generation is delayed after the media data. This provides a transmission method or a reception method that can guarantee decoding of media data.

The reception method using each of the transmission methods shown in FIGS. 21(a) to 21(d) will be described below.

In each transmission method shown in FIG. 21, first, MPU data is configured in the order of MPU metadata, MFU metadata, and media data.

After configuring the MPU data, when the transmitting device transmits the data in the order of MPU metadata, MF metadata, and media data as shown in (a) of FIG. ) and (A-2).

(A-1) After acquiring the MPU header information (MPU metadata and MF metadata), the receiving device decodes the media data using the MPU header information.

(A-2) The receiving device decodes media data without using MPU header information.

In all of these methods, a delay occurs due to encapsulation on the transmitting side, but there is an advantage that the receiving device does not need to buffer the media data in order to obtain the MPU header. If buffering is not performed, there is no need to install memory for buffering, and no buffering delay occurs. Furthermore, since the method (A-1) performs decoding using MPU header information, it is also applicable to conventional receiving devices.

When the transmitting device transmits only media data as shown in FIG. 21(b), the receiving device can perform decoding using the method (B-1) below.

(B-1) The receiving device decodes media data without using MPU header information.

Furthermore, although not shown, if MPU metadata is transmitted before the media data is transmitted in FIG. 21(b), decoding can be performed using the method (B-2) below.

(B-2) The receiving device decodes the media data using the MPU metadata.

The advantages of both methods (B-1) and (B-2) above are that there is no delay due to encapsulation on the sending side, and there is no need to buffer the media data to obtain the MPU header. It is. However, since both methods (B-1) and (B-2) do not perform decoding using MPU header information, special processing may be required for decoding.

When the transmitting device transmits data in the order of media data, MPU metadata, and MF metadata as shown in (c) of FIG. 21, the receiving device performs the following (C-1) and (C-2). Decoding can be done using either method.

(C-1) After acquiring the MPU header information (MPU metadata and MF metadata), the receiving device decodes the media data.

(C-2) The receiving device decodes the media data without using the MPU header information.

When the method (C-1) above is used, it is necessary to buffer the media data in order to obtain the MPU header information. On the other hand, when the above method (C-2) is used, there is no need to perform buffering to obtain MPU header information.

Furthermore, in both methods (C-1) and (C-2), no delay due to encapsulation occurs on the transmitting side. Furthermore, since the method (C-2) does not use MPU header information, special processing may be required.

When the transmitting device transmits data in the order of MPU metadata, media data, and MF metadata as shown in (d) of FIG. 21, the receiving device transmits data in the following order (D-1) and (D-2). ) can be used to decrypt the file.

(D-1) After acquiring the MPU metadata, the receiving device further acquires the MF metadata, and then decodes the media data.

(D-2) After acquiring the MPU metadata, the receiving device decodes the media data without using the MF metadata.

When the method (D-1) above is used, it is necessary to buffer the media data in order to obtain MF metadata, but in the case of method (D-2) above, it is necessary to buffer the media data in order to obtain MF metadata. There is no need to buffer.

Since the above method (D-2) does not perform decoding using MF metadata, special processing may be required.

As explained above, if decoding is possible using MPU metadata and MF metadata, there is an advantage that it can be decoded by a conventional MP4 receiving device.

Note that in FIG. 21, the MPU data is configured in the order of MPU metadata, MFU metadata, and media data, and in moof, position information (offset) for each sample or subsample is determined based on this configuration. ing. The MF metadata also includes data other than media data in the mdat box (box size and type).

Therefore, when the receiving device identifies media data based on MF metadata, the receiving device reconstructs the data in the order in which the MPU data was configured, regardless of the order in which the data was transmitted. , decoding is performed using moov of MPU metadata or moof of MF metadata.

In addition, in FIG. 21, the MPU data is configured in the order of MPU metadata, MFU metadata, and media data, but even if the MPU data is configured in a different order from that in FIG. 21 and the position information (offset) is determined, good.

For example, the MPU data may be configured in the order of MPU metadata, media data, and MF metadata, and negative position information (offset) may be indicated in the MF metadata. In this case as well, regardless of the order in which the data is transmitted, the receiving device reconfigures the data in the order in which the MPU data was configured on the transmitting side, and then performs decoding using MOOV or MOof.

Note that the transmitting device may signal information indicating the order in which MPU data is configured, and the receiving device may reconfigure the data based on the signaled information.

As explained above, the receiving device receives packetized MPU metadata, packetized media data (sample data), and packetized MF metadata as shown in FIG. 21(d). Receive in order. Here, the MPU metadata is an example of first metadata, and the MF metadata is an example of second metadata.

Next, the receiving device reconstructs the MPU data (MP4 format file) including the received MPU metadata, the received MF metadata, and the received sample data. Then, sample data included in the reconstructed MPU data is decoded using the MPU metadata and MF metadata. MF metadata is metadata that includes data (for example, length stored in mbox) that can be generated only after sample data is generated on the transmitting side.

In addition, the operation of the above-mentioned receiving device is performed by each component that constitutes the receiving device in more detail. For example, the receiving device includes a receiving section that receives the data, a reconfiguration section that reconstructs the MPU data, and a decoding section that decodes the MPU data. Note that each of the receiving section, generating section, and decoding section is realized by a microcomputer, a processor, a dedicated circuit, or the like.

[How to decrypt without using header information]
Next, a method of decoding without using header information will be described. Here, a method for decoding without using header information at the receiving device will be described, regardless of whether header information is sent on the transmitting side or not. That is, this method is applicable to any of the transmission methods described using FIG. 21. However, some decoding methods are applicable only to specific transmission methods.

FIG. 22 is a diagram illustrating an example of a method for decoding without using header information. In FIG. 22, only the MMT payload and MMT packet containing only media data are illustrated, and the MMT payload and MMT packet containing MPU metadata and MF metadata are not illustrated. Furthermore, in the following description of FIG. 22, it is assumed that media data belonging to the same MPU is transmitted continuously. Furthermore, an example will be explained in which a sample is stored in the payload as media data, but in the following explanation of FIG. You can leave it there.

In order to decode media data, a receiving device must first obtain initialization information necessary for decoding. Furthermore, if the media is a video, the receiving device must obtain initialization information for each sample, specify the starting position of the MPU, which is a random access unit, and obtain the starting positions of samples and NAL units. . Further, the receiving device needs to specify the decoding time (DTS) and presentation time (PTS) of each sample.

Therefore, the receiving device can perform decoding without using header information using, for example, the following method. Note that if the payload stores a unit of NAL unit or a unit of fragmented NAL unit, "sample" in the following description may be replaced with "NAL unit in sample."

<Random access (=identifying the first sample of MPU)>
When header information is not transmitted, there are methods 1 and 2 below for the receiving device to identify the leading sample of the MPU. Note that when header information is transmitted, method 3 can be used.

[Method 1] The receiving device obtains a sample included in the MMT packet in which 'RAP_flag=1' in the MMT packet header.

[Method 2] The receiving device obtains a sample with 'sample number=0' in the MMT payload header.

[Method 3] If at least one of MPU metadata and MF metadata is transmitted before and after media data, the receiving device detects the fragment type in the MMT payload header. (FT) acquires a sample included in the MMT payload that has been switched to media data.

Note that in method 1 and method 2, when a plurality of samples belonging to different MPUs are mixed in one payload, it is impossible to determine which NAL unit is a random access point (RAP_flag=1 or sample number=0). For this reason, there are restrictions such as not mixing samples of different MPUs in one payload, or when samples of different MPUs are mixed in one payload, if the last (or first) sample is a random access point, RAP_flag Restrictions such as setting 1 to 1 are necessary.

Furthermore, in order for the receiving device to obtain the starting position of the NAL unit, it is necessary to shift the data read pointer by the size of the NAL unit sequentially from the first NAL unit of the sample.

If the data is fragmented, the receiving device can identify the data unit by referring to fragment_indicator and fragment_number.

<Determination of sample DTS>
Methods for determining the DTS of a sample include Method 1 and Method 2 below.

[Method 1] The receiving device determines the DTS of the first sample based on the prediction structure. However, this method requires analysis of encoded data and may be difficult to decode in real time, so the following method 2 is preferable.

[Method 2] The receiving device separately transmits the DTS of the first sample and acquires the transmitted DTS of the first sample. Examples of methods for transmitting the DTS of the first sample include a method of transmitting the DTS of the MPU first sample using MMT-SI, and a method of transmitting the DTS of each sample using the MMT packet header extension area. Note that DTS may be an absolute value or may be a relative value with respect to PTS. Alternatively, the transmitting side may signal whether or not the DTS of the first sample is included.

Note that in both method 1 and method 2, the DTS of subsequent samples is calculated assuming a fixed frame rate.

As a method for storing the DTS of each sample in the packet header, in addition to using an extension area, there is a method of storing the DTS of the sample included in the MMT packet in the 32-bit NTP timestamp field in the MMT packet header. If the DTS cannot be expressed using the number of bits (32 bits) of one packet header, the DTS may be expressed using a plurality of packet headers. Further, the DTS may be expressed by combining the NTP timestamp field of the packet header and an extension field. If DTS information is not included, a known value (for example, ALL0) is assumed.

<Determination of sample PTS>
The receiving device obtains the PTS of the first sample from the MPU timestamp descriptor for each asset included in the MPU. The receiving device calculates the subsequent sample PTS based on a parameter indicating the display order of samples such as POC, assuming that the frame rate is fixed. In this way, in order to calculate DTS and PTS without using header information, transmission at a fixed frame rate is essential.

In addition, when MF metadata is transmitted, the receiving device receives the relative time information of the DTS and PTS from the first sample indicated in the MF metadata, and the timestamp of the MPU first sample indicated in the MPU timestamp descriptor. The absolute values of DTS and PTS can be calculated from the absolute values.

Note that when analyzing the encoded data and calculating the DTS and PTS, the receiving device may use SEI information included in the access unit to calculate the DTS and PTS.

<Initialization information (parameter set)>
[For video]
For video, the parameter set is stored in the sample data. Furthermore, if MPU metadata and MF metadata are not transmitted, it is guaranteed that the parameter set necessary for decoding can be obtained by referring only to sample data.

Furthermore, as shown in FIGS. 21(a) and 21(d), when MPU metadata is transmitted before media data, it may be specified that no parameter set is stored in SampleEntry. In this case, the receiving device does not refer to the parameter set of SampleEntry, but only refers to the parameter set within the sample.

Furthermore, when MPU metadata is transmitted before media data, SampleEntry stores a parameter set common to the MPU and a default parameter set, and the receiving device stores the parameter set of SampleEntry and the parameter set in the sample. You may refer to it. By storing the parameter set in the SampleEntry, it becomes possible to perform decoding even with a conventional receiving device that cannot reproduce data unless the parameter set exists in the SampleEntry.

[For audio]
For audio, a LATM header is required for decoding, and for MP4, it is mandatory that the LATM header be included in the sample entry. However, if the header information is not transmitted, it is difficult for the receiving device to acquire the LATM header, so the LATM header is separately included in control information such as SI. Note that the LATM header may be included in a message, table, or descriptor. Note that the LATM header may be included within the sample.

The receiving device obtains the LATM header from the SI or the like before starting decoding, and starts decoding the audio. Alternatively, as shown in FIGS. 21(a) and 21(d), if the MPU metadata is transmitted before the media data, the receiving device receives the LATM header before the media data. It is possible. Therefore, if MPU metadata is transmitted before media data, it is possible to decode it even using a conventional receiving device.

<Others>
The transmission order and the type of transmission order may be notified as control information such as an MMT packet header, payload header, MPT or other table, message, or descriptor. Note that the types of transmission order here are, for example, the four types of transmission orders shown in (a) to (d) in FIG. 21, and an identifier for identifying each type can be obtained before starting decoding. It should be stored somewhere.

Furthermore, different types of transmission order may be used for audio and video, or a common type may be used for audio and video. Specifically, for example, audio is transmitted in the order of MPU metadata, MF metadata, and media data as shown in FIG. 21(a), and video is transmitted in the order of MPU metadata, MF metadata, and media data as shown in FIG. 21(d). , MPU metadata, media data, and MF metadata may be transmitted in this order.

With the method described above, the receiving device can perform decoding without using header information. Furthermore, if the MPU metadata is transmitted before the media data (FIG. 21(a) and FIG. 21(d)), it becomes possible to decode it even with a conventional receiving device.

In particular, by transmitting the MF metadata after the media data ((d) in FIG. 21), a delay due to encapsulation does not occur, and even a conventional receiving device can perform decoding.

[Configuration and operation of transmitter]
Next, the configuration and operation of the transmitter will be explained. FIG. 23 is a block diagram of a transmitting device according to the second embodiment, and FIG. 24 is a flowchart of a transmitting method according to the second embodiment.

As shown in FIG. 23, the transmitting device 15 includes an encoding section 16, a multiplexing section 17, and a transmitting section 18.

The encoding unit 16 converts video or audio to be encoded into, for example, H. Encoded data is generated by encoding according to H.265 (S10).

The multiplexing unit 17 multiplexes (packets) the encoded data generated by the encoding unit 16 (S11). Specifically, the multiplexing unit 17 packetizes each of sample data, MPU metadata, and MF metadata that constitute an MP4 format file. The sample data is data in which a video signal or an audio signal is encoded, MPU metadata is an example of first metadata, and MF metadata is an example of second metadata. The first metadata and the second metadata are both metadata used to decrypt sample data, but the difference between them is that the second metadata is data that can only be generated after the sample data is generated. It is to include.

Here, data that can be generated only after generation of sample data is, for example, data other than sample data stored in mdat in MP4 format (data in the header of mdat, that is, type and length shown in FIG. 20). ). Here, the second metadata only needs to include length, which is at least a part of this data.

The transmitter 18 transmits the packetized MP4 format file (S12). The transmitter 18 transmits the MP4 format file, for example, by the method shown in FIG. 21(d). That is, packetized MPU metadata, packetized sample data, and packetized MF metadata are transmitted in this order.

Note that each of the encoding section 16, the multiplexing section 17, and the transmitting section 18 is realized by a microcomputer, a processor, a dedicated circuit, or the like.

[Receiving device configuration]
Next, the configuration and operation of the receiving device will be explained. FIG. 25 is a block diagram of a receiving device according to Embodiment 2.

As shown in FIG. 25, the receiving device 20 includes a packet filtering unit 21, a transmission order type determining unit 22, a random access unit 23, a control information acquisition unit 24, a data acquisition unit 25, and a PTS and DTS calculation unit. unit 26 , initialization information acquisition unit 27 , decoding command unit 28 , decoding unit 29 , and presentation unit 30 .

[Receiving device operation 1]
First, when the media is a video, the operation of the receiving device 20 to specify the MPU start position and the NAL unit position will be described. FIG. 26 is a flowchart of such an operation of the receiving device 20. Note that here, it is assumed that the transmission order type of MPU data is stored in the SI information by the transmitting device 15 (multiplexing unit 17).

First, the packet filtering unit 21 performs packet filtering on the received file. The transmission order type determination unit 22 analyzes the SI information obtained by packet filtering and acquires the transmission order type of MPU data (S21).

Next, the transmission order type determining unit 22 determines (determines) whether the data after packet filtering includes MPU header information (at least one of MPU metadata or MF metadata) (S22). If the MPU header information (is included (Yes in S22), the random access unit 23 identifies the MPU leading sample by detecting that the fragment type of the MMT payload header is switched to media data (S23). ).

On the other hand, if the MPU header information is not included (No in S22), the random access unit 23 identifies the MPU leading sample based on the RAP_flag of the MMT packet header or the sample number of the MMT payload header (S24).

Furthermore, the transmission order type determining unit 22 determines whether or not the packet-filtered data includes MF metadata (S25). If it is determined that MF metadata is included (Yes in S25), the data acquisition unit 25 reads the NAL unit based on the sample, subsample offset, and size information included in the MF metadata. A NAL unit is thereby acquired (S26). On the other hand, if it is determined that MF metadata is not included (No in S25), the data acquisition unit 25 reads data of the size of the NAL unit in order from the first NAL unit of the sample to Acquire (S27).

Note that even if it is determined in step S22 that MPU header information is included, the receiving device 20 may identify the MPU first sample using the process in step S24 instead of step S23. Moreover, when it is determined that MPU header information is included, the process of step S23 and the process of step S24 may be used together.

Furthermore, even if it is determined in step S25 that MF metadata is included, the receiving device 20 may acquire the NAL unit using the process in step S27 without using the process in step S26. Further, when it is determined that MF metadata is included, the processing of step S23 and the processing of step S24 may be used together.

Furthermore, a case is assumed in which it is determined in step S25 that MF metadata is included, and the MF data is transmitted after the media data. In this case, the receiving device 20 may perform the process of step S26 after buffering the media data and waiting until the MF metadata is obtained, or the receiving device 20 may buffer the media data and wait until the MF metadata is obtained before performing the process of step S26. It may be determined whether or not to perform the process of step S27.

For example, the receiving device 20 may determine whether to wait for acquisition of MF metadata based on whether it has a buffer with a buffer size that can buffer media data. Further, the receiving device 20 may determine whether to wait for acquisition of MF metadata based on whether the end-to-end delay becomes small. Further, the receiving device 20 may perform the decoding process mainly using the process of step S26, and may use the process of step S27 in a processing mode when a packet loss or the like occurs.

Note that if the transmission order type is predetermined, steps S22 and S26 may be omitted, and in this case, the receiving device 20 selects the MPU in consideration of the buffer size and End-to-End delay. The method for specifying the leading sample and the method for specifying the NAL unit may be determined.

Note that if the transmission order type is known in advance, the transmission order type determination unit 22 in the receiving device 20 is not necessary.

Furthermore, although not explained in FIG. The data acquired in the section is output to the decoding section 29. The decoding unit 29 decodes the data, and the presentation unit 30 presents the decoded data.

[Receiving device operation 2]
Next, an operation in which the receiving device 20 acquires initialization information based on the transmission order type and decodes media data based on the initialization information will be described. FIG. 27 is a flowchart of such an operation.

First, the packet filtering unit 21 performs packet filtering on the received file. The transmission order type determination unit 22 analyzes the SI information obtained by packet filtering and acquires the transmission order type (S301).

Next, the transmission order type determination unit 22 determines whether MPU metadata is being transmitted (S302). If it is determined that the MPU metadata has been transmitted (Yes in S302), the transmission order type determination unit 22 determines whether the MPU metadata is transmitted before the media data as a result of the analysis in step S301. (S303). If the MPU metadata is transmitted before the media data (Yes in S303), the initialization information acquisition unit 27 performs initialization based on the common initialization information included in the MPU metadata and the initialization information of the sample data. and decrypts the media data (S304).

On the other hand, if it is determined that the MPU metadata is transmitted after the media data (No in S303), the data acquisition unit 25 buffers the media data until the MPU metadata is acquired (S305). , the process of step S304 is performed after the MPU metadata is acquired.

Further, if it is determined in step S302 that the MPU metadata has not been transmitted (No in S302), the initialization information acquisition unit 27 decodes the media data based only on the initialization information of the sample data. (S306).

Note that if decoding of the media data is guaranteed only based on the initialization information of the sample data on the transmitting side, the process based on the determinations in steps S302 and S303 is not performed, and the process in step S306 is used.

Further, the receiving device 20 may determine whether or not to buffer the media data before step S305. In this case, if the receiving device 20 determines to buffer the media data, the process proceeds to step S305, and if it determines not to buffer the media data, the process proceeds to step S306. The determination as to whether or not to buffer media data may be made based on the buffer size and occupancy of the receiving device 20, or for example, the one with the smaller End-to-End delay may be selected. The determination may be made taking into account the -to-End delay.

[Receiving device operation 3]
Here, details of the transmission method and reception method in the case where the MF metadata is transmitted after the media data (FIGS. 21(c) and 21(d)) will be described. Below, the case of FIG. 21(d) will be explained as an example. Note that in transmission, only the method shown in FIG. 21(d) is used, and transmission order type signaling is not performed.

As mentioned above, when transmitting data in the order of MPU metadata, media data, and MF metadata as shown in FIG. 21(d),
(D-1) The receiving device 20 decodes the media data after acquiring the MPU metadata and further acquiring the MF metadata.

(D-2) After acquiring the MPU metadata, the receiving device 20 decodes the media data without using the MF metadata.
Two decoding methods are possible.

Here, D-1 requires buffering of media data to obtain MF metadata, but it can be decoded using MPU header information, so it can be decoded by a conventional MP4-compliant receiving device. becomes. Further, D-2 does not require buffering of media data to obtain MF metadata, but requires special processing for decoding because it cannot be decoded using MF metadata.

Furthermore, the method shown in FIG. 21(d) has the advantage that, since the MF metadata is transmitted after the media data, a delay due to encapsulation does not occur, and the end-to-end delay can be reduced.

The receiving device 20 can select one of the above two decoding methods depending on the capability of the receiving device 20 and the quality of service provided by the receiving device 20.

The transmitting device 15 must ensure that decoding can be performed while reducing the occurrence of buffer overflow or underflow in the decoding operation in the receiving device 20. For example, the following parameters can be used as elements for defining a decoder model when decoding is performed using method D-1.

・Buffer size for reconfiguring MPU (MPU buffer)
For example, buffer size=maximum rate×maximum MPU time×α, where the maximum rate is the upper limit rate of the encoded data profile and level+the overhead of the MPU header. Further, the maximum MPU time is the maximum time length of a GOP when 1 MPU = 1 GOP (video).

Here, the audio may be a GOP unit common to the video, or may be a separate unit. α is a margin to prevent overflow, and may be multiplied or added to maximum rate×maximum MPU time. When multiplied, α≧1, and when added, α≧0.

- Upper limit of decoding delay time from when data is input to the MPU buffer until it is decoded. (TSTD_delay in MPEG-TS STD)
For example, at the time of transmission, the maximum MPU time and the upper limit of the decoding delay time are taken into consideration, and the DTS is set so that the acquisition completion time of the MPU data in the receiver<=DTS.

Further, the transmitting device 15 may add DTS and PTS according to the decoder model when decoding using method D-1. Thereby, the transmitting device 15 guarantees the decoding of the receiving device that performs decoding using method D-1, and at the same time transmits the auxiliary information necessary when decoding is performed using method D-2. It's okay.

For example, the transmitting device 15 can guarantee the operation of the receiving device that decodes using method D-2 by signaling the pre-buffering time in the decoder buffer when decoding using method D-2.

The pre-buffering time may be included in SI control information such as a message, table, or descriptor, or may be included in the header of an MMT packet or MMT payload. Furthermore, the SEI within the encoded data may be overwritten. DTS and PTS for decoding using method D-1 are stored in the MPU timestamp descriptor and SampleEntry, and DTS and PTS for decoding using method D-2 or pre-buffering time are stored in the MPU timestamp descriptor and SampleEntry. It may be described in SEI.

If the receiving device 20 supports only MP4-compliant decoding using an MPU header, the receiving device 20 selects decoding method D-1 and supports both D-1 and D-2. If so, you may choose one or the other.

The transmitting device 15 may provide DTS and PTS to ensure the decoding operation of one side (D-1 in this description), and may also transmit supplementary information to assist the decoding operation of one side.

Also, when method D-2 is used, there is a possibility that the End-to-End delay will be larger due to the delay caused by pre-buffering of MF metadata compared to when method D-1 is used. is high. Therefore, when the receiving device 20 wants to reduce the end-to-end delay, it may select method D-2 for decoding. For example, when the receiving device 20 always wants to reduce the End-to-End delay, it may always use method D-2. Further, the receiving device 20 may use method D-2 only when operating in a low-delay presentation mode where it wants to present live content, channel selection, zapping operation, etc. with low delay.

FIG. 28 is a flowchart of such a receiving method.

First, the receiving device 20 receives an MMT packet and obtains MPU data (S401). Then, the receiving device 20 (transmission order type determining unit 22) determines whether or not to present the program in the low-delay presentation mode (S402).

If the program is not presented in the low-delay presentation mode (No in S402), the receiving device 20 (random access unit 23 and initialization information acquisition unit 27) uses the header information to acquire random access and initialization information (S405). ). Further, the receiving device 20 (PTS, DTS calculating unit 26, decoding command unit 28, decoding unit 29, presentation unit 30) performs decoding and presentation processing based on the PTS and DTS given by the transmitting side (S406).

On the other hand, when presenting the program in the low-delay presentation mode (Yes in S402), the receiving device 20 (random access unit 23 and initialization information acquisition unit 27) uses a decoding method that does not use header information to perform random access, Initialization information is acquired (S403). Further, the receiving device 20 performs decoding and presentation processing based on the PTS, DTS, and auxiliary information for decoding without using the header information provided on the transmitting side (S404). Note that in steps S403 and S404, processing may be performed using MPU metadata.

[Transmission/reception method using auxiliary data]
The transmission and reception operations in the case where the MF metadata is transmitted after the media data (cases of (c) and (d) in FIG. 21) have been described above. Next, a method will be described in which the transmitting device 15 can start decoding earlier and reduce End-to-End delay by transmitting auxiliary data having some functions of MF metadata. Here, an example will be described in which auxiliary data is further transmitted based on the transmission method shown in FIG. 21(d), but methods using auxiliary data are shown in FIGS. 21(a) to (c). It is also applicable to other transmission methods.

FIG. 29(a) is a diagram showing an MMT packet transmitted using the method shown in FIG. 21(d). That is, data is transmitted in the order of MPU metadata, media data, and MF metadata.

Here, sample #1, sample #2, sample #3, and sample #4 are samples included in the media data. Note that although an example in which media data is stored in an MMT packet in units of samples will be explained here, media data may be stored in an MMT packet in units of NAL units, or in units of divided NAL units. may be done. Note that a plurality of NAL units may be aggregated and stored in an MMT packet.

As explained in D-1 above, in the case of the method shown in FIG. 21(d), that is, when data is transmitted in the order of MPU metadata, media data, and MF metadata, MPU metadata After the acquisition, there is a method of further acquiring MF metadata and then decoding the media data. In method D-1, buffering of media data is required to obtain MF metadata, but since decoding is performed using MPU header information, conventional MP4-compliant receiving devices can also use D-1. Method -1 has the advantage of being applicable. On the other hand, the receiving device 20 has the disadvantage that it has to wait to start decoding until it obtains the MF metadata.

On the other hand, as shown in FIG. 29(b), in the method using auxiliary data, the auxiliary data is transmitted before the MF metadata.

The MF metadata includes information indicating the DTS, PTS, offset, and size of all samples included in the movie fragment. On the other hand, the auxiliary data includes information indicating the DTS, PTS, offset, and size of some of the samples included in the movie fragment.

For example, MF metadata includes information about all samples (sample #1-sample #4), whereas auxiliary data includes information about some samples (sample #1-#2). .

In the case shown in FIG. 29(b), sample #1 and sample #2 can be decoded by using auxiliary data, so End-to-Ent Delay is reduced. Note that the auxiliary data may include any combination of sample information, and the auxiliary data may be transmitted repeatedly.

For example, in (c) of FIG. 29, when transmitting the auxiliary information at timing A, the transmitting device 15 includes the information of sample #1 in the auxiliary information, and when transmitting the auxiliary information at timing B, Include information on sample #1 and sample #2 in the auxiliary information. When transmitting the auxiliary information at timing C, the transmitting device 15 includes information on sample #1, sample #2, and sample #3 in the auxiliary information.

Note that the MF metadata includes information on sample #1, sample #2, sample #3, and sample #4 (information on all samples in the movie fragment).

Ancillary data does not necessarily need to be transmitted immediately after it is generated.

Note that in the header of the MMT packet or MMT payload, a type indicating that auxiliary data is stored is specified.

For example, when auxiliary data is stored in the MMT payload using MPU mode, a data type indicating that it is auxiliary data is specified as the fragment_type field value (for example, FT=3). The auxiliary data may be data based on the moof configuration, or may have other configurations.

When auxiliary data is stored as a control signal (descriptor, table, message) in the MMT payload, a descriptor tag, table ID, message ID, etc. indicating that the auxiliary data is auxiliary data are specified.

Furthermore, PTS or DTS may be stored in the header of the MMT packet or MMT payload.

[Example of generation of auxiliary data]
An example in which the transmitting device generates auxiliary data based on the configuration of moof will be described below. FIG. 30 is a diagram for explaining an example in which the transmitting device generates auxiliary data based on the configuration of moof.

In normal MP4, a moof is created for a movie fragment, as shown in FIG. The moof contains information indicating the DTS, PTS, offset, and size of the sample included in the movie fragment.

Here, the transmitting device 15 configures MP4 (MP4 file) using only some sample data among the sample data that configures the MPU, and generates auxiliary data.

For example, as shown in FIG. 30(a), the transmitting device 15 generates MP4 using only sample #1 among samples #1 to #4 configuring the MPU, and includes the header of moof+mdat. Use as auxiliary data.

Next, as shown in FIG. 30(b), the transmitting device 15 generates MP4 using sample #1 and sample #2 among samples #1 to #4 configuring the MPU, and among them, The header of moof+mdat is used as the next auxiliary data.

Next, as shown in FIG. 30(c), the transmitting device 15 transmits MP4 using sample #1, sample #2, and sample #3 among samples #1 to #4 that constitute the MPU. The header of moof+mdat is used as the next auxiliary data.

Next, as shown in FIG. 30(d), the transmitting device 15 generates all MP4s out of samples #1 to #4 configuring the MPU, and among them, the header of moof+mdat is movie fragment metadata. becomes.

Although the transmitting device 15 generates auxiliary data for every sample here, it may also generate auxiliary data for every N samples. The value of N is an arbitrary number. For example, when transmitting auxiliary data M times when transmitting one MPU, N=total samples/M may be used.

Note that the information indicating the sample offset in moof may be an offset value after the sample entry area for the number of subsequent samples is secured as a NULL area.

Note that the auxiliary data may be generated in such a manner that the MF metadata is fragmented.

[Example of reception operation using auxiliary data]
The reception of auxiliary data generated as described in FIG. 30 will be explained. FIG. 31 is a diagram for explaining reception of auxiliary data. In FIG. 31(a), it is assumed that the number of samples forming the MPU is 30, and auxiliary data is generated and transmitted every 10 samples.

In (a) of FIG. 30, auxiliary data #1 contains samples #1 to #10, auxiliary data #2 contains samples #1 to #20, and MF metadata contains samples #1 to #30. Information is included for each.

Although samples #1 to #10, samples #11 to #20, and samples #21 to #30 are stored in one MMT payload, they may be stored in sample units or NAL units, or in fragments. It may also be stored in an aggregated unit.

The receiving device 20 receives each packet of MPU meta, sample, MF meta, and auxiliary data.

The receiving device 20 concatenates the sample data in the order of reception (backwards), and updates the previous auxiliary data after receiving the latest auxiliary data. Further, the receiving device 20 can configure a complete MPU by finally replacing the auxiliary data with MF metadata.

When the receiving device 20 receives the auxiliary data #1, it concatenates the data as shown in the upper part of FIG. 31(b) to configure MP4. Thereby, the receiving device 20 can parse samples #1 to #10 using the MPU metadata and the information of the auxiliary data #1, and the information on the PTS, DTS, offset, and size included in the auxiliary data. Decoding can be performed based on

Furthermore, upon receiving the auxiliary data #2, the receiving device 20 concatenates the data as shown in the middle row of FIG. 31(b) to configure MP4. Thereby, the receiving device 20 can parse samples #1 to #20 using the MPU metadata and the information in the auxiliary data #2, and the information on the PTS, DTS, offset, and size included in the auxiliary data. decoding can be performed based on the

Furthermore, upon receiving the MF metadata, the receiving device 20 concatenates the data as shown in the lower part of FIG. 31(b) to configure MP4. Thereby, the receiving device 20 can parse samples #1 to #30 using the MPU metadata and MF metadata, and based on the PTS, DTS, offset, and size information included in the MF metadata. can be decrypted using

If there is no auxiliary data, the receiving device 20 can acquire sample information only after receiving the MF metadata, so it was necessary to start decoding after receiving the MF metadata. However, since the transmitting device 15 generates and transmits the auxiliary data, the receiving device 20 can acquire sample information using the auxiliary data without waiting for the reception of MF metadata, thereby speeding up the decoding start time. be able to. Further, since the transmitting device 15 generates the auxiliary data based on the moof described using FIG. 30, the receiving device 20 can use the conventional MP4 parser as is to perform parsing.

Furthermore, newly generated auxiliary data and MF metadata include information on samples that overlap with previously transmitted auxiliary data. Therefore, even if past auxiliary data cannot be acquired due to packet loss, etc., MP4 can be reconstructed using newly acquired auxiliary data and MF metadata, and sample information (PTS, DTS, size, and offset).

Note that the auxiliary data does not necessarily need to include information on past sample data. For example, auxiliary data #1 may correspond to sample data #1-#10, and auxiliary data #2 may correspond to sample data #11-#20. For example, as shown in FIG. 31C, the transmitting device 15 may sequentially transmit complete MF metadata as a data unit and fragments of the data unit as auxiliary data.

Furthermore, the transmitting device 15 may repeatedly transmit auxiliary data or MF metadata in order to prevent packet loss.

Note that the MMT packet and MMT payload in which auxiliary data are stored include an MPU sequence number and an asset ID, similar to the MPU metadata, MF metadata, and sample data.

The reception operation using the above-mentioned auxiliary data will be explained using the flowchart of FIG. 32. FIG. 32 is a flowchart of a receiving operation using auxiliary data.

First, the receiving device 20 receives an MMT packet and analyzes the packet header and payload header (S501). Next, the receiving device 20 analyzes whether the fragment type is auxiliary data or MF metadata (S502), and if the fragment type is auxiliary data, it updates the previous auxiliary data by overwriting it (S503). . At this time, if there is no past auxiliary data of the same MPU, the receiving device 20 uses the received auxiliary data as new auxiliary data. Then, the receiving device 20 acquires a sample and decodes it based on the MPU metadata, auxiliary data, and sample data (S507).

On the other hand, if the fragment type is MF metadata, the receiving device 20 overwrites the past auxiliary data with MF metadata in step S505 (S505). Then, the receiving device 20 acquires a sample in the form of a complete MPU based on the MPU metadata, MF metadata, and sample data, and decodes it (S506).

Although not shown in FIG. 32, in step S502, the receiving device 20 stores the data in a buffer when the fragment type is MPU metadata, and stores the data in the buffer after each sample when the fragment type is sample data. Store the concatenated data in a buffer.

If the auxiliary data cannot be acquired due to packet loss, the receiving device 20 can decode the sample by overwriting with the latest auxiliary data or by using past auxiliary data.

Note that the sending cycle and the number of sending times of the auxiliary data may be predetermined values. Information on the transmission cycle and number of times (count, countdown) may be transmitted together with the data. For example, a transmission cycle, the number of transmissions, and a timestamp such as initial_cpb_removal_delay may be stored in the data unit header.

By transmitting auxiliary data containing information of the first sample of the MPU one or more times before initial_cpb_removal_delay, it is possible to follow the CPB buffer model. At this time, a value based on the picture timing SEI is stored in the MPU timestamp descriptor.

Note that the transmission method in the reception operation using such auxiliary data is not limited to the MMT method, and can be applied to streaming transmission of packets configured in the ISOBMFF file format, such as MPEG-DASH.

[Transmission method when one MPU is composed of multiple movie fragments]
In the explanation from FIG. 19 onward, one MPU is composed of one movie fragment, but here, a case will be described where one MPU is composed of a plurality of movie fragments. FIG. 33 is a diagram showing the configuration of an MPU composed of a plurality of movie fragments.

In FIG. 33, samples (#1-#6) stored in one MPU are stored separately into two movie fragments. A first movie fragment is generated based on samples #1-#3 and a corresponding moof box is generated. A second movie fragment is generated based on samples #4-#6 and a corresponding moof box is generated.

The moof box and mdat box headers in the first movie fragment are stored in the MMT payload and MMT packet as movie fragment metadata #1. On the other hand, the headers of the moof box and mdat box in the second movie fragment are stored in the MMT payload and MMT packet as movie fragment metadata #2. Note that in FIG. 33, the MMT payload in which movie fragment metadata is stored is hatched.

Note that the number of samples configuring an MPU and the number of samples configuring a movie fragment are arbitrary. For example, two movie fragments may be configured, with the number of samples configuring the MPU being the number of samples per GOP, and the number of samples being half the number of samples per GOP being the movie fragment.

Note that although an example in which one MPU includes two movie fragments (moof box and mdat box) is shown here, the number of movie fragments included in one MPU does not have to be two, but may be three or more. Furthermore, the samples stored in a movie fragment may not be divided into equal numbers of samples, but may be divided into an arbitrary number of samples.

Note that in FIG. 33, the MPU metadata unit and the MF metadata unit are each stored in the MMT payload as a data unit. However, the transmitting device 15 may store units such as ftyp, mmpu, moov, and moof as data units in the MMT payload in data unit units, or may store data units in fragmented units in the MMT payload. Good too. Further, the transmitting device 15 may store data units in the MMT payload in units of aggregated data units.

Further, in FIG. 33, samples are stored in the MMT payload in units of samples. However, the transmitting device 15 may configure a data unit not only in units of samples but also in units of NAL units or units of a plurality of NAL units, and store the data units in the MMT payload in units of data units. Further, the transmitting device 15 may store the fragmented data unit in the MMT payload, or may store the data unit in the MMT payload in the aggregated unit.

In addition, in FIG. 33, the MPU is configured in the order of moof#1, mdat#1, moof#2, and mdat#2, and an offset is given to moof#1 assuming that the corresponding mdat#1 follows. There is. However, if mdat #1 comes before moof #1, offset may be assigned. However, in this case, movie fragment metadata cannot be generated in the form of moof+mdat, and the moof and mdat headers are transmitted separately.

Next, the transmission order of MMT packets when the MPU having the configuration described in FIG. 33 is transmitted will be described. FIG. 34 is a diagram for explaining the transmission order of MMT packets.

FIG. 34(a) shows the transmission order when MMT packets are transmitted in the MPU configuration order shown in FIG. 33. Specifically, (a) in FIG. 34 shows MPU meta, MF meta #1, media data #1 (samples #1-#3), MF meta #2, media data #2 (samples #4-#6). ) is shown below.

In (b) of FIG. 34, MPU meta, media data #1 (samples #1 to #3), MF meta #1, media data #2 (samples #4 to #6), and MF meta #2 are transmitted in this order. Give an example.

In (c) of FIG. 34, media data #1 (samples #1 to #3), MPU meta, MF meta #1, media data #2 (samples #4 to #6), and MF meta #2 are transmitted in this order. Give an example.

MF meta #1 is generated using samples #1 to #3, and MF meta #2 is generated using samples #4 to #6. Therefore, when the transmission method shown in FIG. 34A is used, a delay occurs in the transmission of sample data due to encapsulation.

On the other hand, when the transmission methods shown in FIGS. 34(b) and 34(c) are used, samples can be transmitted without waiting for MF meta to be generated, so the delay due to encapsulation is reduced. This does not occur, and the end-to-end delay can be reduced.

Also, in the transmission order (a) in FIG. 34, one MPU is divided into multiple movie fragments, and the number of samples stored in the MF meta is smaller than in the case of FIG. The amount of delay due to encapsulation can be made smaller than in the case of encapsulation.

Note that, in addition to the method shown here, for example, the transmitting device 15 may connect MF meta #1 and MF meta #2 and transmit them together at the end of the MPU. In this case, MF metas of different movie fragments may be aggregated and stored in one MMT payload. Further, MF metas of different MPUs may be aggregated together and stored in the MMT payload.

[Reception method when one MPU is composed of multiple movie fragments]
Here, an example of the operation of the receiving device 20 that receives and decodes MMT packets transmitted in the transmission order described in FIG. 34(b) will be described. 35 and 36 are diagrams for explaining such an example of operation.

The receiving device 20 receives MMT packets including an MPU meta, a sample, and an MF meta, which are transmitted in the transmission order shown in FIG. 35. Sample data is concatenated in the order received.

The receiving device 20 concatenates data as shown in (1) of FIG. 36 at T1, which is the time when MF meta #1 is received, and forms MP4. Thereby, the receiving device 20 can acquire samples #1 to #3 based on the MPU metadata and the information in the MF meta #1, and the information on the PTS, DTS, offset, and size included in the MF meta. Decoding can be performed based on

Further, the receiving device 20 concatenates data at T2, which is the time when MF meta #2 is received, as shown in (2) of FIG. 36, and configures MP4. Thereby, the receiving device 20 can acquire samples #4 to #6 based on the MPU metadata and the information on the MF meta #2, and can obtain samples #4 to #6 based on the information on the PTS, DTS, offset, and size of the MF meta. can be decrypted using In addition, the receiving device 20 concatenates the data as shown in (3) of FIG. 36 to configure MP4, thereby generating samples #1 to #6 based on the information of MF meta #1 and MF meta #2. You may obtain it.

By dividing one MPU into a plurality of movie fragments, the time it takes for the MPU to acquire the first MF meta is shortened, so the decoding start time can be accelerated. Furthermore, the buffer size for storing samples before decoding can be reduced.

Note that the transmitting device 15 determines that the time from transmitting (or receiving) the first sample in the movie fragment to transmitting (or receiving) the MF meta corresponding to the movie fragment is shorter than the initial_cpb_removal_delay specified by the encoder. The division unit of the movie fragment may be set so that By setting in this way, the reception buffer can follow the cpb buffer, and low-delay decoding can be achieved. In this case, absolute time based on initial_cpb_removal_delay can be used for PTS and DTS.

Further, the transmitting device 15 may divide the movie fragments at equal intervals, or divide subsequent movie fragments at shorter intervals than the previous movie fragment. Thereby, the receiving device 20 can always receive the MF meta including information about the sample before decoding the sample, and continuous decoding becomes possible.

The following two methods can be used to calculate the absolute time of PTS and DTS.

(1) The absolute time of PTS and DTS is determined based on the reception time (T1 or T2) of MF meta #1 and MF meta #2 and the relative time of PTS and DTS included in the MF meta.

(2) The absolute time of the PTS and DTS is determined based on the absolute time signaled from the transmitting side, such as the MPU time stamp descriptor, and the relative time of the PTS and DTS included in the MF meta.

Furthermore, (2-A) the absolute time signaled by the transmitting device 15 may be an absolute time calculated based on initial_cpb_removal_delay specified by the encoder.

Furthermore, (2-B) the absolute time signaled by the transmitting device 15 may be an absolute time calculated based on a predicted value of the MF meta reception time.

Note that MF meta #1 and MF meta #2 may be transmitted repeatedly. By repeatedly transmitting MF meta #1 and MF meta #2, the receiving device 20 can obtain the MF meta again even if it is unable to obtain the MF meta due to packet loss or the like.

An identifier indicating the order of the movie fragments can be stored in the payload header of the MFU containing the samples making up the movie fragments. On the other hand, the MMT payload does not include an identifier indicating the order of MF meta that constitutes a movie fragment. Therefore, the receiving device 20 identifies the order of the MF meta using packet_sequence_number. Alternatively, the transmitting device 15 stores an identifier indicating which movie fragment the MF meta belongs to in control information (message, table, descriptor), an MMT header, an MMT payload header, or a data unit header, and sends the signal. You may.

Note that the transmitting device 15 transmits the MPU meta, the MF meta, and the sample in a predetermined transmission order, and the receiving device 20 performs reception processing based on the predetermined transmission order. It's okay. Alternatively, the transmitting device 15 may signal the transmission order, and the receiving device 20 may select (judge) the receiving process based on the signaling information.

The above reception method will be explained using FIG. 37. FIG. 37 is a flowchart of the operation of the reception method described in FIGS. 35 and 36.

First, the receiving device 20 determines (identifies) whether the data included in the payload is MPU metadata, MF metadata, or sample data (MFU) based on the fragment type indicated in the MMT payload (S601 , S602). If the data is sample data, the receiving device 20 buffers the sample and waits for reception of MF metadata corresponding to the sample and start of decoding (S603).

On the other hand, in step S602, if the data is MF metadata, the receiving device 20 acquires sample information (PTS, DTS, position information, and size) from the MF metadata, and applies the acquired sample information to the sample information. A sample is obtained based on the PTS and the DTS, and the sample is decoded and presented based on the PTS and DTS (S604).

Although not shown, if the data is MPU metadata, the MPU metadata includes initialization information necessary for decoding. Therefore, the receiving device 20 accumulates this and uses it to decode the sample data in step S604.

Note that when the receiving device 20 stores the received MPU data (MPU metadata, MF metadata, and sample data) in the storage device, it rearranges the received MPU data into the MPU configuration described in FIG. 19 or FIG. 33. After that, it accumulates.

Note that on the transmitting side, packet sequence numbers are assigned to MMT packets with respect to packets having the same packet ID. At this time, packet sequence numbers may be assigned after the MMT packets including MPU metadata, MF metadata, and sample data are rearranged in the transmission order, or packet sequence numbers may be assigned in the order before rearranging. Good too.

When packet sequence numbers are assigned in the order before rearrangement, the receiving device 20 can rearrange the data in the configuration order of the MPU based on the packet sequence numbers, making storage easier.

[Method of detecting the beginning of an access unit and the beginning of a slice segment]
A method for detecting the beginning of an access unit or the beginning of a slice segment based on the information of the MMT packet header and the MMT payload header will be described.

Here, when non-VCL NAL units (access unit delimiter, VPS, SPS, PPS, SEI, etc.) are collectively stored in the MMT payload as a data unit, and when each non-VCL NAL unit is stored as a data unit, Two examples of aggregating and storing in one MMT payload are shown below.

FIG. 38 is a diagram showing a case where non-VCL NAL units are individually treated as data units and aggregated.

In the case of FIG. 38, the head of the access unit is an MMT packet whose fragment_type value is MFU, and is the head data of an MMT payload that includes a data unit whose aggregation_flag value is 1 and whose offset value is 0. At this time, the Fragmentation_indicator value is 0.

In the case of FIG. 38, the beginning of the slice segment is an MMT packet whose fragment_type value is MFU, and is the beginning data of an MMT payload whose aggregation_flag value is 0 and fragmentation_indicator value is 00 or 01.

FIG. 39 is a diagram showing a case in which non-VCL NAL units are combined into a data unit. Note that the field values of the packet header are as shown in FIG. 17 (or FIG. 18).

In the case of FIG. 39, the head of the access unit is the head data of the payload in the packet whose Offset value is 0.

Further, in the case of FIG. 39, the beginning of the slice segment has an Offset value different from 0, and the beginning data of the payload of the packet whose fragmentation indicator value is 00 or 01 becomes the beginning of the slice segment.

[Reception processing when packet loss occurs]
Normally, when transmitting MP4 format data in an environment where packet loss occurs, the receiving device 20 restores the packet using ALFEC (Application Layer FEC), packet retransmission control, or the like.

However, if AL-FEC is not used in streaming such as broadcasting and packet loss occurs, the packets cannot be restored.

After data is lost due to packet loss, the receiving device 20 needs to restart decoding of video and audio. To do this, the receiving device 20 needs to detect the beginning of the access unit or NAL unit and start decoding from the beginning of the access unit or NAL unit.

However, since a start code is not attached to the beginning of a NAL unit in MP4 format, the receiving device 20 cannot detect the beginning of an access unit or a NAL unit even if it analyzes the stream.

FIG. 40 is a flowchart of the operation of the receiving device 20 when a packet loss occurs.

The receiving device 20 detects packet loss based on the packet sequence number, packet counter, fragment counter, etc. in the header of the MMT packet or MMT payload (S701), and determines which packet has been lost from the context relationship (S702). .

If it is determined that no packet loss has occurred (No in S702), the receiving device 20 constructs an MP4 file and decodes the access unit or NAL unit (S703).

If it is determined that a packet loss has occurred (Yes in S702), the receiving device 20 generates a NAL unit corresponding to the NAL unit in which the packet loss occurred using dummy data, and configures an MP4 file (S704). When putting dummy data into a NAL unit, the receiving device 20 indicates that the data is dummy data as the type of the NAL unit.

Further, the receiving device 20 detects the beginning of the next access unit or NAL unit based on the method described in FIGS. 17, 18, 38, and 39, and inputs the data from the beginning to the decoder. Decoding can be restarted (S705).

Note that when a packet loss occurs, the receiving device 20 may restart decoding from the beginning of the access unit and the NAL unit based on information detected based on the packet header, or may resume decoding from the beginning of the access unit and the NAL unit or decode the dummy data NAL unit. Decoding may be restarted from the beginning of the access unit and NAL unit based on the header information of the reconstructed MP4 file, including the header information of the reconstructed MP4 file.

When storing MP4 files (MPU), the receiving device 20 may separately acquire packet data (NAL units, etc.) lost due to packet loss from broadcasting or communication and store (replace) them.

At this time, when acquiring the lost packet from the communication, the receiving device 20 sends the information of the lost packet (packet ID, MPU sequence number, packet sequence number, IP data flow number, IP address, etc.) to the server. and obtain the packet. The receiving device 20 may obtain not only the lost packet but also a group of packets before and after the lost packet at the same time.

[How to configure movie fragments]
Here, a method for configuring movie fragments will be explained in detail.

As explained with reference to FIG. 33, the number of samples making up a movie fragment and the number of movie fragments making up one MPU are arbitrary. For example, the number of samples that make up a movie fragment and the number of movie fragments that make up one MPU may be a fixed predetermined number, or may be determined dynamically.

Here, by configuring a movie fragment on the transmitting side (transmitting device 15) so as to satisfy the following conditions, decoding with low delay in the receiving device 20 can be guaranteed.

The conditions are as follows.

The transmitting device 15 transmits the sample data so that the receiving device 20 can always receive the MF meta including information about the sample (Sample(i)) before the decoding time (DTS(i)) of the sample. MF meta is generated and transmitted using the divided units as movie fragments.

Specifically, the transmitting device 15 constructs a movie fragment using samples encoded before DTS(i) (including the i-th sample).

As a method for dynamically determining the number of samples making up a movie fragment and the number of movie fragments making up one MPU so as to guarantee low-delay decoding, for example, the following method is used.

(1) At the start of decoding, the decoding time DTS(0) of the first sample of the GOP, Sample(0), is a time based on initial_cpb_removal_delay. The transmitting device constructs a first movie fragment using encoded samples at a time before DTS(0). Furthermore, the transmitting device 15 generates MF metadata corresponding to the first movie fragment and transmits it at a time before DTS(0).

(2) The transmitting device 15 configures movie fragments in subsequent samples as well so as to satisfy the above conditions.

For example, when the first sample of a movie fragment is the k-th sample, the MF meta of the movie fragment including the k-th sample is transmitted by the decoding time DTS(k) of the k-th sample. If the encoding completion time of the l-th sample is before DTS(k) and the encoding completion time of the (l+1)th sample is after DTS(k), the transmitting device 15 transmits the A movie fragment is constructed using the lth sample from the samples in .

Note that the transmitting device 15 may configure a movie fragment using samples from the k-th sample to samples less than the l-th sample.

(3) After the encoding of the last MPU sample is completed, the transmitting device 15 configures a movie fragment using the remaining samples, generates MF metadata corresponding to the movie fragment, and transmits it.

Note that the transmitting device 15 may not configure a movie fragment using all encoded samples, but may configure a movie fragment using some encoded samples.

Note that the above example shows an example in which the number of samples forming a movie fragment and the number of movie fragments forming one MPU are dynamically determined based on the above conditions to ensure low-delay decoding. Ta. However, the method for determining the number of samples and the number of movie fragments is not limited to this method. For example, the number of movie fragments constituting one MPU may be fixed to a predetermined value, and the number of samples may be determined to satisfy the above conditions. Furthermore, the number of movie fragments constituting one MPU and the time at which the movie fragments are divided (or the code amount of the movie fragments) may be fixed to predetermined values, and the number of samples may be determined so as to satisfy the above conditions.

In addition, if the MPU is divided into multiple movie fragments, information indicating whether the MPU is divided into multiple movie fragments, attributes of the divided movie fragments, or attributes of the MF meta for the divided movie fragments. may be sent.

Here, the attribute of a movie fragment is information indicating whether the movie fragment is the first movie fragment of the MPU, the last movie fragment of the MPU, or any other movie fragment.

In addition, the attributes of the MF meta include whether the MF meta corresponds to the first movie fragment of the MPU, the MF meta that corresponds to the last movie fragment of the MPU, or the MF meta corresponds to other movie fragments. This is information indicating whether the MF meta is used or not.

Note that the transmitting device 15 may store and transmit the number of samples making up a movie fragment and the number of movie fragments making up one MPU as control information.

[Operation of receiving device]
The operation of the receiving device 20 based on the movie fragment configured as described above will be explained.

The receiving device 20 determines the absolute time of each of the PTS and DTS based on the absolute time signaled from the transmitting side, such as the MPU time stamp descriptor, and the relative time of the PTS and DTS included in the MF meta.

The receiving device 20 performs the following processing based on the information as to whether the MPU is divided into a plurality of movie fragments, and, if the MPU is divided, based on the attributes of the divided movie fragments. .

(1) If the movie fragment is the first movie fragment of the MPU, the receiving device 20 determines the absolute time of the PTS of the first sample included in the MPU timestamp descriptor and the relative time of the PTS and DTS included in the MF meta. to generate the absolute time of PTS and DTS.

(2) If the movie fragment is not the first movie fragment of the MPU, the receiving device 20 uses the relative time of the PTS and DTS included in the MF meta, without using the information of the MPU timestamp descriptor, to generates an absolute time.

(3) If the movie fragment is the last movie fragment of the MPU, the receiving device 20 resets the PTS and DTS calculation process (relative time addition process) after calculating the absolute time of the PTS and DTS of all samples. do. Note that the reset process may be performed in the movie fragment at the beginning of the MPU.

The receiving device 20 may determine whether the movie fragment is divided as described below. Furthermore, the receiving device 20 may acquire attribute information of movie fragments as described below.

For example, the receiving device 20 may determine whether the movie fragments are divided based on the identifier movie_fragment_sequence_number field value indicating the order of movie fragments indicated in the MMTP (MMT Protocol) payload header.

Specifically, when the number of movie fragments included in one MPU is 1, the movie_fragment_sequence_number field value is 1, and the field value has a value of 2 or more, , it may be determined that the MPU is divided into multiple movie fragments.

Furthermore, when the number of movie fragments included in one MPU is 1, the movie_fragment_sequence_number field value is 0, and the field value has a value other than 0, the receiving device 20 may be determined to be divided into multiple movie fragments.

The attribute information of the movie fragment may also be determined based on the movie_fragment_sequence_number.

Note that without using the movie_freagment_sequence_number, whether or not a movie fragment is divided and the attribute information of the movie fragment may be determined by counting the transmission of movie fragments and MF metas included in one MPU.

With the configuration of the transmitting device 15 and receiving device 20 as described above, the receiving device 20 can receive movie fragment metadata at shorter intervals than the MPU, and can start decoding with low delay. Further, by using decoding processing based on the MP4 parsing method, it is possible to perform decoding with low delay.

The reception operation when the MPU is divided into a plurality of movie fragments as described above will be explained using a flowchart. FIG. 41 is a flowchart of the reception operation when the MPU is divided into a plurality of movie fragments. Note that this flowchart illustrates the operation of step S604 in FIG. 37 in more detail.

First, the receiving device 20 acquires MF metadata based on the data type indicated in the MMTP payload header when the data type is MF meta (S801).

Next, the receiving device 20 determines whether the MPU is divided into a plurality of movie fragments (S802), and if the MPU is divided into a plurality of movie fragments (Yes in S802), the received MF It is determined whether the metadata is the first metadata of the MPU (S803). If the received MF metadata is the MF metadata at the beginning of the MPU (Yes in S803), the receiving device 20 sets the absolute time of the PTS indicated in the MPU timestamp descriptor, and the PTS and MF metadata indicated in the MF metadata. The absolute time of PTS and DTS is calculated from the relative time of DTS (S804), and it is determined whether it is the last metadata of the MPU (S805).

On the other hand, if the received MF metadata is not the MF metadata at the beginning of the MPU (No in S803), the receiving device 20 does not use the information of the MPU timestamp descriptor and uses the relative information of the PTS and DTS indicated in the MF metadata. The absolute time of PTS and DTS is calculated using the time (S808), and the process moves to step S805.

If it is determined in step S805 that it is the last MF metadata of the MPU (Yes in S805), the receiving device 20 resets the PTS and DTS calculation process after calculating the absolute time of the PTS and DTS of all samples. do. If it is determined in step S805 that the MF metadata is not the last MF metadata of the MPU (No in S805), the receiving device 20 ends the process.

Further, if it is determined in step S802 that the MPU is not divided into multiple movie fragments (No in S802), the receiving device 20 acquires sample data based on the MF metadata transmitted after the MPU. Then, PTS and DTS are determined (S807).

Although not shown, the receiving device 20 finally performs decoding processing and presentation processing based on the determined PTS and DTS.

[Issues that occur when dividing movie fragments and their solutions]
So far, we have described how to reduce end-to-end delay by dividing movie fragments. From now on, we will explain new problems that arise when movie fragments are divided, and solutions to them.

First, as background, the picture structure in encoded data will be explained. FIG. 42 is a diagram illustrating an example of a predicted structure of a picture in each TemporalId when achieving temporal scalability.

In coding methods such as MPEG-4 AVC and HEVC (High Efficiency Video Coding), temporal scalability is achieved by using B pictures (bidirectional reference predicted pictures) that can be referenced from other pictures. can.

TemporalId shown in (a) of FIG. 42 is an identifier of the hierarchy of the encoding structure, and the larger the value of TemporalId, the deeper the hierarchy. A square block indicates a picture, Ix in the block is an I picture (intra-screen predicted picture), Px is a P picture (forward reference predicted picture), and Bx and bx are B pictures (bidirectional reference predicted picture). show. The x in Ix/Px/Bx indicates a display order and indicates the order in which pictures are displayed. Arrows between pictures indicate reference relationships; for example, picture B4 indicates that a predicted image is generated using I0 and B8 as reference images. Here, one picture is prohibited from using another picture whose TemporalId is larger than its own TemporalId as a reference image. The reason why the layers are specified is to provide temporal scalability. For example, in FIG. 42, when all pictures are decoded, a video of 120 fps (frame per second) is obtained, but only the layers with TemporalId from 0 to 3 When decoded, a 60fps video is obtained.

FIG. 43 is a diagram showing the relationship between decoding time (DTS) and display time (PTS) in each picture in FIG. 42. For example, picture I0 shown in FIG. 43 is displayed after the decoding of B4 is completed so that no gap occurs in decoding and display.

As shown in FIG. 43, when the prediction structure includes a B picture, the decoding order and the display order are different, so the receiving device 20 performs picture delay processing and rearranges the pictures after decoding them. (Reorder) processing is required.

Examples of picture prediction structures with temporal scalability have been described above, but even when temporal scalability is not used, picture delay processing and reorder processing may be necessary depending on the prediction structure. be. FIG. 44 is a diagram illustrating an example of a predicted structure of a picture that requires picture delay processing and reorder processing. Note that the numbers in FIG. 44 indicate the decoding order.

As shown in FIG. 44, depending on the prediction structure, the first sample in decoding order may be different from the first sample in presentation order. In FIG. 44, the first sample in presentation order is different from the first sample in decoding order. This is the fourth sample. Note that FIG. 44 shows an example of a prediction structure, and the prediction structure is not limited to such a structure. In other prediction structures, the first sample in decoding order and the first sample in presentation order may be different.

Similar to FIG. 33, FIG. 45 is a diagram showing an example in which an MPU configured in MP4 format is divided into a plurality of movie fragments and stored in MMTP payloads and MMTP packets. Note that the number of samples configuring an MPU and the number of samples configuring a movie fragment are arbitrary. For example, two movie fragments may be configured, with the number of samples configuring the MPU being the number of samples per GOP, and the number of samples being half the number of samples per GOP being the movie fragment. One sample may be one movie fragment, and the samples making up the MPU may not be divided.

Although FIG. 45 shows an example in which one MPU includes two movie fragments (moof box and mdat box), one MPU does not need to include two movie fragments. The number of movie fragments included in one MPU may be three or more, or may be the number of samples included in the MPU. Furthermore, the samples stored in a movie fragment may not be divided into equal numbers of samples, but may be divided into an arbitrary number of samples.

The movie fragment metadata (MF metadata) includes information on the PTS, DTS, offset, and size of the sample included in the movie fragment, and when the receiving device 20 decodes the sample, the receiving device 20 The PTS and DTS are extracted from the MF meta that includes information, and the decoding timing and presentation timing are determined.

From here on, for detailed explanation, the absolute value of the decoding time of the i sample will be written as DTS(i), and the absolute value of the presentation time will be written as PTS(i).

Specifically, the information of the i-th sample among the timestamp information stored in the moof in MF meta is the relative value of the decoding time of the i-th sample and the (i+1)-th sample, and the i-th These are the relative values of the decoding time and presentation time of the sample, and these are hereinafter referred to as DT(i) and CT(i).

Movie fragment metadata #1 includes DT(i) and CT(i) of samples #1-#3, and movie fragment metadata #2 includes DT(i) of samples #4-#6. ) and CT(i).

Further, the absolute value of PTS of the access unit at the beginning of the MPU is stored in the MPU time stamp descriptor, etc., and the receiving device 20 calculates the PTS and DTS based on the PTS_MPU of the access unit at the beginning of the MPU, and the CT and DT. calculate.

FIG. 46 is a diagram for explaining a method of calculating PTS and DTS and problems when an MPU is configured by samples #1 to #10.

(a) of FIG. 46 shows an example in which the MPU is not divided into movie fragments, (b) in FIG. 46 shows an example in which the MPU is divided into two movie fragments of 5 samples each, and (c) in FIG. ) shows an example in which the MPU is divided into 10 movie fragments in units of samples.

As explained in FIG. 45, when PTS and DTS are calculated using the MPU timestamp descriptor and the timestamp information (CT and DT) in MP4, the first sample in the presentation order in FIG. is the fourth in the decoding order. Therefore, the PTS stored in the MPU timestamp descriptor is the PTS (absolute value) of the fourth sample in the decoding order. Note that, hereinafter, this sample will be referred to as the A sample. Further, the first sample in decoding order is called a B sample.

Since the absolute time information related to the time stamp is only the information of the MPU time stamp descriptor, the receiving device 20 calculates the PTS (absolute time) and DTS (absolute time) of the other samples until the A sample arrives. Can not. The receiving device 20 also cannot calculate the PTS and DTS of the B sample.

In the example of FIG. 46(a), the A sample is included in the same movie fragment as the B sample, and is stored in one MF meta. Therefore, the receiving device 20 can determine the DTS of the B sample immediately after receiving the MF meta.

In the example of FIG. 46(b), the A sample is included in the same movie fragment as the B sample, and is stored in one MF meta. Therefore, the receiving device 20 can determine the DTS of the B sample immediately after receiving the MF meta.

In the example of FIG. 46(c), the A sample is included in a different movie fragment from the B sample. Therefore, the receiving device 20 cannot determine the DTS of the B sample until after receiving the MF meta including the CT and DT of the movie fragment including the A sample.

Therefore, in the example of FIG. 46(c), the receiving device 20 cannot start decoding immediately after the B sample arrives.

In this way, if a movie fragment including a B sample does not include an A sample, the receiving device 20 decodes the B sample only after receiving the MF meta related to the movie fragment including the A sample. Unable to start.

This problem occurs when the first sample in the presentation order and the first sample in the decoding order do not match, and the movie fragment is divided until the A sample and B sample are no longer stored in the same movie fragment. do. Moreover, this problem occurs regardless of whether the MF meta is postponed or postponed.

In this way, if the first sample in the presentation order and the first sample in the decoding order do not match, and if the A sample and the B sample are not stored in the same movie fragment, the DTS cannot be determined. Therefore, the transmitting device 15 separately transmits the DTS (absolute value) of the B samples or information that allows the receiving side to calculate the DTS (absolute value) of the B samples. Such information may be transmitted using control information, a packet header, or the like.

The receiving device 20 uses such information to calculate the DTS (absolute value) of the B samples. FIG. 47 is a flowchart of the reception operation when the DTS is calculated using such information.

The receiving device 20 receives the movie fragment at the beginning of the MPU (S901), and determines whether the A sample and the B sample are stored in the same movie fragment (S902). If they are stored in the same movie fragment (Yes in S902), the receiving device 20 calculates the DTS using only the MF meta information without using the DTS (absolute time) of the B sample, and starts decoding. (S904). Note that in step S904, the receiving device 20 may determine the DTS using the DTS of B samples.

On the other hand, if the A sample and the B sample are not stored in the same movie fragment in step S902 (No in S902), the receiving device 20 obtains the DTS (absolute time) of the B sample, determines the DTS, and decodes the sample. (S903).

Note that in the above explanation, the absolute value of the decoding time and the absolute value of the presentation time of each sample is calculated using MF meta (time stamp information stored in MP4 format moof) in the MMT standard. Although an example has been described, it goes without saying that the MF meta may be replaced with any control information that can be used to calculate the absolute value of the decoding time and the absolute value of the presentation time of each sample. An example of such control information is the relative value CT(i) of the decoding times of the i-th sample and the (i+1)-th sample described above, and the relative value CT(i) of the decoding times of the i-th sample and the (i+1)-th sample. The control information replaced with relative values, the relative value CT(i) of the decoding time of the i-th sample and the (i+1)-th sample, and the relative value of the presentation time of the i-th sample and the (i+1)-th sample. There is control information that includes both.

(Embodiment 3)
[overview]
In Embodiment 3, a content transmission method and data structure when transmitting content such as video, audio, subtitles, and data broadcasting by broadcasting will be described. That is, a content transmission method and data structure specialized for playing a broadcast stream will be explained.

Note that in Embodiment 3, an example will be described in which the MMT method (hereinafter also simply referred to as MMT) is used as the multiplexing method, but other multiplexing methods such as MPEG-DASH or RTP may also be used. .

First, details of a method for storing a data unit (DU: Data Unit) in a payload in MMT will be explained. FIG. 48 is a diagram for explaining a method of storing a data unit in a payload in MMT.

In MMT, a transmitting device stores a part of data that constitutes an MPU in an MMTP payload as a data unit, attaches a header, and transmits the data. The header includes an MMTP payload header and an MMTP packet header. Note that the data unit may be a NAL unit or a sample.

FIG. 48(a) shows an example in which a transmitting device aggregates a plurality of data units and stores them in one payload. In the example of FIG. 48(a), a data unit header (DUH) and a data unit length (DUL) are added to the beginning of each of a plurality of data units, and the data unit header and A plurality of data units to which a data unit length is assigned are collectively stored in the payload.

FIG. 48(b) shows an example in which one data unit is stored in one payload. In the example of FIG. 48(b), a data unit header is added to the beginning of the data unit and stored in the payload. FIG. 48C shows an example in which one data unit is divided, a data unit header is added to the divided data unit, and the data unit header is stored in the payload.

Data units include timed-MFU, which is media that includes information related to synchronization such as video, audio, or subtitles, non-timed-MFU, which is media that does not include information related to synchronization, such as files, MPU metadata, MF metadata, etc. There are several types, and the data unit header is determined according to the type of data unit. Note that there is no data unit header in the MPU metadata and MF metadata.

Further, although the transmitting device cannot in principle aggregate different types of data units, it may be specified to be able to aggregate different types of data units. For example, if the size of MF metadata is small, such as when each sample is divided into movie fragments, the number of packets can be reduced by aggregating MF metadata and media data, and furthermore, the transmission capacity can be reduced. You can also do that.

When the data unit is an MFU, some information of the MPU, such as information for configuring the MPU (MP4), is stored as a header.

For example, the header of a timed-MFU includes movie_fragment_sequence_number, sample_number, offset, priority, dependency_counter, etc., and the header of a non-timed-MFU includes item_iD. The meaning of each field is shown in standards such as ISO/IEC23008-1 or ARIB STD-B60. The meaning of each field defined in such standards will be explained below.

movie_fragment_sequence_number indicates the sequence number of the movie fragment to which the MFU belongs, and is also indicated in ISO/IEC14496-12.

sample_number indicates the sample number to which the MFU belongs, and is also indicated in ISO/IEC14496-12.

offset indicates the offset amount of the MFU in the sample to which the MFU belongs, in bytes.

The priority indicates the relative importance of the MFU in the MPU to which the MFU belongs, and an MFU with a large priority number indicates that it is more important than an MFU with a small priority number.

dependency_counter indicates the number of MFUs on which the decoding process depends on the MFU (that is, the number of MFUs for which the decoding process cannot be performed unless this MFU is decoded). For example, when the MFU is HEVC and a B picture or a P picture references an I picture, the B picture or P picture cannot be decoded unless the I picture is decoded.

Therefore, when the MFU is in units of samples, the dependency_counter in the MFU of an I picture indicates the number of pictures that refer to the I picture. If the MFU is in units of NAL units, dependency_counter in the MFU belonging to an I picture indicates the number of NAL units belonging to a picture that refers to the I picture. Furthermore, in the case of a temporally hierarchically encoded video signal, the MFU of the enhancement layer depends on the MFU of the base layer, so the dependency_counter in the MFU of the base layer indicates the number of MFUs of the enhancement layer. This field can only be generated after the number of dependent MFUs has been determined.

item_id indicates an identifier that uniquely identifies an item.

[MP4 non-support mode]
As explained in FIGS. 19 and 21, methods for the transmitting device to transmit MPU in MMT include transmitting MPU metadata or MF metadata before or after media data, and transmitting only media data. There is a way to send it. Further, in the receiving device, there are a method of decoding using a receiving device and a receiving method compliant with MP4, and a method of decoding without using a header.

As a data transmission method specialized for broadcast stream playback, there is, for example, a transmission method that does not support MP4 reconfiguration in a receiving device.

A transmission method that does not support MP4 reconfiguration in the receiving device is, for example, a method in which metadata (MPU metadata and MF metadata) is not transmitted, as shown in FIG. 21(b). In this case, the field value of the fragment type (information indicating the type of data unit) included in the MMTP packet is fixed to 2 (=MFU).

If metadata is not sent, as explained above, an MP4-compliant receiving device cannot decode the received data as MP4, but it is possible to decode it without using metadata (header). It is.

Therefore, metadata is not necessarily essential information for broadcast stream decoding and playback. Similarly, the information in the data unit header in the timed-MFU described with reference to FIG. 48 is information for reconfiguring the MP4 in the receiving device. Since there is no need to reconstruct MP4 in broadcast stream playback, the information in the data unit header (hereinafter also referred to as timed-MFU header) in the timed-MFU is not necessarily information necessary for broadcast stream playback.

The receiving device can easily reconstruct MP4 by using metadata and information for reconstructing MP4 in the data unit header (hereinafter also referred to as MP4 configuration information). However, the receiving device cannot reconstruct the MP4 even if only one of the metadata and the MP4 configuration information in the data unit header is transmitted. There is little merit in transmitting only either the metadata or the information for reconstructing the MP4, and generating and transmitting unnecessary information results in an increase in processing and a decrease in transmission efficiency.

Therefore, the transmitter uses the method described below to control the data structure and transmission of the MP4 configuration information. The transmitting device determines whether to indicate MP4 configuration information in the data unit header based on whether metadata is transmitted. Specifically, the transmitting device indicates MP4 configuration information in the data unit header when metadata is transmitted, and does not indicate MP4 configuration information in the data unit header when metadata is not transmitted.

As a method of not indicating MP4 configuration information in the data unit header, for example, the following method can be used.

1. The transmitting device sets the MP4 configuration information to reserved and does not operate it. This makes it possible to reduce the amount of processing on the sending side that generates MP4 configuration information (the amount of processing on the transmitting device).

2. The transmitting device deletes the MP4 configuration information and compresses the header. Thereby, it is possible to reduce the amount of processing on the sending side that generates MP4 configuration information, and it is also possible to reduce the transmission capacity.

Note that, when deleting MP4 configuration information and compressing the header, the transmitting device may display a flag indicating that the MP4 configuration information has been deleted (compressed). The flag is indicated in a header (MMTP packet header, MMTP payload header, data unit header), control information, or the like.

Further, information as to whether or not metadata is to be transmitted may be determined in advance, or may be separately signaled in a header or control information and transmitted to the receiving device.

For example, information as to whether metadata corresponding to the MFU is being transmitted may be stored in the MFU header.

On the other hand, the receiving device can determine whether MP4 configuration information is indicated based on whether metadata is being transmitted.

Here, if the data transmission order is fixed (e.g., MPU metadata, MF metadata, media data, etc.), the receiving device can transmit the data based on whether the metadata is received before the media data. It may be determined by

If MP4 configuration information is indicated, the receiving device can use the MP4 configuration information for MP4 reconfiguration. Alternatively, the receiving device can use the MP4 configuration information to detect the beginning of other access units or NAL units.

Note that the MP4 configuration information may be all or a part of the timed-MFU header.

Similarly, the transmitting device selects an item in the non-timed-MFU header based on whether or not metadata is transmitted in the non-timed-MFU header.
It may also be determined whether to indicate the id.

The transmitting device may indicate MP4 configuration information only in either the timed-MFU or the non-timed-MFU. If MP4 configuration information is to be shown to only one of the MFUs, the transmitting device determines whether to show the MP4 configuration information based on whether the MFU is a timed-MFU or a non-timed-MFU, in addition to whether metadata is transmitted. do. The receiving device can determine whether metadata is transmitted and whether MP4 configuration information is indicated based on the timed/non-timed flag.

Note that in the above description, the transmitting device determines whether to indicate MP4 configuration information based on whether metadata (both MPU metadata and MF metadata) is transmitted. However, the transmitting device may not indicate the MP4 configuration information when part of the metadata (either MPU metadata or MF metadata) is not transmitted.

Furthermore, the transmitting device may determine whether to indicate MP4 configuration information based on information other than metadata.

For example, modes such as MP4 support mode/MP4 non-support mode are defined, and the transmitting device indicates the MP4 configuration information in the data unit header in case of MP4 support mode and the data unit header in case of MP4 non-support mode. MP4 configuration information may not be indicated in the unit header. Further, in the case of MP4 support mode, the transmitting device transmits metadata and indicates MP4 configuration information in the data unit header, and in the case of MP4 non-support mode, the transmitting device transmits metadata and indicates MP4 configuration information in the data unit header without transmitting metadata. Also, the MP4 configuration information may not be shown.

[Operation flow of transmitter]
Next, the operational flow of the transmitter will be explained. FIG. 49 is an operation flow of the transmitting device.

The transmitting device first determines whether to transmit metadata (S1001). When the transmitting device determines to transmit metadata (Yes in S1002), the transmitting device moves to step S1003, generates MP4 configuration information, stores it in a header, and transmits it (S1003). In this case, the transmitting device also generates and transmits metadata.

On the other hand, if the transmitting device determines not to transmit metadata (No in S1002), it transmits the MP4 configuration information without generating it or storing it in the header (S1004). In this case, the transmitting device does not generate or transmit metadata.

Note that whether or not to transmit metadata in step S1001 may be determined in advance, or may depend on whether metadata is generated within the transmitting device or whether metadata is transmitted within the transmitting device. The determination may be made based on

[Operation flow of receiving device]
Next, the operation flow of the receiving device will be explained. FIG. 50 is an operation flow of the receiving device.

The receiving device first determines whether metadata is being transmitted (S1101). Whether metadata is being transmitted can be determined by monitoring the fragment type in the MMTP packet payload. Further, whether or not the information is being transmitted may be determined in advance.

If the receiving device determines that metadata is being transmitted (Yes in S1102), it reconfigures the MP4 and executes decoding processing using the MP4 configuration information (S1103). On the other hand, if it is determined that metadata has not been transmitted (No in S1102), the MP4 reconstruction process is not performed and the decoding process is executed without using the MP4 configuration information (S1104).

Note that the receiving device can detect a random access point, detect the beginning of an access unit, detect the beginning of a NAL unit, etc. without using the MP4 configuration information using the method described above, and can perform decoding processing. , detecting packet loss, and processing recovery from packet loss.

For example, the beginning of the access unit is the beginning data of the MMT payload whose aggregation_flag value is 1. At this time, the Fragmentation_indicator value is 0.

Further, the beginning of the slice segment is the beginning data of the MMT payload whose aggregation_flag value is 0 and the fragmentation_indicator value is 00 or 01.

The receiving device can detect the beginning of the access unit and the slice segment based on the above information.

Note that the receiving device analyzes the NAL unit header in a packet containing the beginning of a data unit whose fragmentation_indicator value is 00 or 01, and determines that the type of the NAL unit is an AU delimiter and that the type of the NAL unit is a slice segment. It may be detected that

[Broadcast simple mode]
So far, we have explained a method that does not support MP4 configuration information in the receiving device as a data transmission method specialized for broadcast stream playback, but this is not the only method for transmitting data specialized for broadcast stream playback. do not have.

For example, the following method may be used as a data transmission method specialized for broadcast stream playback.

- The transmitter does not need to use AL-FEC in a fixed broadcast reception environment. When AL-FEC is not used, FEC_type in the MMTP packet header is always fixed to 0.

- The transmitter may always use AL-FEC in a broadcast mobile reception environment and in communication UDP transmission mode. When AL-FEC is used, FEC_type in the MMTP packet header is always 0 or 1.

- The transmitting device does not need to perform bulk transmission of assets. If bulk transmission of assets is not performed, location_infolocation indicating the number of transmission locations of assets within the MPT may be fixed to 1.

- The sending device does not have to perform hybrid transmission of assets, programs, and messages.

Further, for example, if the broadcast simple mode is defined and the transmitter is in the broadcast simple mode, the transmitting device may set the MP4 non-support mode, or use the data transmission method specialized for broadcast stream playback as described above. You can also use it as Whether or not it is broadcast simple mode may be determined in advance, or the transmitting device may store a flag indicating that it is broadcast simple mode as control information and transmit it to the receiving device.

Furthermore, based on whether or not the transmitting device is in the MP4 non-support mode (whether metadata is being transmitted or not), as explained in FIG. 49, if the transmitting device is in the MP4 non-support mode, The data transmission method specialized for broadcast stream playback shown in 2 may also be used.

When the receiving device is in the broadcast simple mode, it is assumed that the receiving device is in an MP4 non-support mode and can perform decoding processing without reconstructing the MP4.

Furthermore, when the receiving device is in the broadcast simple mode, it can determine that the function is specialized for broadcasting, and perform reception processing specialized for broadcasting.

As a result, in the case of broadcast simple mode, by using only broadcast-specific functions, it is possible to not only reduce unnecessary processing for the transmitting device and receiving device, but also to avoid compressing and transmitting unnecessary information. Therefore, transmission overhead can be reduced.

Note that when the MP4 non-support mode is used, hint information that supports a storage method other than the MP4 configuration may be shown.

Examples of storage methods other than the MP4 configuration include a method of directly storing MMT packets and IP packets, and a method of converting MMT packets into MPEG-2 TS packets.

Note that in the case of the MP4 non-support mode, a format that does not follow the MP4 configuration may be used.

For example, in the case of an MP4 non-support mode, the data stored in the MFU is not in the MP4 format with the NAL unit size appended to the beginning of the NAL unit, but in the format with a byte start code attached. good.

In MMT, the asset type indicating the type of asset is described in 4CC registered in MP4REG (http://www.mp4ra.org), and when HEVC is used as the video signal, 'HEV1' or 'HVC1' is used. It will be done. 'HVC1' is a format that may include a parameter set in the sample, and 'HEV1' is a format that does not include a parameter set in the sample but includes a parameter set in the sample entry in the MPU metadata.

In the case of broadcast simple mode or MP4 non-support mode, if MPU metadata and MF metadata are not transmitted, it may be specified that the parameter set is always included in the sample. Furthermore, it may be specified that the format is always 'HVC1' regardless of whether 'HEV1' or 'HVC1' is indicated as the asset type.

[Supplement 1: Transmitting device]
As described above, when metadata is not transmitted, the MP4 configuration information is set to reserved, and the transmitting device that is not operated can be configured as shown in FIG. 51. FIG. 51 is a diagram showing an example of a specific configuration of a transmitting device.

The transmitting device 300 includes an encoding section 301, an adding section 302, and a transmitting section 303. Each of the encoding section 301, the adding section 302, and the transmitting section 303 is realized by, for example, a microcomputer, a processor, a dedicated circuit, or the like.

The encoding unit 301 encodes a video signal or an audio signal to generate sample data. Specifically, the sample data is a data unit.

The adding unit 302 adds header information including MP4 configuration information to sample data, which is data obtained by encoding a video signal or an audio signal. The MP4 configuration information is information for reconstructing the sample data as an MP4 format file on the receiving side, and the content differs depending on whether or not the presentation time of the sample data is determined.

As described above, the adding unit 302 adds movie_fragment_sequence_number, sample_number, offset, priority to the header (header information) of timed-MFU, which is an example of sample data (sample data including information regarding synchronization) for which the presentation time is determined. , and MP4 configuration information such as dependency_counter.

On the other hand, the adding unit 302 adds MP4 configuration information such as item_id to the header (header information) of timed-MFU, which is an example of sample data for which the presentation time is not determined (sample data that does not include information regarding synchronization). Include.

Then, if the transmitting unit 303 does not transmit metadata corresponding to the sample data (for example, in a case like (b) in FIG. 21), the assigning unit 302 determines whether the presentation time of the sample data has been determined Depending on the situation, header information that does not include MP4 configuration information is added to the sample data.

Specifically, when the presentation time of the sample data is determined, the adding unit 302 adds header information that does not include the first MP4 configuration information to the sample data, and when the presentation time of the sample data is determined. If not, header information including second MP4 configuration information is added to the sample data.

For example, as shown in step S1004 in FIG. 49, when the transmitting unit 303 does not transmit the metadata corresponding to the sample data, the providing unit 302 sets the MP4 configuration information to reserved (fixed value). MP4 configuration information is not substantially generated and is not substantially stored in the header (header information). Note that the metadata includes MPU metadata and movie fragment metadata.

The transmitter 303 transmits sample data to which header information is added. More specifically, the transmitter 303 packetizes the sample data to which header information has been added using the MMT method and transmits the packet.

As described above, in the transmission method and reception method specialized for playing back broadcast streams, there is no need for the receiving device to reconfigure the data units into MP4. If the receiving device does not need to reconfigure to MP4, the processing of the transmitting device is reduced by not generating unnecessary information such as MP4 configuration information.

On the other hand, although the transmitting device must transmit the necessary information, it must maintain consistency with the standard so that it does not have to separately transmit unnecessary additional information.

According to the configuration of transmitting device 300, by setting the area where MP4 configuration information is stored to a fixed value, MP4 configuration information is not transmitted, only necessary information is transmitted based on the standard, and unnecessary information is transmitted. This has the effect of not having to send additional information. In other words, the configuration of the transmitting device and the amount of processing of the transmitting device can be reduced. Furthermore, since unnecessary data is not transmitted, transmission efficiency can be improved.

[Supplement 2: Receiving device]
Further, the receiving device corresponding to the transmitting device 300 may be configured as shown in FIG. 52, for example. FIG. 52 is a diagram showing another example of the configuration of the receiving device.

Receiving device 400 includes a receiving section 401 and a decoding section 402. The receiving section 401 and the decoding section 402 are realized by, for example, a microcomputer, a processor, a dedicated circuit, or the like.

The receiving unit 401 receives sample data, which is data obtained by encoding a video signal or an audio signal, to which header information including MP4 configuration information for reconstructing the sample data as an MP4 format file is added. Receive data.

The decoding unit 402 decodes the sample data without using MP4 configuration information when the metadata corresponding to the sample data is not received by the receiving unit and the presentation time of the sample data is determined. do.

For example, as shown in step S1104 in FIG. 50, if metadata corresponding to the sample data is not received by the receiving unit 401, the decoding unit 402 executes the decoding process without using the MP4 configuration information.

Thereby, the configuration of the receiving device 400 and the amount of processing in the receiving device 400 can be reduced.

(Embodiment 4)
[overview]
In the fourth embodiment, a method for storing non-timed media such as files that do not include information regarding synchronization in the MPU and a method for transmitting them using MMTP packets will be described. In the fourth embodiment, an MPU in MMT will be described as an example, but it is also applicable to DASH, which is also based on MP4.

First, the details of the method for storing non-timed media (hereinafter also referred to as "asynchronous media data") in the MPU will be explained using FIG. 53. FIG. 53 is a diagram showing a method of storing non-timed media in the MPU and a method of transmitting it using MMTP packets.

The MPU that stores non-timed media is composed of boxes such as ftyp, mmpu, moov, and meta, and stores information regarding files to be stored in the MPU. A plurality of idat boxes can be stored in the meta box, and one file is stored as an item in the idat box.

Part of the ftyp, mmpu, moov, and meta boxes constitute one data unit as MPU metadata, and the item or idat box constitutes a data unit as MFU.

After the data unit is aggregated or fragmented, a data unit header, an MMTP payload header, and an MMTP packet header are added to the data unit, and the data unit is transmitted as an MMTP packet.

Note that FIG. 53 shows an example in which File #1 and File #2 are stored in one MPU. Although the MPU metadata is not divided and the MFU is divided and stored in the MMTP packet, the present invention is not limited to this, and the data unit may be aggregated or fragmented depending on the size of the data unit. Also, the MPU metadata may not be transmitted, in which case only the MFU is transmitted.

Header information such as a data unit header indicates an itemID (an identifier that uniquely identifies an item), and the MMTP payload header and MMTP packet header include a packet sequence number (sequence number for each packet) and an MPU sequence number. (MPU sequence number, unique number within the asset).

Note that the data structure of the MMTP payload header and MMTP packet header other than the data unit header is the same as that of the timed media (hereinafter also referred to as "synchronous media data") described so far, and includes aggregation_flag, fragmentation_indicator, fragment_counter, etc. .

Next, a specific example of header information when dividing a file (=Item=MFU) into packets will be described using FIGS. 54 and 55.

FIGS. 54 and 55 are diagrams showing an example in which a plurality of pieces of divided data obtained by dividing a file are packetized and transmitted. Specifically, FIGS. 54 and 55 show information (packet sequence number, fragment counter, fragmentation indicator, MPU sequence number, item ID). Note that FIG. 54 is a diagram showing an example in which File #1 is divided into M pieces (M<=256), and FIG. 55 is a diagram showing an example in which File #2 is divided into N pieces (256<N). FIG.

The divided data number indicates the index of divided data from the beginning of the file, and this information is not transmitted. In other words, the divided data number is not included in the header information. In addition, the divided data number is a number assigned to a packet corresponding to each of a plurality of divided data obtained by dividing a file, and is a number that is added by 1 in ascending order starting from the first packet. be.

The packet sequence number is a sequence number of packets having the same packet ID, and in FIGS. 54 and 55, consecutive numbers are assigned from the first divided data of the file to A to the last divided data of the file. The packet sequence number is a number that is added by 1 in ascending order starting from the divided data at the beginning of the file, and is a number that corresponds to the divided data number.

The fragment counter indicates the number of pieces of divided data that come after the piece of divided data among the pieces of divided data obtained by dividing one file. Furthermore, when the number of divided data, which is the number of multiple pieces of divided data obtained by dividing one file, exceeds 256, the fragment counter indicates the remainder when the number of divided data is divided by 256. In the example of FIG. 54, since the number of divided data is 256 or less, the field value of the fragment counter is (M-divided data number). On the other hand, in the example of FIG. 55, the number of divided data exceeds 256, so the value is (N-divided data number) divided by 256 ((N-divided data number)%256).

Note that the fragmentation indicator indicates the state of division of data stored in the MMTP packet, and indicates whether it is the first divided data, the last divided data, or other divided data in the divided data unit. Alternatively, it is a value indicating whether the data unit is one or more undivided data units. Specifically, the fragmentation indicator is "01" for the first segmented data, "11" for the last segmented data, "10" for the remaining segmented data, and "10" for the unfragmented data unit. 00".

In this embodiment, when the number of divided data exceeds 256, the remainder after dividing the number of divided data by 256 will be described. However, the number of divided data is not limited to 256, and may be any other number (predetermined number) It's okay.

When a file is divided as shown in FIGS. 54 and 55, and conventional header information is added to each of a plurality of pieces of divided data obtained by dividing the file and transmitted, the receiving device receives a received MMTP packet. There is no information from which to derive the number of divided data in the original file (divided data number), the number of divided data of the file, or the divided data number and the number of divided data. Therefore, in the conventional transmission method, even if an MMTP packet is received, it is not possible to uniquely detect the divided data number or the number of divided data of the data stored in the received MMTP packet.

For example, if the number of divided data is 256 or less as shown in Figure 54, and it is known in advance that the number of divided data is 256 or less, the divided data number and the number of divided data can be determined by referring to the fragment counter. It is possible to specify. However, if the number of divided data is 256 or more, the divided data number or the number of divided data cannot be specified.

In addition, when limiting the number of divided data of a file to 256 or less, if the data size that can be transmitted in one packet is x [bytes], the maximum size of the file that can be transmitted is limited to x * 256 [bytes]. Ru. For example, in broadcasting, x=4k [bytes] is assumed, and in this case, the maximum size of a file that can be transmitted is limited to 4k*256=1M [bytes]. Therefore, when it is desired to transmit a file exceeding 1 [Mbytes], it is not possible to limit the number of divided data of the file to 256 or less.

Also, for example, by referring to the fragmentation indicator, it is possible to detect the beginning fragmented data and the last fragmented data of a file, so the number of MMTP packets is counted until the MMTP packet containing the last fragmented data of the file is received. It is possible to calculate the fragment data number and the number of fragment data by combining it with the packet sequence number after receiving the MMTP packet containing the last fragment data of the file. The divided data number and the number of divided data may be signaled by combining the numbers. However, if reception starts from an MMTP packet that includes divided data in the middle of a file (that is, divided data that is not the first divided data of the file or the last divided data of the file), the divided data number of the divided data and the divided data I can't determine the number. The divided data number and number of divided data of the divided data can be specified only after receiving the MMTP packet containing the last divided data of the file.

In order to solve the problem explained in FIGS. 54 and 55, that is, to uniquely determine the divided data number and the number of divided data of the divided data of a file when a packet containing divided data of a file is received midway, the following method is used. Use.

First, the divided data numbers will be explained.

As for the divided data number, the packet sequence number in the first divided data of the file (item) is signaled.

As a signaling method, it is stored in the control information that manages the file. Specifically, in FIGS. 54 and 55, the packet sequence number A of the divided data at the beginning of the file is stored in the control information. The receiving device obtains the value of A from the control information and calculates the divided data number from the packet sequence number indicated in the packet header.

The divided data number of the divided data is obtained by subtracting the packet sequence number A of the first divided data from the packet sequence number of the divided data.

As control information for managing files, for example, there is an asset management table defined in ARIB STD-B60. The asset management table shows file size, version information, etc. for each file, and is stored and transmitted in a data transmission message. FIG. 56 is a diagram showing the syntax of a loop for each file in the asset management table.

If the area of the existing asset management table cannot be expanded, signaling may be performed using a 32-bit area that is part of the item_info_byte field that indicates item information. A flag indicating whether a packet sequence number in the first divided data of the file (item) is indicated in a partial area of item_info_byte may be indicated in, for example, the reserved_future_use field of the control information.

When repeatedly transmitting a file such as a data carousel, a plurality of packet sequence numbers may be indicated, or the first packet sequence number of the file to be transmitted immediately after may be indicated.

The information is not limited to the packet sequence number of the divided data at the beginning of the file, but may be any information that links the divided data number of the file and the packet sequence number.

Next, the number of divided data will be explained.

The loop order for each file included in the asset management table may be defined as the file transmission order. This allows you to know the first packet sequence number of two consecutive files in the transmission order, so by subtracting the first packet sequence number of the file transmitted earlier from the first packet sequence number of the file transmitted later, It is possible to specify the number of divided data of the file to be transmitted. That is, for example, if File #1 shown in FIG. 54 and File #2 shown in FIG. 55 are consecutive files in this order, the last packet sequence number of File #1 and the first packet sequence of File #2 Consecutive numbers are assigned.

Further, by specifying the file division method, it may be specified that the number of data to be divided into a file can be specified. For example, if the number of divided data is N, the size of each of the 1st to (N-1)th divided data is L, and the size of the Nth divided data is a fraction (item_size-L*(N-1)). By specifying this, the number of divided data can be calculated backward from item_size shown in the asset management table. In this case, the integer value obtained by rounding up (item_size/L) becomes the number of divided data. Note that the file division method is not limited to this.

Alternatively, the number of divided data may be directly stored in the asset management table.

The receiving device receives the control information and calculates the number of divided data based on the control information by using the above method. Furthermore, the packet sequence number corresponding to the divided data number of the file can be calculated based on the control information. Note that if the timing of receiving the divided data packet is earlier than the timing of receiving the control information, the divided data number and the number of divided data may be calculated at the timing of receiving the control information.

Note that when signaling the divided data number or the number of divided data using the above method, the divided data number or the number of divided data is not specified based on the fragment counter, and the fragment counter becomes unnecessary data. Therefore, in the transmission of asynchronous media, if the fragment data number and information that can identify the number of fragment data are signaled using the above method, etc., the fragment counter may not be operated, or the header may be compressed. . Thereby, the processing amount of the transmitting device and the receiving device can be reduced, and transmission efficiency can also be improved. That is, when transmitting asynchronous media, the fragment counter may be reserved (invalidated). Specifically, the value of the fragment counter may be set to a fixed value of "0", for example. Additionally, the fragment counter may be ignored when receiving asynchronous media.

When storing synchronous media such as video and audio, the order of transmission of MMTP packets at the sending device and the order of arrival of MMTP packets at the receiving device match, and the packets are not retransmitted. In such a case, if there is no need to detect packet loss and reconfigure it, the fragment counter may not be operated. In other words, in this case, the fragment counter may be reserved (invalidated).

Furthermore, even without using a fragment counter, it is possible to detect random access points, detect the beginning of an access unit, detect the beginning of a NAL unit, etc., and perform decoding processing, detect packet loss, and recover from packet loss. can be processed.

In addition, in the transmission of real-time content such as live broadcasting, transmission with lower delay is required, and data that has been encoded must be sequentially packetized and transmitted. However, in real-time content transmission, conventional fragment counters cannot determine the number of fragmented data when sending the first fragmented data. After the number has been determined, there will be a delay. Even in such a case, this delay can be reduced by using the above method and not operating the fragment counter.

FIG. 57 is an operation flow for specifying a divided data number in a receiving device.

The receiving device acquires control information in which file information is described (S1201). The receiving device determines whether the packet sequence number at the beginning of the file is indicated in the control information (S1202), and if the packet sequence number at the beginning of the file is indicated in the control information (Yes at S1202), the receiving device The packet sequence number corresponding to the divided data number of the divided data is calculated (S1203). After acquiring the MMTP packet in which the divided data is stored, the receiving device identifies the divided data number of the file from the packet sequence number stored in the packet header of the acquired MMTP packet (S1204). On the other hand, if the packet sequence number at the beginning of the file is not indicated in the control information (No in S1202), the receiving device acquires the MMTP packet that includes the last divided data of the file, and then reads the packet header of the acquired MMTP packet. The divided data number is specified using the fragment indicator stored in the packet sequence number and the fragment indicator stored in the packet sequence number (S1205).

FIG. 58 is an operation flow for specifying the number of divided data in the receiving device.

The receiving device acquires control information in which file information is described (S1301). The receiving device determines whether the control information includes information that allows calculation of the number of data divisions of a file (S1302), and if it is determined that information that allows calculation of the number of division data of a file is included, (Yes in S1302), the number of divided data is calculated based on the information included in the control information (S1303). On the other hand, if the receiving device determines that the number of divided data cannot be calculated (No in S1302), after acquiring the MMTP packet that includes the last divided data of the file, the receiving device stores it in the packet header of the acquired MMTP packet. The number of divided data is specified using the fragment indicator and the packet sequence number (S1304).

FIG. 59 is an operational flow for determining whether to operate a fragment counter in a transmitting device.

First, the transmitting device determines whether the medium to be transmitted (hereinafter also referred to as "media data") is synchronous media or asynchronous media (S1401).

If the result of the determination in step S1401 is synchronous media (synchronous media in S1402), the transmitting device determines whether the MMTP packet order for transmission and reception matches in an environment for transmitting synchronous media, and whether packet reassembly is not required in the event of packet loss. It is determined whether or not (S1403). If the transmitting device determines that it is unnecessary (Yes in S1403), it does not operate the fragment counter (S1404). On the other hand, if the transmitting device determines that it is not necessary (No in S1403), it operates a fragment counter (S1405).

If the result of the determination in step S1401 is asynchronous media (asynchronous media in S1402), the transmitting device uses the fragment Decide whether to use the counter. Specifically, when the divided data number and the number of divided data are signaled (Yes in S1406), the transmitting device does not operate the fragment counter (S1404). On the other hand, if the divided data number or the number of divided data is not signaled (No in S1406), the transmitting device operates a fragment counter (S1405).

Note that when the transmitting device does not operate the fragment counter, the transmitting device may reserve the value of the fragment counter, or may perform header compression.

Note that the transmitting device may decide whether to signal the above-described divided data number or the number of divided data based on whether or not to operate a fragment counter.

Note that if the synchronous media does not operate a fragment counter, the transmitting device may signal the divided data number or the number of divided data using the method described above for the asynchronous media. Conversely, the operation of synchronous media may be determined based on whether or not asynchronous media operates a fragment counter. In this case, whether or not to use fragments can be the same for synchronous media and asynchronous media.

Next, a method for specifying the number of divided data and the divided data number (when using a fragment counter) will be explained. FIG. 60 is a diagram for explaining a method for specifying the number of divided data and the divided data number (when using a fragment counter).

As explained using FIG. 54, if the number of divided data is 256 or less, and it is known in advance that the number of divided data is 256 or less, by referring to the fragment counter, the divided data number and It is possible to specify the number of data.

When limiting the number of divided data of a file to 256 or less, if the data size that can be transmitted in one packet is x [bytes], the maximum size of a file that can be transmitted is limited to x*256 [bytes]. For example, in broadcasting, x=4k [bytes] is assumed, and in this case, the maximum size of a file that can be transmitted is limited to 4k*256=1M [bytes].

If the file size exceeds the maximum size of a file that can be transmitted, the file is divided in advance so that the size of the divided file becomes x*256 [bytes] or less. Each of the multiple divided files obtained by dividing the file is treated as one file (item), and is further divided into 256 or less pieces, and the divided data obtained by further division is converted into MMTP packets. stored and transmitted.

Note that information indicating that the item is a divided file, the number of divided files, and the sequence number of the divided files may be stored in the control information and transmitted to the receiving device. Further, this information may be stored in the asset management table, or may be indicated using a part of the existing field item_info_byte.

If the item is one of a plurality of split files obtained by splitting one file, the receiving device identifies the other split files and reconstructs the original file. I can do it. Furthermore, the receiving device can uniquely identify the number of divided data and the divided data number by using the number of divided files, the index of the divided file, and the fragment counter in the control information. Furthermore, the number of divided data and the divided data number can be uniquely specified without using a packet sequence number or the like.

Here, it is desirable that the item_id of each of the plurality of divided files obtained by dividing one file is the same. Note that when assigning another item_id, the item_id of the first divided file may be indicated in order to uniquely refer to the file from other control information or the like.

Further, a plurality of divided files may always belong to the same MPU. When storing a plurality of files in the MPU, files of different types may not be stored, but files obtained by dividing one file may always be stored. The receiving device can detect file updates by checking the version information of each MPU without checking the version information of each item.

FIG. 61 is an operation flow of the transmitting device when utilizing a fragment counter.

First, the transmitting device checks the size of the file to be transmitted (S1501). Next, the transmitting device determines whether the file size exceeds x*256 [bytes] (x is the data size that can be transmitted in one packet. For example, MTU size) (S1502), and determines whether the file size exceeds x If the size exceeds *256 [bytes] (Yes in S1502), the file is divided so that the size of the divided file becomes less than x*256 [bytes] (S1503). Then, the divided file is transmitted as an item, and information regarding the divided file (for example, the fact that it is a divided file, the sequence number of the divided file, etc.) is stored in the control information and transmitted (S1504). On the other hand, if the file size is less than x*256 [bytes] (No in S1502), the file is transmitted as an item as usual (S1505).

FIG. 62 is an operation flow of the receiving device when utilizing a fragment counter.

First, the receiving device acquires and analyzes control information regarding file transmission, such as an asset management table (S1601). Next, the receiving device determines whether the desired item is a split file (S1602). If the receiving device determines that the desired file is a divided file (Yes in S1602), the receiving device acquires information for reconfiguring the file, such as the divided file and the index of the divided file, from the control information (S1603). . The receiving device then obtains the items that make up the divided file and reconstructs the original file (S1604). On the other hand, if the receiving device determines that the desired file is not a divided file (No in S1602), it acquires the file as usual (S1605).

In short, the transmitting device signals the packet sequence number of the divided data at the beginning of the file. Further, the transmitting device signals information that can specify the number of divided data. Alternatively, the transmitting device defines a division rule that can specify the number of divided data. Further, the transmitting device performs reserved or header compression without operating a fragment counter.

If the packet sequence number of the data at the beginning of the file is signaled, the receiving device identifies the divided data number and the number of divided data from the packet sequence number of the divided data at the beginning of the file and the packet sequence number of the MMTP packet. .

From another perspective, the transmitting device divides a file, divides data for each divided file, and transmits the data. Signal information that links divided files (sequence number, number of divisions, etc.).

The receiving device identifies the divided data number and the number of divided data using the fragment counter and the sequence number of the divided file.

This makes it possible to uniquely identify the divided data number and divided data. Further, since the divided data number of the divided data in the middle can be specified at the time of receiving the divided data, the waiting time and memory can be reduced.

Furthermore, by not operating a fragment counter, the configuration of the transmitter/receiver can reduce the processing amount and improve transmission efficiency.

FIG. 63 is a diagram showing a service configuration when the same program is transmitted using multiple IP data flows. Here, part of the data (video/audio) of the program with service ID = 2 is transmitted by IP data flow using the MMT method, and data that has the same service ID and is different from the part of the data is sent to the advanced BS. An example is shown in which data is transmitted using an IP data flow using a data transmission method (in this example, the file transmission protocol is different, but may be the same protocol).

The transmitting device multiplexes the IP data so that it can be guaranteed that the data composed of multiple IP data flows will be completed by the decoding time in the receiving device.

The receiving device can realize guaranteed receiver operation by processing data made up of multiple IP data flows based on decoding time.

[Supplement: Transmitting device and receiving device]
As described above, a transmitting device that transmits data without operating a fragment counter can also be configured as shown in FIG. 64. Further, a receiving device that receives data without operating a fragment counter can also be configured as shown in FIG. 65. FIG. 64 is a diagram showing an example of a specific configuration of a transmitting device. FIG. 65 is a diagram showing an example of a specific configuration of a receiving device.

Transmitting device 500 includes a dividing section 501, a configuration section 502, and a transmitting section 503. Each of the dividing unit 501, the configuration unit 502, and the transmitting unit 503 is realized by, for example, a microcomputer, a processor, a dedicated circuit, or the like.

The receiving device 600 includes a receiving section 601, a determining section 602, and a configuration section 603. Each of the receiving section 601, the determining section 602, and the configuring section 603 is realized by, for example, a microcomputer, a processor, a dedicated circuit, or the like.

A detailed explanation of each component of the transmitting device 500 and the receiving device 600 will be given in the description of the transmitting method and the receiving method, respectively.

First, the transmission method will be explained using FIG. 66. FIG. 66 is an operation flow (transmission method) by the transmitting device.

First, the dividing unit 501 of the transmitting device 500 divides data into a plurality of divided data (S1701).

Next, the configuration unit 502 of the transmitting device 500 configures a plurality of packets by adding header information to each of the plurality of divided data and packetizing the divided data (S1702).

Then, the transmitting unit 503 of the transmitting device 500 transmits the plurality of configured packets (S1703). The transmitter 503 transmits the fragment data information and the invalidated fragment counter value. Note that the divided data information is information for specifying the divided data number and the number of divided data. Further, the divided data number is a number indicating which divided data the corresponding divided data is among the plurality of divided data. The number of divided data is the number of multiple pieces of divided data.

Thereby, the processing amount of the transmitting device 500 can be reduced.

Next, the reception method will be explained using FIG. 67. FIG. 67 is an operation flow (receiving method) by the receiving device.

First, the receiving unit 601 of the receiving device 600 receives a plurality of packets (S1801).

Next, the determining unit 602 of the receiving device 600 determines whether the divided data information has been acquired from the plurality of received packets (S1802).

Then, when the determining unit 602 determines that the divided data information has been acquired (Yes in S1802), the configuration unit 603 of the receiving device 600 performs reception without using the value of the fragment counter included in the header information. Data is constructed from a plurality of packets (S1803).

On the other hand, if the determining unit 602 determines that the divided data information has not been acquired (No in S1802), the configuration unit 603 uses the value of the fragment counter included in the header information to Data may be configured from packets (S1804).

Thereby, the processing amount of the receiving device 600 can be reduced.

(Embodiment 5)
[overview]
In Embodiment 5, a method for transmitting a transmission packet (TLV packet) when NAL units are stored in a multiplexing layer in a NAL size format will be described.

As explained in Embodiment 1, H. 264 and H. There are two types of formats for storing H.265 NAL units in the multiplexing layer. One is a format called a byte stream format in which a start code consisting of a specific bit string is added immediately before the NAL unit header. The other format is called the NAL size format, which adds a field indicating the size of the NAL unit. The byte stream format is used in the MPEG-2 system, RTP, etc., and the NAL size format is used in MP4, or DASH, MMT, etc. that use MP4.

In the byte stream format, the start code consists of 3 bytes, and further arbitrary bytes (bytes whose value is 0) can be added.

On the other hand, in the general NAL size format in MP4, size information is indicated as 1 byte, 2 bytes, or 4 bytes. This size information is indicated in the lengthSizeMinusOne field in the HEVC sample entry. A value of "0" in the field indicates 1 byte, a value of "1" indicates 2 bytes, and a value of "3" indicates 4 bytes.

Here, in ARIB STD-B60 "Media Transport Method Using MMT in Digital Broadcasting", which was standardized in July 2014, when storing NAL units in the multiplex layer, the output of the HEVC encoder is a byte stream. In this case, the byte start code is removed, and the size of the NAL unit in bytes indicated by 32 bits (unsigned integer) is added as length information immediately before the NAL unit. Note that the MPU metadata including the HEVC sample entry is not transmitted, and the size information is fixed at 32 bits (4 bytes).

In addition, in ARIB STD-B60 "Media Transport Method Using MMT in Digital Broadcasting", in the receive buffer model that the transmitter considers during transmission to ensure buffer operation in the receiver, the buffer before decoding of the video signal is is defined as CPB.

However, there are the following issues. CPB in the MPEG-2 system and HRD in HEVC are defined on the premise that the video signal is in a byte stream format. For this reason, for example, if the rate of a transmission packet is controlled on the assumption that it is in a byte stream format with a 3-byte start code, a transmission packet in NAL size format with a 4-byte size field added is received. The receiving device may not be able to satisfy the receiving buffer model in ARIB STD-B60. Furthermore, since the reception buffer model in ARIB STD-B60 does not indicate a specific buffer size and extraction rate, it is difficult to guarantee buffer operation in the reception device.

Therefore, in order to solve the above problem, a reception buffer model for guaranteeing buffer operation in the receiver is defined as follows.

FIG. 68 shows a receive buffer model based on the receive buffer model specified in ARIB STD B-60, particularly when only a broadcast transmission path is used.

The reception buffer model includes a TLV packet buffer (first buffer), an IP packet buffer (second buffer), an MMTP buffer (third buffer), and a pre-decoding buffer (fourth buffer). Note that the broadcast transmission path does not require a de-jitter buffer or a buffer for FEC, so they are omitted.

The TLV packet buffer receives TLV packets (transmission packets) from the broadcast transmission path, and stores variable-length packet headers (IP packet headers, full headers when compressing IP packets, and full headers when compressing IP packets) stored in the received TLV packets. An IP packet obtained by converting an IP packet consisting of a compressed header (compressed header) and a variable length payload into an IP packet (first packet) having a fixed length IP packet header with the header expanded. Output at a constant bit rate.

The IP packet buffer converts an IP packet into an MMTP packet (second packet) having a packet header and a variable length payload, and outputs the MMTP packet obtained by the conversion at a constant bit rate. Note that the IP packet buffer may be merged with the MMTP buffer.

The MMTP buffer converts the output MMTP packet into a NAL unit, and outputs the NAL unit obtained by the conversion at a constant bit rate.

The pre-decoding buffer sequentially accumulates the output NAL units, generates an access unit from the accumulated NAL units, and outputs the generated access unit to the decoder at the timing of the decoding time corresponding to the access unit.

The reception buffer model shown in FIG. 68 is characterized in that the MMTP buffer and pre-decoding buffer, which are buffers other than the TLV packet buffer and IP packet buffer at the previous stage, follow the reception buffer model in MPEG-2 TS.

For example, the MMTP buffer (MMTP B1) in video is composed of a buffer corresponding to a transport buffer (TB) and a multiplexing buffer (MB) in MPEG-2 TS. Furthermore, the MMTP buffer (MMTP Bn) in audio is configured with a buffer equivalent to the transport buffer (TB) in MPEG-2 TS.

The buffer size of the transport buffer is the same as MPEG-2 TS and is a fixed value. For example, it is assumed to be n times the MTU size (n may be a decimal number or an integer, and is 1 or more).

Furthermore, the MMTP packet size is defined so that the overhead rate of the MMTP packet header is smaller than the overhead rate of the PES packet header. As a result, the transport buffer extraction rates RX1, RXn, and RXs in the MPEG-2 TS can be directly applied to the extraction rate from the transport buffer.

Also, the size of the multiplexing buffer and the extraction rate are respectively MPEG-2
Let MB size in TS and RBX1 be.

In addition to the above reception buffer model, the following constraints are established to solve the problem.

The HEVC HRD specification is based on a byte stream format, and MMT is a NAL size format that adds a 4-byte size area to the beginning of a NAL unit. Therefore, during encoding, rate control is performed in the NAL size format so as to satisfy the HRD.

That is, the transmitter performs rate control of transmission packets based on the above-described receive buffer model and constraints.

By performing reception processing using the above-mentioned signals, the receiving device can perform a decoding operation that does not cause underflow or overflow.

Note that the extraction rate of the TLV packet buffer (the bit rate at which the TLV packet buffer outputs IP packets) is set in consideration of the transmission rate after IP header expansion.

That is, after inputting a TLV packet whose data size is variable and removing the TLV header and decompressing (restoring) the IP header, the transmission rate of the output IP packet is considered. In other words, the increase or decrease of the header is considered with respect to the input transmission rate.

Specifically, the data size is variable length, packets with IP header compression and packets with uncompressed IP header coexist, and the size of the IP header differs depending on the packet type such as IPv4 or IPv6. , the transmission rate of the output IP packets is not unique. For this reason, the average packet length of the variable-length data size is determined, and the transmission rate of the IP packet output from the TLV packet is determined.

Here, in order to define the maximum transmission rate after IP header expansion, the transmission rate is determined assuming that the IP header is always compressed.

In addition, when IPv4 and IPv6 packet types coexist, or when specifying packet types without distinguishing them, the transmission rate is determined assuming IPv6 packets with a large header size and a large increase rate after header expansion. .

For example, if the average packet length of the TLV packets input to the TLV packet buffer is S, and all IP packets stored in the TLV packets are IPv6 packets, and the headers are compressed, the TLV header removal and The maximum output transmission rate after decompressing the IP header is:
Input rate x {S/(S+IPv6 header compression amount)}
becomes.

More specifically, the average packet length S of TLV packets is
S = 0.75 x 1500 (1500 assumes maximum MTU size)
set as the standard,
IPv6 header compression amount = TLV header length - IPv6 header length - UDP header length
=3-40-8
In this case, the maximum output transmission rate after removing the TLV header and expanding the IP header is:
Input rate x 1.0417 ≒ Input rate x 1.05
becomes.

FIG. 69 is a diagram illustrating an example of aggregating a plurality of data units and storing them in one payload.

In the MMT method, when aggregating data units, a data unit length and a data unit header are added before the data unit, as shown in FIG. 69.

However, for example, when storing a video signal in the NAL size format as one data unit, as shown in FIG. 70, there are two fields indicating the size for one data unit, and the information is duplicated. . FIG. 70 is a diagram showing an example in which a plurality of data units are aggregated and stored in one payload, and a video signal in the NAL size format is used as one data unit. Specifically, both the first size area in the NAL size format (referred to as the "size area" in the following explanation) and the data unit length field located before the data unit header in the MMTP payload header have the same size. This is a field that indicates information, and is duplicated as information. For example, if the length of the NAL unit is L bytes, the size area indicates L bytes, and the data unit length field indicates L bytes + "length of size area" (byte). Although the values shown in the size area and the data unit length field do not completely match, it can be said that they overlap because the value of one can be easily calculated from the other.

In this way, when data that includes data size information is stored as a data unit, and multiple data units are aggregated and stored in one payload, the size information overlaps, resulting in large overhead. There is a problem with poor transmission efficiency.

Therefore, when the sending device stores data that includes data size information as a data unit and aggregates a plurality of data units and stores them in one payload, as shown in FIGS. 71 and 72, It is possible to store it.

As shown in FIG. 71, it is conceivable that the NAL unit including the size area is stored as a data unit, and the data unit length conventionally included in the MMTP payload header is not indicated. FIG. 71 is a diagram showing the structure of the payload of an MMTP packet in which the data unit length is not indicated.

Further, as shown in FIG. 72, a flag indicating whether the data unit length is indicated or information indicating the length of the size area may be newly stored in the header. The location where information indicating the length of the flag and size area is stored may be indicated in units of data units, such as in data unit headers, or may be indicated in units of aggregation of a plurality of data units (units of packets). FIG. 72 is an example of an extend area added to each packet. Note that the storage location of the newly indicated information is not limited to this, and may be the MMTP payload header, MMTP packet header, or control information.

On the receiving side, if the flag indicating whether the data unit length is compressed indicates that the data unit length is compressed, it obtains the length information of the size area inside the data unit, and By acquiring the size area based on the length information of the size area, the data unit length can be calculated using the acquired length information of the size area and the size area.

By the above method, the amount of data can be reduced on the sending side and transmission efficiency can be improved.

Note that the overhead may be reduced by reducing the size area instead of reducing the data unit length. When reducing the size area, information indicating whether the size area has been reduced or information indicating the length of the data unit length field may be stored.

Note that the MMTP payload header also includes length information.

When storing a NAL unit including a size area as a data unit, the payload size area in the MMTP payload header may be reduced regardless of whether aggregation is performed or not.

Furthermore, even when data that does not include the size area is stored as a data unit, if it is aggregated and the data unit length is indicated, the payload size area in the MMTP payload header may be reduced.

When the payload size area is to be reduced, a flag indicating whether the payload size area has been reduced or the length information of the reduced size field may be shown in the same way as described above.

FIG. 73 shows the operation flow of the receiving device.

In the sending device, as described above, the NAL unit including the size area is stored as a data unit, and the data unit length included in the MMTP payload header is not indicated in the MMTP packet.

In the following, a case will be explained using a flag to determine whether the data unit length is indicated or not, and a case where length information of the size area is indicated in the MMTP packet.

The receiving device determines whether the data unit includes a size area and the data unit length has been reduced based on information transmitted from the sending side (S1901).

If it is determined that the data unit length has been reduced (Yes in S1902), obtain length information of the size area inside the data unit, and then analyze the size area inside the data unit to calculate the data unit length. (S1903).

On the other hand, if it is determined that the data unit length has not been reduced (No in S1902), the data unit length is calculated from either the data unit length or the size area inside the data unit as usual (S1904).

[Supplement: Transmitting device and receiving device]
As described above, a transmitting device that performs rate control so as to satisfy the specifications of the reception buffer model during encoding can also be configured as shown in FIG. 74. Further, a receiving device that receives and decodes a transmission packet transmitted from a transmitting device can also be configured as shown in FIG. 75. FIG. 74 is a diagram illustrating an example of a specific configuration of a transmitting device. FIG. 75 is a diagram showing an example of a specific configuration of a receiving device.

Transmitting device 700 includes a generating section 701 and a transmitting section 702. Each of the generation section 701 and the transmission section 702 is realized by, for example, a microcomputer, a processor, a dedicated circuit, or the like.

The receiving device 800 includes a receiving section 801, a first buffer 802, a second buffer 803, a third buffer 804, a fourth buffer 805, and a decoding section 806. Each of the receiving section 801, first buffer 802, second buffer 803, third buffer 804, fourth buffer 805, and decoding section (decoder) 806 is a microcomputer, a processor, a dedicated circuit, etc. realized by

A detailed explanation of each component of the transmitting device 700 and the receiving device 800 will be given in the description of the transmitting method and the receiving method, respectively.

First, the transmission method will be explained using FIG. 76. FIG. 76 is an operation flow (transmission method) by the transmitting device.

First, the generating unit 701 of the transmitting device 700 generates an encoded stream by performing rate control so as to satisfy a predetermined specification based on a receiving buffer model in order to guarantee the buffer operation of the receiving device (S2001).

Next, the transmitter 702 of the transmitter 700 packetizes the generated encoded stream and transmits the transmission packet obtained by packetizing (S2002).

Note that the receiving buffer model used in the transmitting device 700 has a configuration including the first to fourth buffers 802 to 805 of the configuration of the receiving device 800, and therefore the description thereof will be omitted.

Thereby, the transmitting device 700 can guarantee the buffer operation of the receiving device 800 when transmitting data using a method such as MMT.

Next, the reception method will be explained using FIG. 77. FIG. 77 is an operation flow (receiving method) by the receiving device.

First, the receiving unit 801 of the receiving device 800 receives a transmission packet composed of a variable-length packet header and a variable-length payload (S2101).

Next, the first buffer 802 of the receiving device 800 converts the packet consisting of the variable-length packet header and variable-length payload stored in the received transmission packet into a fixed-length packet header that has been header-expanded. and outputs the first packet obtained by the conversion at a constant bit rate (S2102).

Next, the second buffer 803 of the receiving device 800 converts the first packet obtained by the conversion into a second packet consisting of a packet header and a variable length payload, and The second packet thus obtained is output at a constant bit rate (S2103).

Next, the third buffer 804 of the receiving device 800 converts the output second packet into a NAL unit, and outputs the NAL unit obtained by the conversion at a constant bit rate (S2104).

Next, the fourth buffer 805 of the receiving device 800 sequentially accumulates the output NAL units, generates an access unit from the accumulated plurality of NAL units, and transfers the generated access unit to the decoding time corresponding to the access unit. It is output to the decoder at the timing of (S2105).

Then, the decoding unit 806 of the receiving device 800 decodes the access unit output by the fourth buffer (S2106).

Thereby, the receiving device 800 can perform a decoding operation that causes underflow or overflow.

(Embodiment 6)
[overview]
In Embodiment 6, a specific transmission method and reception method using the reception buffer model described in Embodiment 5 will be described.

FIG. 78 is a diagram showing a protocol stack of the MMT/TLV method defined in ARIB STD-B60.

In the MMT method, data such as video and audio is sent to multiple MPUs (Media
The MMTP packet is stored in each predetermined data unit such as Presentation Unit (Presentation Unit) or MFU (Media Fragment Unit), and an MMTP packet is generated as a predetermined packet by adding an MMTP packet header (making it into an MMTP packet). Further, by adding an MMTP packet header to control information such as a control message in MMTP, an MMTP packet as a predetermined packet is generated. The MMTP packet header is provided with a field for storing a 32-bit short format NTP (Network Time Protocol: defined in IETF RFC 5905), which can be used for QoS control of a communication line, etc.

In addition, the reference clock on the transmitting side (transmitting device) is synchronized with the 64-bit long format NTP specified in RFC 5905, and based on the synchronized reference clock, PTS (Presentation Time Stamp) and DTS (Decode Time Stamp) A timestamp such as Stamp) is attached to the synchronized media. Further, reference clock information on the transmitting side is transmitted to the receiving side, and the receiving device generates a system clock in the receiving device based on the reference clock information received from the transmitting side.

Specifically, PTS and DTS are stored in the MPU timestamp descriptor and MPU extended timestamp descriptor, which are MMTP control information, stored in the MP table for each asset, and converted into MMTP packets as control messages. After that, it will be transmitted.

The MMTP packetized data is encapsulated into an IP packet with a UDP header or an IP header added thereto. At this time, a set of packets having the same source IP address, destination IP address, source port number, destination port number, and protocol type in the IP header or UDP header is defined as an IP data flow. Note that since IP packets of the same IP data flow have redundant headers, some IP packets are header compressed.

Further, as reference clock information, a 64-bit NTP time stamp is stored in the NTP packet and in the IP packet. At this time, in the IP packet storing the NTP packet, the source IP address, destination IP address, source port number, destination port number, and protocol type are fixed values, and the header of the IP packet is not compressed.

FIG. 79 is a diagram showing the structure of a TLV packet.

As shown in FIG. 79, the TLV packet can include, as data, IP packets, compressed IP packets, and transmission control information such as AMT (Address Map Table) and NIT (Network Information Table). Identified using an 8-bit data type. Furthermore, in the TLV packet, the data length (in bytes) is indicated using a 16-bit field, followed by storing the data value. Further, the TLV packet has 1 byte of header information before the data type, and the header information is stored in a header area of 4 bytes in total. Further, the TLV packet is mapped to a transmission slot in the advanced BS transmission system, and mapping information is stored in TMCC (Transmission and Multiplexing Configuration Control) control information.

FIG. 80 is a diagram illustrating an example of a block diagram of a receiving device.

In the receiving device 900, first, the demodulating means 902 decodes the transmission line encoded data, performs error correction, etc. on the broadcast signal received by the tuner 901, and extracts the TLV packet. The TLV/IP DEMUX unit 903 performs TLV DEMUX processing and IP DEMUX processing. TLV DEMUX processing is performed according to the data type of the TLV packet. For example, if the TLV packet includes a compressed IP packet, the compressed header of the compressed IP packet is restored. The IP DEMUX performs processing such as header analysis of IP packets and UDP packets, and extracts MMTP packets and NTP packets.

The NTP clock generating means 904 reproduces the NTP clock from the extracted NTP packet. The MMTP DEMUX unit 905 performs filtering processing on components such as video and audio and control information based on the packet ID stored in the extracted MMTP packet header. Control information acquisition means 906 acquires the time stamp descriptor stored in the MP table, and PTS/DTS calculation means 907 calculates PTS and DTS for each access unit. Note that the timestamp descriptor includes both an MPU timestamp descriptor and an MPU extended timestamp descriptor.

The access unit reproduction means 908 converts the video, audio, etc. filtered from the MMTP packet into data in units to be presented. Specifically, the unit of data to be presented is a NAL unit or access unit of a video signal, an audio frame, a subtitle presentation unit, or the like. The decoding and presenting means 909 decodes and presents the access unit at the time when the PTS/DTS of the access unit match based on the reference time information of the NTP clock.

Note that the configuration of the receiving device is not limited to this.

FIG. 81 is a diagram showing the configuration of a transmission slot. Each frame consists of 120 transmission slots. Modulation schemes are assigned to transmission slots in units of 5 slots. Furthermore, a maximum of 16 streams can be transmitted within one frame. A tlv_stream_id for identifying a TLV stream constituted by a TLV packet string is stored in the TMCC, and a tlv_stream_id stored in a slot can be specified. The modulation method and coding rate can be changed for each slot, and once the number of slots to be used is determined, the transmission rate of the TLV stream is uniquely determined.

Next, a method for defining the buffer size of the TLV packet buffer in the reception buffer model described in FIG. 68 of the fifth embodiment will be described. Note that the extraction rate explained in the fifth embodiment is used as the extraction rate Rxi of the TLV packet buffer. The reception buffer model is the same as in Embodiment 5, so the explanation will be omitted.

The buffer size of the TLV packet buffer is composed of TLV packets with an average packet length S in unit time T, and is based on the maximum buffer occupancy expected when a TLV stream input at a transmission rate Rt is extracted at a extraction rate Rxi. This is a value defined by calculating as follows. However, the average packet length S is the average size when packets are transmitted using all the TLV stream bandwidth, and is not the average size of the TLV stream bandwidth excluding the TLV null packet and the bandwidth transmitted by TLV-SI. . Note that the packet with the largest amount of header compression and header expansion is the smallest size packet among variable-length packets, and in the case where the amount of occupancy in the TLV packet buffer is the largest, the packet with the smallest packet size is the most consecutive packet. This is the case. The minimum packet size may be defined as the header amount of the TLV buffer, or may be the minimum value of the TLV/IP/UDP/MMT header when storing the MMT packet.

In other words, the regulation of the buffer size of the TLV packet buffer in the above reception buffer model is equivalent to the regulation of the maximum number of TLV packets transmitted per unit time. Furthermore, the above buffer size regulation can also be said to be a regulation that limits the number of consecutive packets of the minimum packet size per unit time.

Note that it is desirable that the transmission rate Rt of the input TLV stream, which is used to define the buffer size, be based on the maximum transmission rate expected in the system. Note that the transmission rate does not necessarily have to be the maximum transmission rate. Furthermore, although the method for setting the unit time T is arbitrary, it is desirable to set it in consideration of ease of control on the sending side.

The transmission equipment (transmission device) must perform packetization so as not to overflow the buffer size determined using, for example, the input transmission rate Rt and the extraction rate Rxi.

Note that the transmission rate Rt, unit time T, and average packet length S used in calculating the buffer size of the TLV packet buffer are values used as calculation standards, and do not necessarily stipulate that the above values are used for transmission. isn't it. In reality, the transmission speed of a TLV stream varies depending on the transmission rate. For example, when transmitting using the advanced BS transmission method (ARIB STD-B44), how many slots out of 120 slots are used for transmission? Alternatively, the transmission rate differs depending on which combination of modulation method and coding rate is used to transmit the slot. Therefore, no matter what transmission rate the transmission equipment transmits, it is sufficient to transmit without overflowing the buffer. Any control or algorithm may be used for transmission as long as it can be transmitted without overflowing the buffer.

In other words, the maximum number of TLV packets transmitted per unit time includes multiple numbers of packets with different values set according to the above transmission rate, and corresponds to the transmission rate of a predetermined data unit. The TLV packet buffer may be prevented from overflowing by transmitting a plurality of TLV packets equal to or less than the number of packets per unit time.

No matter what transmission rate transmission is performed, the receiving device can perform stable receiving operations without overflowing the buffer by implementing a buffer based on the buffer size and extraction rate described above. In other words, the receiving device only needs to implement a buffer with a buffer size larger than the maximum buffer occupancy determined using the transmission rate Rt, average packet length S, and extraction rate Rxi.

Next, a method for defining the TLV buffer model size that can define a predetermined maximum number of TLV packets per transmission frame length unit (unit transmission period) will be described.

FIG. 82 is a diagram showing a transmission frame and one TLV stream (TLV packet sequence) having a specific tlv_stream_id stored in the transmission frame.

Further, (1) in FIG. 82 is a diagram showing a case where the unit time T is the transmission frame length when the transmission frame is transmitted at a predetermined transmission rate in the buffer size regulation method described above. The transmission frame length is the average section of the average packet length S, or the section where the maximum number of TLV packets and the maximum number of consecutive minimum packet sizes are restricted. For example, in the advanced BS transmission method (ARIB STD-B44), the transmission frame length is 33.04647 ms. Here, it is assumed that the maximum consecutive number of minimum packet sizes that can be transmitted per transmission frame length is M.

In this case, the sending side (transmitting device) sends out the TLV stream without overflowing the TLV packet buffer, regardless of the boundaries of the transmission frame. In any section of section C, the TLV stream may be transmitted so as to satisfy the buffer model for generating the TLV stream so that the number of TLV packets is equal to or less than the maximum number of TLV packets stored in the TLV stream. Here, the length of each of the A section, B section, and C section is the transmission frame length. That is, in this case, the transmission equipment (transmission device) may transmit (transmit) a plurality of TLV packets that are equal to or less than a predetermined number of packets (maximum number of TLV packets) per unit time.

Further, the predetermined number of packets may be a value set so that TLV packets of the minimum packet size are not consecutive. For example, as described above, the predetermined number of packets may be a value calculated when the maximum consecutive number M of the minimum packet size is set.

Further, the maximum number of packets per transmission frame, the average packet length, or the number of consecutive packets of the minimum packet size may be defined in a predetermined data unit such as a transmission frame unit.

For example, the maximum number of packets, the average packet length, or the number of consecutive packets of the minimum packet size may be defined for each transmission frame, as shown in section (2) in FIG. 82, section E, and section F. . In other words, the transmission equipment (transmission device) transmits data in units of time that do not overlap each other, for each of a plurality of consecutive unit transmission periods (periods in a plurality of transmission frame lengths corresponding to a plurality of transmission frames). By transmitting a transmission frame during the transmission period, a plurality of TLV packets equal to or less than a predetermined number of packets (maximum number of TLV packets) may be sent out (sent) per unit time. In this case, a plurality of TLV packets equal to or less than a predetermined number of packets are stored in a transmission frame unit (that is, one transmission frame) as a predetermined data unit.

Next, a specific example of a method for calculating the buffer size of the TLV packet buffer in the reception buffer model will be described.

Here, when the maximum number of consecutive packets of the minimum packet size that can be transmitted per transmission frame unit is set to M, the maximum number of consecutive packets of the minimum packet size in the TLV stream that is actually transmitted is 2×M. Note that the maximum consecutive number M is the same value as the maximum consecutive number of minimum packet sizes that can be transmitted per the transmission frame length. In the example of FIG. 82, the maximum number of consecutive packets is the case where the TLV stream has M packets consecutively in the second half of the D section and M packets consecutively in the first half of the E section, and (2×M) packets are consecutive in total. In order for the receiving device to have a buffer size that can withstand successive minimum packet sizes of (2 x M) packets, the above method should be used, assuming that the maximum number of consecutive minimum packet sizes that can be transmitted per transmission frame length is M. A buffer size that is twice the buffer size calculated using this method is required.

In other words, by specifying the TLV buffer size to have a buffer size that is more than twice the buffer size calculated using the method described above, where the unit time T is the transmission frame length, the transmission equipment can The maximum number of packets, average packet length, or number of consecutive packets with the minimum packet size calculated when the transmission frame length is calculated as the maximum number of packets, average packet length, or number of consecutive packets with the minimum packet size per transmission frame. Even in this case, it can be sent to satisfy the buffer model.

As described above, by setting the buffer size and extraction rate of the TLV packet buffer, it is possible to define the maximum number of packets per transmission frame, the average packet length, or the number of consecutive packets of the minimum packet size. Note that the maximum number of packets, the average packet length, and the number of consecutive packets of the minimum packet size can be calculated from the buffer size of the TLV packet buffer, the input rate (transmission rate), the extraction rate, and the minimum packet size.

Therefore, if the buffer size, extraction rate, and minimum packet size of the TLV packet buffer are fixed, the maximum number of packets, average packet length, and Alternatively, the number of consecutive packets of the minimum packet size can be defined, which facilitates transmission control (packetization or mapping to transmission frames).

Next, using the method explained in FIG. 82, a transmission method when the buffer size of the TLV packet buffer and the maximum number of packets per transmission frame (or the maximum number of consecutive packets of the minimum packet size) are specified. This will be explained using FIG. 83. FIG. 83 is a flowchart of a sending method when the buffer size of the TLV packet buffer and the maximum number of packets per transmission frame are defined.

Note that although the flowchart and description show an example of storing TLV packets in transmission frames, the same flow can be used not only when storing TLV packets in transmission frames but also when considering storage.

First, we will start considering storage of TLV packets in which MMT packets are stored in transmission frame units.

It is determined whether the TLV packet stored in the transmission frame is a TLV packet (actual packet) storing an MMTP packet (S2201).

If it is determined that the TLV packet is a TLV packet storing an MMTP packet (Yes in S2201), the TLV packet is stored in the transmission frame, and the number of packets stored in the transmission frame is counted by +1 (S2202). ).

On the other hand, if it is determined that the TLV packet is a TLV packet (NULL packet or TLV-SI) that stores other than MMT packets (No in S2201), the size of TLV packets that store other than MMT packets is accumulated and the size is 1.5 kB. The number of packets is counted by 1 for each (MTU size) (S2203).

In other words, there are two types of TLV packets: a first transmission packet that stores a real packet (MMTP packet) that is different from an invalid packet (NULL packet), and a second transmission packet that stores a NULL packet. be. Furthermore, the first transmission packet is defined to have a predetermined maximum packet size, and the second transmission packet is a packet whose packet size may exceed the maximum packet size of the first transmission packet. I can do things. When storing TLV packets in a transmission frame, when storing a first transmission packet in a transmission frame, the number of first transmission packets is counted as the number of first packets, and when storing a second transmission packet in a transmission frame, The integer value obtained by dividing the packet size of the second transmission packet by the maximum packet size (1.5 kB) of the first transmission packet is counted as the number of second packets, and the A plurality of TLV packets are stored in a transmission frame such that the sum of the number of first packets and the number of second packets is equal to or less than a predetermined number of packets. Note that the second packet number is, more specifically, an integer value obtained by adding 1 to the quotient obtained by dividing the packet size of the second transmission packet by the maximum value of the packet size of the first transmission packet. For example, if the packet size of the second packet is always smaller than the maximum packet size of the first packet, it goes without saying that the number of second transmission packets will be equivalent to the number of second packets.

After step S2202 or step S2203, the cumulative TLV packet size (total data size) within the transmission frame is calculated (S2204). That is, the total sum of packet sizes of TLV packets stored in one transmission frame is calculated.

Next, it is determined whether the total data size calculated in step S2204 has reached the maximum data size in transmission frame units (S2205).

If the total data size has reached the maximum data size in transmission frame units (Yes in step S2205), a null packet is placed without inserting a TLV packet, and the process ends.

On the other hand, if the total data size does not reach the maximum data size per transmission frame (No in step S2205), the sum of the first packet number calculated in step S2202 and the second packet number calculated in step S2203 is the transmission frame. It is determined whether the maximum number of packets per unit (that is, a predetermined number of packets) has been reached (S2206).

If it is determined that the sum of the first packet number and the second packet number has reached the maximum number of packets (Yes in step S2206), the transmission frame of the TLV packet (first transmission packet) in which the MMTP packet is stored is transmitted. The TLV packet in which the next MMTP packet has been stored is stored in the next transmission frame.

In addition, in the remaining transmission band of the transmission frame, other than the TLV packet (first transmission packet) in which the MMTP packet is stored, for example, a NULL packet or a TLV packet (second transmission packet) in which TLV-SI is stored is stored. (S2207).

As described above, control to satisfy the TLV packet buffer specifications in the receive buffer model is limited to the number of packets to be stored in a transmission frame, the packet size, the number of bytes that can be stored in a transmission frame, and the maximum number of packets that can be stored in a transmission frame. can be easily controlled using

Furthermore, by being able to define the maximum number of TLV packets stored in a transmission frame, the design of the receiving device becomes easier. In other words, the receiving device only needs to design the buffer size of the buffer that functions as a TLV packet buffer to be larger than the maximum buffer occupancy determined using the transmission rate, the average packet length of transmission packets, and the extraction rate.

In addition, for each maximum TLV stream transmission rate (modulation method, coding rate, number of used transmission slots), the maximum number of packets per TLV stream per frame, the average packet length, or the number of consecutive packets with the minimum packet size. If specified, it may be specified together with other constraints. For example, the average packet length may be restricted to be less than or equal to a TS packet (188 bytes or 187 bytes on average).

This restricts the number of variable-length packets from becoming larger than when storing fixed-length TS packets, which facilitates implementation and design of the receiving device.

Note that the data input to the TLV packet buffer is one TLV stream with a specific tlv_stream_id, but by setting it as one IP data flow with a specific tlv_stream_id, a specific IP address, and a specific UDP port number. , may be replaced as a regulation for each IP data flow.

Further, for example, in advanced BS/CS broadcasting, in the case of CS, a plurality of IP data flows (=services) are stored in one TLV stream. In this case, a reception buffer model, a maximum number of packets per transmission frame, etc. can be defined for each service.

[Supplement: Transmitting device and receiving device]
As described above, a transmitting device that transmits a predetermined data unit in which a plurality of transmission packets are stored while satisfying the regulations according to a predetermined receive buffer model to guarantee the buffer operation of the receiving device is shown in FIG. It is also possible to configure as follows. Further, a receiving device that receives a predetermined data unit can also be configured as shown in FIG. 85. FIG. 84 is a diagram showing an example of a specific configuration of a transmitting device. FIG. 85 is a diagram showing an example of a specific configuration of a receiving device.

Transmitting device 1000 includes a transmitting section 1001. The transmitter 1001 is realized by, for example, a microcomputer, a processor, a dedicated circuit, or the like. That is, each processing unit configuring transmitting device 1000 may be realized by software or hardware.

Receiving device 1100 includes a receiving section 1101 and a buffer 1102. Each of the receiving unit 1101 and the buffer 1102 is realized by, for example, a microcomputer, a processor, or a dedicated circuit. That is, each processing unit configuring receiving device 1100 may be realized by software or hardware.

A detailed explanation of each component of the transmitting device 1000 and the receiving device 1100 will be given in the description of the transmitting method and the receiving method, respectively.

First, the transmission method will be explained using FIG. 86. FIG. 86 is an operation flow (transmission method) by the transmitting device.

The transmission method is a transmission method in which a plurality of transmission packets are transmitted while satisfying regulations based on a predetermined reception buffer model in order to guarantee buffer operation of the reception device 1100.

First, the transmitter 1001 of the transmitter 1000 transmits a plurality of transmission packets equal to or less than a predetermined number of packets per unit time (S2301).

Note that the transmission packet is composed of a variable length packet header and a variable length payload. In addition, the reception buffer model receives a transmission packet, converts the first packet consisting of a variable-length packet header and a variable-length payload stored in the received transmission packet into a fixed-length packet header that has been header expanded. , and outputs the second packet obtained by the conversion at a predetermined extraction rate.

Thereby, the transmitting device 1000 can guarantee the buffer operation of the receiving device 1100 when transmitting data using a method such as MMT.

Next, the reception method will be explained using FIG. 87. FIG. 87 is an operation flow (receiving method) by the receiving device.

First, the receiving unit 1101 of the receiving device 1100 receives a plurality of transmission packets each composed of a variable-length packet header and a variable-length payload (S2401).

Next, the buffer 1102 of the receiving device 1100 converts the first packet composed of the variable-length packet header and variable-length payload stored in the plurality of received transmission packets into a header-expanded fixed-length packet. The second packet is converted into a second packet having a header, and the second packet obtained by the conversion is output at a predetermined extraction rate (S2402).

Note that the buffer size of the buffer is larger than the maximum buffer occupancy calculated using the transmission rate of a plurality of transmission packets, the average packet length of transmission packets, and the extraction rate.

Thereby, the receiving device 1100 can perform processing without underflow or overflow occurring in the buffer.

(Other embodiments)
Although the transmitting device, the receiving device, the transmitting method, and the receiving method according to the embodiment have been described above, the present invention is not limited to this embodiment.

Furthermore, each processing section included in the transmitting device and receiving device according to the above embodiments is typically realized as an LSI, which is an integrated circuit. These may be integrated into one chip individually, or may be integrated into one chip including some or all of them.

Further, circuit integration is not limited to LSI, and may be realized using a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after the LSI is manufactured, or a reconfigurable processor that can reconfigure the connections and settings of circuit cells inside the LSI may be used.

In each of the embodiments described above, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

In other words, the transmitting device and the receiving device include a processing circuit and a storage electrically connected to the processing circuit (accessible from the control circuit). The processing circuit includes at least one of dedicated hardware and a program execution unit. Further, when the processing circuit includes a program execution section, the storage device stores a software program executed by the program execution section. The processing circuit uses the storage device to execute the transmission method or reception method according to the above embodiment.

Furthermore, the present invention may be the above-mentioned software program, or may be a non-transitory computer-readable recording medium on which the above-mentioned program is recorded. Furthermore, it goes without saying that the above program can be distributed via a transmission medium such as the Internet.

Moreover, all the numbers used above are exemplified to specifically explain the present invention, and the present invention is not limited to the illustrated numbers.

Furthermore, the division of functional blocks in the block diagram is just an example; multiple functional blocks can be realized as one functional block, one functional block can be divided into multiple functional blocks, or some functions can be moved to other functional blocks. It's okay. Further, functions of a plurality of functional blocks having similar functions may be processed in parallel or in a time-sharing manner by a single piece of hardware or software.

Furthermore, the order in which the steps included in the above transmission method or reception method are executed is for illustrative purposes in order to concretely explain the present invention, and an order other than the above may be used. Further, some of the above steps may be executed simultaneously (in parallel) with other steps.

The transmitting device, receiving device, transmitting method, and receiving method according to one or more aspects of the present invention have been described above based on the embodiments, but the present invention is not limited to these embodiments. do not have. Unless departing from the spirit of the present invention, various modifications that can be thought of by those skilled in the art to this embodiment, and embodiments constructed by combining components of different embodiments may also be applied to one or more of the present invention. may be included within the scope of the embodiment.

INDUSTRIAL APPLICATION This invention is applicable to the apparatus or apparatus which performs media transport, such as video data and audio data.

15, 100, 300, 500, 700, 1000 Transmitting device 16, 101, 301 Encoding section 17, 102 Multiplexing section 18, 104 Transmitting section 20, 200, 400, 600, 800, 1100 Receiving device 21 Packet filtering section 22 Transmission order type determination section 23 Random access section 24, 212 Control information acquisition section 25 Data acquisition section 26 Calculation section 27 Initialization information acquisition section 28, 206 Decoding instruction section 29, 204A, 204B, 204C, 204D, 402 Decoding section 30 Presentation Section 201 Tuner 202 Demodulation section 203 Demultiplexing section 205 Display section 211 Type discrimination section 213 Slice information acquisition section 214 Decoded data generation section 302 Addition section 303, 503, 702, 1001 Transmission section 401, 601, 801, 1101 Receiving section 501 Division unit 502, 603 configuration unit 602 determination unit 701 generation unit 802 first buffer 803 second buffer 804 third buffer 805 fourth buffer 806 decoding unit 1102 buffer

Claims (2)

A transmission method for transmitting multiple transmission packets, the transmission method comprising:
The transmission packet is composed of a variable length packet header and a variable length payload,
A reception buffer model for receiving the plurality of transmission packets to be transmitted is:
The transmission packet is received, and a first packet consisting of a variable-length packet header and a variable-length payload stored in the received transmission packet is converted into a second packet having a header-expanded fixed-length packet header. and a buffer for outputting the second packet obtained by the conversion at a predetermined extraction rate,
storing the plurality of transmission packets of a predetermined number of packets or less in a predetermined data unit;
transmitting the plurality of transmission packets;
The plurality of transmission packets include a first transmission packet storing a real packet different from an invalid packet, and a second transmission packet storing the invalid packet,
In the storage,
When storing the first transmission packets in the predetermined data unit, the number of the first transmission packets is counted as the number of first packets,
When storing the second transmission packet in the predetermined data unit, 1 is added to the quotient obtained by dividing the packet size of the second transmission packet by the maximum packet size of the first transmission packet. Count the integer value as the second packet number,
storing the plurality of transmission packets in the predetermined data unit such that the sum of the first number of packets and the second number of packets obtained by counting is equal to or less than the predetermined number of packets;
The maximum value of the packet size of the first transmission packet is 1500B. The transmission method. A transmitting device that transmits a predetermined data unit in which a plurality of transmission packets are stored,
The transmission packet is composed of a variable length packet header and a variable length payload,
A reception buffer model for receiving the plurality of transmission packets to be transmitted is:
The transmission packet is received, and a first packet consisting of a variable-length packet header and a variable-length payload stored in the received transmission packet is converted into a second packet having a header-expanded fixed-length packet header. and a buffer for outputting the second packet obtained by the conversion at a predetermined extraction rate,
comprising a transmitting unit that stores the plurality of transmission packets of a predetermined number or less in a predetermined data unit and transmits the plurality of transmission packets,
The plurality of transmission packets include a first transmission packet storing a real packet different from an invalid packet, and a second transmission packet storing the invalid packet,
The transmitter includes:
When storing the first transmission packets in the predetermined data unit, the number of the first transmission packets is counted as the number of first packets,
When storing the second transmission packet in the predetermined data unit, 1 is added to the quotient obtained by dividing the packet size of the second transmission packet by the maximum packet size of the first transmission packet. Count the integer value as the second packet number,
storing the plurality of transmission packets in the predetermined data unit such that the sum of the first number of packets and the second number of packets obtained by counting is equal to or less than the predetermined number of packets;
The maximum value of the packet size of the first transmission packet is 1500B. The transmitting device.

Images