JP2020108170A - Receiving method, transmitting method, receiving device, and transmitting device - Google Patents

Abstract

To provide a receiving method and the like capable of reproducing a plurality of second data units stored in a first data unit at an intended time.SOLUTION: In a receiving method, a first data unit storing a plurality of second data units, first time information indicating a presentation time of the first data unit, second time information used to calculate a presentation time or a decoding time of each of the plurality of second data units together with the first time information, and identification information are received, the presentation time or the decoding time of each of the plurality of second data units is calculated using the first time information and the second time information, and the presentation time or the decoding time is corrected on the basis of the identification information for each of the plurality of second data units, and the identification information is information indicating whether the first time information is generated on the basis of reference time information before leap second adjustment, and the first time information is time information indicating the presentation time of the second data unit at the head of the presentation order of the plurality of second data units.SELECTED DRAWING: Figure 103

Description

The present invention relates to a receiving method, a transmitting method, a receiving device and a transmitting device.

With the sophistication of broadcasting and communication services, the introduction of ultra-high-definition moving image content such as 8K (7680×4320 pixels: hereinafter also referred to as 8K4K) and 4K (3840×2160 pixels: hereinafter also referred to as 4K2K) is being considered Has been done. The receiving device needs to decode and display the received encoded data of the ultra-high-definition moving image in real time, and in particular, a moving image having a resolution such as 8K has a heavy processing load at the time of decoding. It is difficult to decode a moving image with a single decoder in real time. Therefore, a method for reducing the processing load per decoder and achieving real-time processing by parallelizing the decoding processing using a plurality of decoders has been studied.

Also, the encoded data is transmitted after being multiplexed based on a multiplexing method such as MPEG-2 TS (Transport Stream) or MMT (MPEG Media Transport). For example, Non-Patent Document 1 discloses a technique of transmitting encoded media data packet by packet according to MMT.

Information technology-High efficiency coding and media delivery in heterogeneous environment-Part1: MPEG media transport (MMT), ISO23/IEC-1 DIS

Meanwhile, with the advancement of broadcasting and communication services, introduction of ultra-high-definition moving image content such as 8K and 4K (3840×2160 pixels) is being considered. In the MMT/TLV method, a reference clock on the transmission side is synchronized with a 64-bit long format NTP defined in RFC 5905, and based on the reference clock, PTS (Presentation Time Stamp), DTS (Decode Time Stamp), or the like is used. Add a time stamp to the synchronized media. Further, the reference clock information of the transmitting side is transmitted to the receiving side, and the receiving device generates a system clock in the receiving device based on the reference clock information.

However, in such an MMT transmission method, when the leap second adjustment is performed on the reference time information such as NTP, the plurality of second data units stored in the MPU which is the first data unit received by the receiving device There is a problem that the plurality of access units cannot be decoded or presented at the intended time even according to the DTS or PTS associated with the plurality of access units.

According to the present invention, even when the leap second adjustment is performed on the reference time information serving as the reference of the reference clocks of the transmitting side and the receiving device, the plurality of second data stored in the first data unit is used. It provides a reception method that allows the unit to be played back at the intended time.

In order to achieve the above object, a receiving method according to an aspect of the present invention is a receiving method for receiving a first data unit in which data forming an encoded stream is stored. Stores a plurality of second data units, and in the receiving method, the first data unit, first time information indicating a presentation time of the first data unit, and the first time information. Using the received first time information and the second time information, the second time information indicating the presentation time or the decoding time of each of the plurality of second data units, and the identification information are received. , Calculating the presentation time or decoding time of each of the received second data units, and based on the received identification information, the presentation time or decoding time of each of the calculated second data units For each of the plurality of second data units, the identification information is information indicating whether the first time information is generated based on the reference time information before leap second adjustment, The first time information is time information indicating a presentation time of the first second data unit in the presentation order among the plurality of second data units.

In order to achieve the above object, a transmission method according to an aspect of the present invention is a transmission method for transmitting a first data unit in which data forming an encoded stream is stored. Stores a plurality of second data units, and in the transmitting method, generates first time information indicating a presentation time of the first data unit based on reference time information received from the outside, No. 1 data unit, the generated first time information, second time information indicating the presentation time or decoding time of each of the plurality of second data units using the first time information, and identification information. And the identification information is information indicating whether the first time information is generated based on the reference time information before leap second adjustment, and the first time information is the plurality of second information. It is time information indicating the presentation time of the first second data unit in the presentation order of the data units.

In order to achieve the above object, a receiving method according to an aspect of the present invention is a receiving method for receiving a first data unit in which data forming an encoded stream is stored. Stores a plurality of second data units, and in the receiving method, the first data unit, first time information indicating a presentation time of the first data unit, and the first time information. And receiving second time information indicating the presentation time or decoding time of each of the plurality of second data units and identification information, and using the received first time information and the second time information, The presentation time or decoding time of each of the received second data units is calculated, and the presentation time or the decoding time of each of the calculated second data units is calculated based on the received identification information. to correct.

Note that these general or specific aspects may be realized by a recording medium such as a system, a device, an integrated circuit, a computer program, or a computer-readable CD-ROM, and the system, the device, the integrated circuit, the computer program. And may be realized by any combination of recording media.

According to the present invention, even when the leap second adjustment is performed on the reference time information serving as the reference of the reference clocks of the transmitting side and the receiving device, the plurality of second data stored in the first data unit is used. The unit can be regenerated at the intended time.

FIG. 1 is a diagram showing an example of dividing a picture into slice segments. FIG. 2 is a diagram showing an example of a PES packet sequence in which picture data is stored. FIG. 3 is a diagram showing an example of picture division according to the first embodiment. FIG. 4 is a diagram showing an example of picture division according to a comparative example of the first embodiment. FIG. 5 is a diagram showing an example of data of the access unit according to the first embodiment. FIG. 6 is a block diagram of the transmitting apparatus according to the first embodiment. FIG. 7 is a block diagram of the receiving apparatus according to the first embodiment. FIG. 8 is a diagram showing an example of the MMT packet according to the first embodiment. FIG. 9 is a diagram showing another example of the MMT packet according to the first embodiment. FIG. 10 is a diagram showing an example of data input to each decoding unit according to the first embodiment. FIG. 11 is a diagram showing an example of the MMT packet and header information according to the first embodiment. FIG. 12 is a diagram showing another example of data input to each decoding unit according to the first embodiment. FIG. 13 is a diagram showing an example of picture division according to the first embodiment. FIG. 14 is a flowchart of the transmission method according to the first embodiment. FIG. 15 is a block diagram of the receiving apparatus according to the first embodiment. FIG. 16 is a flowchart of the receiving method according to the first embodiment. FIG. 17 is a diagram showing an example of the MMT packet and header information according to the first embodiment. FIG. 18 is a diagram showing an example of the MMT packet and header information according to the first embodiment. FIG. 19 is a diagram showing the configuration of the MPU. FIG. 20 is a diagram showing the structure of MF metadata. FIG. 21 is a diagram for explaining the data transmission order. FIG. 22 is a diagram showing an example of a method for decoding without using header information. FIG. 23 is a block diagram of the transmitting apparatus according to the second embodiment. FIG. 24 is a flowchart of the transmission method according to the second embodiment. FIG. 25 is a block diagram of the receiving apparatus according to the second embodiment. FIG. 26 is a flowchart of the operation for identifying the MPU head position and the NAL unit position. FIG. 27 is a flowchart of the operation of acquiring the initialization information based on the transmission order type and decoding the media data based on the initialization information. FIG. 28 is a flowchart of the operation of the receiving device when the low delay presentation mode is provided. FIG. 29 is a diagram showing an example of a transmission order of MMT packets when auxiliary data is transmitted. FIG. 30 is a diagram for explaining an example in which the transmission device generates auxiliary data based on the moof configuration. FIG. 31 is a diagram for explaining reception of auxiliary data. FIG. 32 is a flowchart of the receiving operation using the auxiliary data. FIG. 33 is a diagram showing the structure of an MPU including a plurality of movie fragments. FIG. 34 is a diagram for explaining the transmission order of MMT packets when the MPU having the configuration of FIG. 33 is transmitted. FIG. 35 is a first diagram for explaining an operation example of the receiving device when one MPU is composed of a plurality of movie fragments. FIG. 36 is a second diagram for explaining an operation example of the receiving device when one MPU is composed of a plurality of movie fragments. FIG. 37 is a flowchart of the operation of the receiving method described with reference to FIGS. FIG. 38 is a diagram showing a case where non-VCL NAL units are individually used as data units and are aggregated. FIG. 39 is a diagram showing a case where non-VCL NAL units are collectively used as a data unit. FIG. 40 is a flowchart of the operation of the receiving device when packet loss occurs. FIG. 41 is a flowchart of the receiving operation when the MPU is divided into a plurality of movie fragments. FIG. 42 is a diagram illustrating an example of a prediction structure of a picture in each TemporalId when realizing temporal scalability. FIG. 43 is a diagram showing the relationship between the decoding time (DTS) and the display time (PTS) in each picture of FIG. FIG. 44 is a diagram illustrating an example of a picture prediction structure that requires picture delay processing and picture reorder processing. FIG. 45 is a diagram showing an example in which an MPU configured in MP4 format is divided into a plurality of movie fragments and stored in an MMTP payload and an MMTP packet. FIG. 46 is a diagram for explaining the PTS and DTS calculation method and problems. FIG. 47 is a flowchart of the receiving operation when the DTS is calculated using the DTS calculation information. FIG. 48 is a diagram for explaining a method of storing a data unit in a payload in MMT. FIG. 49 is an operation flow of the transmitting apparatus according to the third embodiment. FIG. 50 is an operation flow of the receiving device according to the third embodiment. FIG. 51 is a diagram showing an example of a specific configuration of the transmitting apparatus according to the third embodiment. FIG. 52 is a diagram showing an example of a specific configuration of the receiving apparatus according to the third embodiment. FIG. 53 is a diagram showing a method for storing a non-timed medium in an MPU and a method for transmitting an MMTP packet. FIG. 54 is a diagram showing an example in which a plurality of pieces of divided data obtained by dividing a file are packetized and transmitted. FIG. 55 is a diagram showing another example in which a plurality of pieces of divided data obtained by dividing a file are packetized and transmitted. FIG. 56 is a diagram showing the syntax of a loop for each file in the asset management table. FIG. 57 is an operation flow for identifying a divided data number in the receiving device. FIG. 58 is an operation flow for specifying the number of division data in the receiving device. FIG. 59 is an operation flow for determining whether to operate the fragment counter in the transmission device. FIG. 60 is a diagram for explaining a method of specifying the number of pieces of divided data and the number of divided data (when a fragment counter is used). FIG. 61 is an operation flow of the transmission device when utilizing the fragment counter. FIG. 62 is an operation flow of the receiving device when utilizing the fragment counter. FIG. 63 is a diagram showing a service configuration when the same program is transmitted by a plurality of IP data flows. FIG. 64 is a diagram illustrating an example of a specific configuration of the transmission device. FIG. 65 is a diagram illustrating an example of a specific configuration of the receiving device. FIG. 66 is an operation flow of the transmitting device. FIG. 67 is an operation flow of the receiving device. FIG. 68 shows a reception buffer model based on the reception buffer model defined in ARIB STD-B60, particularly when only a broadcast transmission path is used. FIG. 69 is a diagram showing an example in which a plurality of data units are aggregated and stored in one payload. FIG. 70 is an example of aggregating a plurality of data units and storing them in one payload, and is a diagram showing an example in which a video signal in the NAL size format is one data unit. FIG. 71 is a diagram showing the structure of the payload of an MMTP packet in which the data unit length is not shown. FIG. 72 is an example shown in the extend area added in units of packets. FIG. 73 shows an operation flow of the receiving device. FIG. 74 is a diagram illustrating an example of a specific configuration of the transmission device. FIG. 75: is a figure which shows the example of a concrete structure of a receiver. FIG. 76 is an operation flow of the transmitting device. FIG. 77 is an operation flow of the receiving device. FIG. 78 is a diagram showing a protocol stack of the MMT/TLV method defined in ARIB STD-B60. FIG. 79 is a diagram showing the structure of a TLV packet. FIG. 80 is a diagram showing an example of a block diagram of a receiving device. FIG. 81 is a diagram for explaining the time stamp descriptor. FIG. 82 is a diagram for explaining leap second adjustment. FIG. 83 is a diagram showing the relationship between the NTP time, the MPU time stamp, and the MPU presentation timing. FIG. 84 is a diagram for explaining a correction method for correcting the time stamp on the transmission side. FIG. 85 is a diagram for explaining a correction method for correcting the time stamp in the receiving device. FIG. 86 is an operation flow by the transmission side (transmission device) when the MPU time stamp is corrected on the transmission side (transmission device). FIG. 87 is an operation flow by the receiving device when correcting the MPU time stamp on the transmitting side (transmitting device). FIG. 88 is an operation flow by the transmitting side (transmitting apparatus) when the receiving apparatus corrects the MPU time stamp. FIG. 89 is an operation flow by the receiving device when the receiving device corrects the MPU time stamp. FIG. 90 is a diagram illustrating an example of a specific configuration of the transmission device. FIG. 91 is a diagram illustrating an example of a specific configuration of the receiving device. FIG. 92 is an operation flow of the transmitting device. FIG. 93 is an operation flow of the receiving device. FIG. 94 is a diagram showing an extension example of the MPU extension time stamp descriptor. FIG. 95 is a diagram for describing a case where discontinuity occurs in MPU sequence numbers due to the adjustment of MPU sequence numbers. FIG. 96 is a diagram for explaining a case where packet sequence numbers are discontinuous at the timing of switching from normal equipment to redundant equipment. FIG. 97 is an operation flow by the receiving device when discontinuity of MPU sequence number or packet sequence number occurs. FIG. 98 is a diagram for explaining a correction method for correcting the time stamp in the receiving device when the leap second is inserted. FIG. 99 is a diagram for explaining a correction method for correcting the time stamp in the receiving device when the leap second is deleted. FIG. 100 is an operation flow of the receiving device. FIG. 101 is a diagram showing an example of a specific configuration of the transmission/reception system. FIG. 102 is a diagram showing a specific configuration of the receiving device. FIG. 103 is an operation flow of the receiving device.

A receiving method according to an aspect of the present invention is a receiving method of receiving a first data unit in which data forming an encoded stream is stored, wherein the first data unit is a plurality of second data. A unit is stored, and in the reception method, the plurality of second data units include the first data unit, first time information indicating a presentation time of the first data unit, and the first time information. The second time information indicating the presentation time or the decoding time of each data unit and the identification information are received, and the received second time information is used by using the received first time information and the second time information. The presentation time or the decoding time of each data unit is calculated, and the presentation time or the decoding time of each of the calculated second data units is corrected based on the received identification information.

According to this, even if the leap second adjustment is performed with respect to the reference time information serving as the reference of the reference clocks of the transmitting side and the receiving device, such a receiving method can provide the first time at the intended time. An encoded stream composed of data units can be played.

Further, the first time information may be absolute time information indicating a presentation time of a head second data unit in the presentation order among the plurality of second data units.

Further, the first time information may be time information that is not corrected on the transmission side of the first data unit during leap second adjustment.

Further, the second time information may be relative time information for calculating a presentation time or a decoding time of each of the plurality of second data units with the first time information.

The identification information is information indicating whether the first time information is generated based on reference time information before leap second adjustment, and in the correction, for each of the plurality of second data units, The first time information of the first data unit storing the second data unit is generated based on the reference time information before leap second adjustment, and the calculated second time information is calculated. It is determined whether the presentation time or the decoding time of the data unit satisfies a correction condition that is after a predetermined time, and in the determination, the presentation of the second data unit that is determined to satisfy the correction condition. The time or the decoding time may be corrected.

Further, the reference time information may be NTP (Network Time Protocol).

Also, whether the first time information is generated based on the reference time information from a time that is a predetermined period before the time immediately before the leap second adjustment is performed to a time immediately before the first time information is the identification information. It may be information indicating whether or not.

Further, in the reception, even if a predetermined packet in which the control information in which the first time information corresponding to the first data unit is stored and a plurality of the first data units are stored are received. Good.

Further, in the reception, an MMTP (MPEG Media Transport Protocol) packet as the predetermined packet, in which MPUs (Media Presentation Units) as a plurality of the first data units are stored, is received, and the MMTP packet is The control information includes an MPU time stamp descriptor including the first time information, an MPU extended time stamp descriptor including the second time information, and identification information, and the MPU includes the plurality of second data. A plurality of access units as a unit may be stored.

Note that these comprehensive or specific aspects may be realized by a recording medium recording medium such as a device, a system, an integrated circuit, a computer program or a computer-readable CD-ROM, and the device, the system, the integrated circuit, It may be realized by any combination of computer programs or recording media.

Hereinafter, embodiments will be specifically described with reference to the drawings.

It should be noted that each of the embodiments described below shows a comprehensive or specific example. Numerical values, shapes, materials, constituent elements, arrangement positions and connection forms of constituent elements, steps, order of steps, and the like shown in the following embodiments are examples, and are not intended to limit the present invention. Further, among the constituent elements in the following embodiments, constituent elements not described in the independent claim showing the highest concept are described as arbitrary constituent elements.

(Findings that form the basis of the present invention)
In recent years, the resolution of displays such as TVs, smartphones, and tablet terminals has been increasing. Especially for broadcasting in Japan, a service of 8K4K (resolution is 8K×4K) is scheduled for 2020. Since it is difficult for a single decoder to perform real-time decoding of an ultra-high resolution moving image such as 8K4K, a method of performing decoding processing in parallel using a plurality of decoders is being studied.

Since the encoded data is multiplexed and transmitted based on a multiplexing scheme such as MPEG-2 TS or MMT, the receiving device needs to separate the encoded data of the moving image from the multiplexed data before decoding. .. Hereinafter, the process of separating encoded data from multiplexed data is called demultiplexing.

When parallelizing the decoding process, it is necessary to distribute the encoded data to be decoded to each of the decoders. When distributing the encoded data, it is necessary to analyze the encoded data itself, and particularly in the case of contents such as 8K, since the bit rate is very high, the processing load related to the analysis is large. Therefore, there is a problem that the demultiplexing part becomes a bottleneck and reproduction cannot be performed in real time.

By the way, H.264 standardized by MPEG and ITU. H.264 and H.264. In a moving image coding method such as H.265, the transmitting device can divide a picture into a plurality of areas called slices or slice segments, and perform coding so that each of the divided areas can be independently decoded. Therefore, for example, H.264. In the case of H.265, the receiving device that receives the broadcast can realize parallelization of the decoding process by separating the data for each slice segment from the received data and outputting the data of each slice segment to different decoders. ..

FIG. 1 is a diagram showing an example of dividing one picture into four slice segments in HEVC. For example, the receiving device includes four decoders, and each decoder decodes any one of the four slice segments.

In conventional broadcasting, the transmitting device stores one picture (access unit in the MPEG system standard) in one PES packet and multiplexes the PES packet into a TS packet sequence. Therefore, the receiving device separates the payload of the PES packet and then analyzes the data of the access unit stored in the payload to separate each slice segment, and the data of each separated slice segment is decoded by the decoder. Had to output to.

However, the present inventor has found that there is a problem that it is difficult to perform this processing in real time because the processing amount when analyzing the data of the access unit and separating the slice segments is large.

FIG. 2 is a diagram showing an example in which picture data divided into slice segments is stored in the payload of a PES packet.

As shown in FIG. 2, for example, data of a plurality of slice segments (slice segments 1 to 4) is stored in the payload of one PES packet. The PES packet is multiplexed in the TS packet sequence.

(Embodiment 1)
In the following, H.264 is used as a moving image coding method. The case of using H.265 will be described as an example. The present embodiment can be applied to the case where another encoding method such as H.264 is used.

FIG. 3 is a diagram showing an example in which the access unit (picture) according to the present embodiment is divided into division units. The access unit is H.264. The feature called tiles introduced by H.265 divides each into two halves horizontally and vertically, for a total of four tiles. In addition, the slice segment and the tile are associated with each other one to one.

The reason for dividing into two in the horizontal and vertical directions will be described. First, at the time of decoding, a line memory for storing data of one horizontal line is generally required. However, when the resolution becomes extremely high, such as 8K4K, the size in the horizontal direction increases and the size of the line memory increases. In the implementation of the receiver, it is desirable to be able to reduce the size of the line memory. Vertical division is necessary to reduce the size of the line memory. A data structure called a tile is required for vertical division. For these reasons, tiles are used.

On the other hand, since an image generally has a high correlation in the horizontal direction, it is better to refer to a wide range in the horizontal direction to improve coding efficiency. Therefore, from the viewpoint of coding efficiency, it is desirable that the access unit be divided in the horizontal direction.

By bisecting the access unit in the horizontal and vertical directions, these two characteristics can be made compatible, and both the mounting surface and the encoding efficiency can be considered. When a single decoder can decode a 4K2K moving image in real time, the 8K4K image is divided into four equal parts, and each slice segment is divided into 4K2K, so that the receiving device Can decode 8K4K images in real time.

Next, the reason why the tiles obtained by dividing the access unit in the horizontal and vertical directions and the slice segments are in one-to-one correspondence will be described. H. In the H.265, the access unit is composed of a plurality of units called NAL (Network Adaptation Layer) units.

The payload of the NAL unit is an access unit delimiter indicating the start position of the access unit, SPS (Sequence Parameter Set) that is initialization information commonly used in decoding in sequence units, and an initial decoding time commonly used in a picture. Either PPS (Picture Parameter Set) which is the encoding information, SEI (Supplemental Enhancement Information) which is not necessary for the decoding process itself but is necessary for processing and displaying the decoding result, and encoded data of the slice segment. To store. The header of the NAL unit includes type information for identifying the data stored in the payload.

Here, the transmission device encodes encoded data into MPEG-2 TS, MMT (MPEG Media Transport), and MPEG DASH (Dynamic Adaptive).
When multiplexing by a multiplexing format such as Streaming over HTTP) or RTP (Real-time Transport Protocol), the basic unit can be set to the NAL unit. In order to store one slice segment in one NAL unit, it is desirable to divide the access unit into slice segments when dividing the access unit into regions. For this reason, the transmission device associates tiles with slice segments in a one-to-one correspondence.

Note that, as illustrated in FIG. 4, the transmission device can collectively set tiles 1 to 4 to one slice segment. However, in this case, all the tiles are stored in one NAL unit, and it is difficult for the receiving device to separate the tiles in the multiplexing layer.

Although there are independent slice segments that can be independently decoded and reference slice segments that refer to the independent slice segments, the slice segments will be described here.

FIG. 5 is a diagram showing an example of data of access units divided so that the boundaries between tiles and slice segments match as shown in FIG. The data of the access unit includes the NAL unit in which the access unit delimiter placed at the head is stored, the NPS units of SPS, PPS, and SEI placed after that, and tiles 1 to 4 placed after that. Data of the slice segment in which the data is stored. The access unit data may not include a part or all of the NPS units of SPS, PPS, and SEI.

Next, the configuration of transmitting apparatus 100 according to the present embodiment will be described. FIG. 6 is a block diagram showing a configuration example of transmitting apparatus 100 according to the present embodiment. This transmission device 100 includes an encoding unit 101, a multiplexing unit 102, a modulation unit 103, and a transmission unit 104.

The encoding unit 101 converts the input image into, for example, H.264. Encoded data is generated by encoding in accordance with H.265. Further, the encoding unit 101 divides the access unit into four slice segments (tiles) and encodes each slice segment, as shown in FIG. 3, for example.

The multiplexing unit 102 multiplexes the encoded data generated by the encoding unit 101. The modulator 103 modulates the data obtained by the multiplexing. The transmission unit 104 transmits the modulated data as a broadcast signal.

Next, the configuration of receiving apparatus 200 according to the present embodiment will be described. FIG. 7 is a block diagram showing a configuration example of receiving apparatus 200 according to the present embodiment. The receiving device 200 includes a tuner 201, a demodulation unit 202, a demultiplexing unit 203, a plurality of decoding units 204A to 204D, and a display unit 205.

The tuner 201 receives a broadcast signal. The demodulation unit 202 demodulates the received broadcast signal. The demodulated data is input to the demultiplexing unit 203.

The demultiplexing unit 203 separates the demodulated data into division units, and outputs the data for each division unit to the decoding units 204A to 204D. Here, the division unit is a division area obtained by dividing the access unit. It is a slice segment in H.265. Further, here, the 8K4K image is divided into four 4K2K images. Therefore, there are four decoding units 204A to 204D.

The plurality of decoding units 204A to 204D operate in synchronization with each other based on a predetermined reference clock. Each decoding unit decodes the encoded data in units of division according to the DTS (Decoding Time Stamp) of the access unit, and outputs the decoding result to the display unit 205.

The display unit 205 generates an 8K4K output image by integrating a plurality of decoding results output from the plurality of decoding units 204A to 204D. The display unit 205 displays the generated output image according to the PTS (Presentation Time Stamp) of the access unit, which is acquired separately. When the decoding results are integrated, the display unit 205 performs a filtering process such as a deblocking filter so that the boundary becomes visually inconspicuous in the boundary area of the division units adjacent to each other such as the tile boundary. Good.

In the above description, the transmitting device 100 and the receiving device 200 that transmit or receive the broadcast have been described as an example, but the content may be transmitted and received via a communication network. When the receiving device 200 receives the content via the communication network, the receiving device 200 separates the multiplexed data from the IP packet received by the network such as Ethernet.

In broadcasting, the transmission line delay from when the broadcast signal is transmitted to when it reaches the receiving device 200 is constant. On the other hand, in a communication network such as the Internet, due to the influence of congestion, the transmission line delay until the data transmitted from the server reaches the receiving device 200 is not constant. Therefore, the receiving device 200 often does not perform strict synchronous reproduction based on a reference clock such as PCR in MPEG-2 TS of broadcasting. Therefore, the receiving device 200 may display the 8K4K output image on the display unit in accordance with the PTS without strictly synchronizing the decoding units.

In addition, due to congestion of the communication network or the like, there are cases where decoding processing in all division units is not completed at the time indicated by the PTS of the access unit. In this case, the receiving apparatus 200 skips the display of the access unit or delays the display until the decoding of at least four division units is completed and the generation of the 8K4K image is completed.

The content may be transmitted and received using both broadcasting and communication. The present method can also be applied when reproducing multiplexed data stored in a recording medium such as a hard disk or a memory.

Next, a method of multiplexing access units divided into slice segments when MMT is used as the multiplexing method will be described.

FIG. 8 is a diagram showing an example of packetizing data of an HEVC access unit into an MMT. Although SPS, PPS, SEI, and the like do not necessarily have to be included in the access unit, the case where they exist is illustrated here.

NAL units such as access unit delimiter, SPS, PPS, and SEI that are arranged before the first slice segment in the access unit are collectively stored in the MMT packet #1. Subsequent slice segments are stored in separate MMT packets for each slice segment.

Note that, as shown in FIG. 9, the NAL unit arranged before the head slice segment in the access unit may be stored in the same MMT packet as the head slice segment.

Also, when NAL units such as End-of-Sequence or End-of-Bitstream indicating the end of the sequence or stream are added after the last slice segment, they are included in the same MMT packet as the last slice segment. Is stored. However, since NAL units such as End-of-Sequence or End-of-Bitstream are inserted at the end point of the decoding process, the connection point of two streams, or the like, the receiving apparatus 200 inserts these NAL units. , It may be desirable to be able to easily obtain in the multiplexing layer. In this case, these NAL units may be stored in an MMT packet other than the slice segment. By this means, receiving apparatus 200 can easily separate these NAL units in the multiplexing layer.

Note that TS, DASH, RTP, or the like may be used as the multiplexing method. Also in these methods, the transmitting device 100 stores different slice segments in different packets. This can ensure that receiving apparatus 200 can separate slice segments in the multiplexing layer.

For example, when TS is used, encoded data is PES packetized in slice segment units. When RTP is used, encoded data is RTP packetized in slice segment units. Also in these cases, the NAL unit and the slice segment arranged before the slice segment may be packetized separately as in the MMT packet #1 shown in FIG.

When the TS is used, the transmission device 100 indicates the unit of data stored in the PES packet by using the data alignment descriptor or the like. Further, since DASH is a method of downloading a data unit of MP4 format called a segment by HTTP or the like, the transmission device 100 does not packetize the encoded data during transmission. Therefore, the transmitting apparatus 100 creates subsamples in slice segment units and stores information indicating the storage positions of subsamples in the MP4 header so that the receiving apparatus 200 can detect slice segments in the multiplexing layer in MP4. You may.

The MMT packetization of slice segments will be described in detail below.

As shown in FIG. 8, the encoded data is packetized so that data commonly referred to when decoding all slice segments in the access unit such as SPS and PPS is stored in MMT packet #1. In this case, receiving apparatus 200 concatenates the payload data of MMT packet #1 and the data of each slice segment, and outputs the obtained data to the decoding unit. In this way, the receiving device 200 can easily generate the input data to the decoding unit by connecting the payloads of the plurality of MMT packets.

FIG. 10 is a diagram showing an example in which input data to the decoding units 204A to 204D is generated from the MMT packet shown in FIG. The demultiplexing unit 203 concatenates the payload data of the MMT packet #1 and the MMT packet #2, so that the decoding unit 204A generates the data necessary for decoding the slice segment 1. The demultiplexing unit 203 similarly generates input data for the decoding units 204B to 204D. That is, the demultiplexing unit 203 concatenates the payload data of the MMT packet #1 and the MMT packet #3 to generate the input data of the decoding unit 204B. The demultiplexing unit 203 generates the input data of the decoding unit 204C by connecting the payload data of the MMT packet #1 and the MMT packet #4. The demultiplexing unit 203 generates the input data of the decoding unit 204D by linking the payload data of the MMT packet #1 and the MMT packet #5.

Note that the demultiplexing unit 203 removes NAL units that are not necessary for decoding processing, such as access unit delimiters and SEI, from the payload data of MMT packet #1, and only NAL units for SPS and PPS that are necessary for decoding processing. May be separated and added to the data of the slice segment.

When the encoded data is packetized as shown in FIG. 9, the demultiplexing unit 203 outputs the MMT packet #1 including the head data of the access unit in the multiplexing layer to the first decoding unit 204A. .. In addition, the demultiplexing unit 203 analyzes the MMT packet including the head data of the access unit in the multiplexing layer, separates the NPS units of SPS and PPS, and separates the separated NAL units of SPS and PPS into the second and subsequent slices. Input data is generated for each of the second and subsequent decoding units by adding the data to each segment data.

Furthermore, the receiving device 200 uses the information included in the header of the MMT packet to determine the type of data stored in the MMT payload and the slice segment in the access unit when the slice segment is stored in the payload. It is desirable to be able to identify the index number. Here, the type of data includes the data before the slice segment (the NAL units arranged before the first slice segment in the access unit are collectively referred to as such), and the data of the slice segment. Which is it. When storing a fragmented unit of MPU such as a slice segment in the MMT packet, a mode for storing an MFU (Media Fragment Unit) is used. When this mode is used, the transmission device 100 samples, for example, Data Unit, which is a basic unit of data in MFU, as a sample (a data unit in MMT and corresponds to an access unit) or a sub-sample (samples). It can be set to divided units.

At this time, the header of the MMT packet includes a field called Fragmentation indicator and a field called Fragment counter.

The Fragmentation indicator indicates whether or not the data stored in the payload of the MMT packet is a fragment of Data unit, and if it is a fragmented fragment, the fragment is the first or last fragment in the Data unit, or , Indicates whether the fragment is neither the beginning nor the end. In other words, the Fragmentation indicator included in the header of a packet is (1) Data unit that is a basic data unit includes only the packet, and (2) Data unit is divided and stored in a plurality of packets, and The packet is a packet at the beginning of the Data unit, (3) the Data unit is divided into a plurality of packets and stored, and the packet is a packet other than the beginning and the end of the Data unit, and (4) The data unit is identification information that indicates whether the Data unit is divided into a plurality of packets and stored, and the packet is the last packet of the Data unit.

Fragment counter is an index number indicating to which fragment in the data unit the data stored in the MMT packet corresponds.

Therefore, the transmitting apparatus 100 sets the sample in the MMT to Data unit, and sets the slice segment pre-data and each slice segment in Data unit fragment units, so that the receiving apparatus 200 causes the MMT packet header to be set. The information contained in can be used to identify the type of data stored in the payload. That is, the demultiplexing unit 203 can generate the input data to the decoding units 204A to 204D by referring to the header of the MMT packet.

FIG. 11 is a diagram showing an example in which the sample is set to Data unit and the data before the slice segment and the slice segment are packetized as a fragment of the Data unit.

The data before the slice segment and the slice segment are divided into five fragments from fragment #1 to fragment #5. Each fragment is stored in a separate MMT packet. At this time, the values of Fragmentation indicator and Fragment counter included in the header of the MMT packet are as illustrated.

For example, Fragment indicator is a binary 2-bit value. The Fragment indicator of the MMT packet #1 at the beginning of the Data unit, the Fragment indicator of the final MMT packet #5, and the Fragment indicator from the MMT packet #2 to the MMT packet #4, which are packets between them, are different from each other. Set to the value. Specifically, the Fragment indicator of the MMT packet #1 at the beginning of the Data unit is set to 01, the Fragment indicator of the final MMT packet #5 is set to 11, and the MMT packet #2, which is the packet in between, is set. Fragment indicator up to MMT packet #4 is set to 10. When the Data unit includes only one MMT packet, Fragment indicator is set to 00.

Further, Fragment counter is 4, which is a value obtained by subtracting 1 from 5 which is the total number of fragments in MMT packet #1, is successively decreased by 1 in subsequent packets, and is 0 in the last MMT packet #5. is there.

Therefore, the receiving device 200 can identify the MMT packet that stores the data before the slice segment by using either the Fragment indicator or the Fragment counter. Further, the receiving device 200 can identify the MMT packet storing the Nth slice segment by referring to the Fragment counter.

The header of the MMT packet separately includes the sequence number within the MPU of the Movie Fragment to which the Data unit belongs, the sequence number of the MPU itself, and the sequence number within the Movie Fragment of the sample to which the Data unit belongs. The demultiplexing unit 203 can uniquely determine the sample to which the Data unit belongs by referring to these.

Furthermore, since the demultiplexing unit 203 can determine the index number of the fragment in the Data unit from the Fragment counter or the like, the slice segment stored in the fragment can be uniquely specified even when a packet loss occurs. For example, the demultiplexing unit 203 knows that the fragment received next to the fragment #3 is the fragment #5 even when the fragment #4 illustrated in FIG. It is possible to correctly output the slice segment 4 stored in the decoding section 204D instead of the decoding section 204C.

When a transmission path that guarantees no packet loss is used, the demultiplexing unit 203 refers to the header of the MMT packet and determines the type of data stored in the MMT packet or the slice segment. The arrived packet may be processed periodically without determining the index number of the. For example, when the access unit transmits the pre-slice data and a total of five MMT packets of four slice segments, the receiving device 200 determines after the pre-slice data of the access unit to start decoding. By sequentially processing the received MMT packets, the pre-slice data and the data of the four slice segments can be sequentially acquired.

Hereinafter, a modification of packetization will be described.

The slice segment does not necessarily have to be divided into both the horizontal direction and the vertical direction within the plane of the access unit, and may be the one obtained by dividing the access unit only in the horizontal direction as shown in FIG. However, it may be divided only in the vertical direction.

Moreover, when the access unit is divided only in the horizontal direction, the tile need not be used.

Further, the number of in-plane divisions in the access unit is arbitrary and is not limited to four. However, the area size of the slice segment and tile is H.264. It must be equal to or higher than the lower limit of the coding standard such as H.265.

The transmitting device 100 may store the identification information indicating the in-plane division method in the access unit in the MMT message, the TS descriptor, or the like. For example, information indicating the number of divisions in the plane in the horizontal direction and the number of divisions in the vertical direction may be stored. Alternatively, identification information unique to the division method, such as being divided into two equal parts in the horizontal direction and the vertical direction as shown in FIG. 3, or being divided into four equal parts in the horizontal direction as shown in FIG. May be assigned. For example, when the access unit is divided as shown in FIG. 3, the identification information indicates the mode 1, and when the access unit is divided as shown in FIG. 1, the identification information indicates the mode 1. ..

Further, the information indicating the constraint of the coding condition related to the in-plane division method may be included in the multiplexing layer. For example, information indicating that one slice segment is composed of one tile may be used. Alternatively, the reference block in the case of performing motion compensation at the time of decoding a slice segment or tile is limited to the slice segment or tile at the same position in the screen, or limited to a block within a predetermined range in an adjacent slice segment. Information indicating a thing may be used.

Further, the transmission device 100 may switch whether to divide the access unit into a plurality of slice segments according to the resolution of the moving image. For example, the transmission device 100 does not perform in-plane division when the moving image to be processed has a resolution of 4K2K, and may divide the access unit into four when the moving image to be processed is 8K4K. Good. By predefining the division method for a moving image of 8K4K, the receiving apparatus 200 acquires the resolution of the moving image to be received, thereby determining the presence/absence of the in-plane division and the division method, and decoding The operation can be switched.

Further, the receiving device 200 can detect the presence or absence of in-plane division by referring to the header of the MMT packet. For example, when the access unit is not divided, if the Data unit of MMT is set in the sample, the Data unit is not fragmented. Therefore, if the value of Fragment counter included in the header of the MMT packet is always zero, receiving apparatus 200 can determine that the access unit is not divided. Alternatively, the receiving device 200 may detect whether the value of Fragmentation indicator is always 01. The receiving device 200 can determine that the access unit is not divided even when the value of Fragmentation indicator is always 01.

The receiving apparatus 200 can also handle the case where the number of in-plane divisions in the access unit does not match the number of decoding units. For example, when the receiving device 200 includes two decoding units 204A and 204B capable of decoding 8K2K encoded data in real time, the demultiplexing unit 203 instructs the decoding unit 204A to perform 8K4K encoded data. 2 out of the 4 slice segments making up

FIG. 12 is a diagram showing an operation example when the MMT packetized data as shown in FIG. 8 is input to the two decoding units 204A and 204B. Here, it is desirable that the receiving apparatus 200 be able to directly output the decoding results of the decoding units 204A and 204B as they are. Therefore, the demultiplexing unit 203 selects slice segments to be output to the decoding units 204A and 204B so that the decoding results of the decoding units 204A and 204B are spatially continuous.

In addition, the demultiplexing unit 203 may select a decoding unit to be used according to the resolution or frame rate of encoded data of moving images. For example, when the receiving device 200 includes four 4K2K decoding units, if the resolution of the input image is 8K4K, the receiving device 200 performs decoding processing using all four decoding units. In addition, the receiving apparatus 200 performs the decoding process using only one decoding unit if the resolution of the input image is 4K2K. Alternatively, the demultiplexing unit 203 integrates all the division units into one decoding unit when 8K4K can be decoded in real time by a single decoding unit even if the in-plane is divided into four. Output to.

Further, the receiving device 200 may determine the decoding unit to be used in consideration of the frame rate. For example, when the receiving device 200 includes two decoding units whose upper limit of the frame rate that can be decoded in real time is 60 fps when the resolution is 8K4K, there is a case where coded data of 120 fps is input at 8K4K. is there. At this time, if the in-plane is composed of four division units, the slice segment 1 and the slice segment 2 are input to the decoding unit 204A and the slice segment 3 and the slice segment 4 are input, as in the example of FIG. It is input to the decoding unit 204B. Since each of the decoding units 204A and 204B can perform decoding in real time up to 120 fps if 8K2K (resolution is half of 8K4K), the decoding process is performed by these two decoding units 204A and 204B.

Even if the resolution and the frame rate are the same, the profile or level in the encoding method, or the H.264 standard. H.264 or H.264. The amount of processing varies depending on the encoding method such as H.265. Therefore, the receiving device 200 may select the decoding unit to use based on these pieces of information. It should be noted that the receiving device 200, when all the encoded data received by broadcasting or communication cannot be decoded, or when all the slice segments or tiles forming the area selected by the user cannot be decoded, the decoding unit. Decodable slice segments or tiles may be automatically determined within the processing range of. Alternatively, the receiving device 200 may provide a user interface for the user to select a region to be decoded. At this time, the receiving apparatus 200 may display a warning message indicating that all areas cannot be decoded, or may display information indicating the number of decodable areas, slice segments, or tiles.

The above method can also be applied to the case where an MMT packet storing a slice segment of the same encoded data is transmitted and received using a plurality of transmission lines such as broadcasting and communication.

In addition, the transmitting apparatus 100 may perform encoding so that the areas of the slice segments overlap in order to make the boundaries of the division units inconspicuous. In the example shown in FIG. 13, an 8K4K picture is divided into four slice segments 1 to 4. Each of the slice segments 1 to 3 is, for example, 8K×1.1K, and the slice segment 4 is 8K×1K. Also, adjacent slice segments overlap each other. By doing so, the motion compensation at the time of encoding can be efficiently executed at the boundary in the case of being divided into four parts shown by the dotted line, so that the image quality at the boundary portion is improved. In this way, the image quality deterioration at the boundary portion is reduced.

In this case, the display unit 205 cuts out an 8K×1K area from the 8K×1.1K area and integrates the obtained areas. Note that the transmitting device 100 includes information indicating whether slice segments are encoded in an overlapping manner and the range of the overlapping in a multiplexing layer or encoded data, and separately transmits the information. Good.

The same method can be applied when tiles are used.

Hereinafter, the flow of operation of the transmission device 100 will be described. FIG. 14 is a flowchart showing an operation example of the transmission device 100.

First, the encoding unit 101 divides a picture (access unit) into a plurality of slice segments (tiles) that are a plurality of areas (S101). Next, the encoding unit 101 generates encoded data corresponding to each of the plurality of slice segments by encoding each of the plurality of slice segments so that they can be independently decoded (S102). The encoding unit 101 may encode a plurality of slice segments with a single encoding unit or may perform parallel processing with a plurality of encoding units.

Next, the multiplexing unit 102 multiplexes the plurality of encoded data by storing the plurality of encoded data generated by the encoding unit 101 in the plurality of MMT packets (S103). Specifically, as shown in FIGS. 8 and 9, the multiplexing unit 102 stores a plurality of encoded data in a plurality so that the encoded data corresponding to different slice segments is not stored in one MMT packet. Store in the MMT packet. Further, as shown in FIG. 8, the multiplexing unit 102 sets the control information commonly used for all the decoding units in the picture as a plurality of MMT packets #2 to # where a plurality of encoded data are stored. Stored in MMT packet #1 different from No. Here, the control information includes at least one of an access unit delimiter, SPS, PPS, and SEI.

The multiplexing unit 102 may store the control information in the same MMT packet as any of the plurality of MMT packets in which the plurality of encoded data are stored. For example, as illustrated in FIG. 9, the multiplexing unit 102 stores the control information in the first MMT packet (MMT packet #1 in FIG. 9) of the plurality of MMT packets in which the plurality of encoded data are stored. You may.

Finally, the transmitter 100 transmits a plurality of MMT packets. Specifically, the modulator 103 modulates the data obtained by the multiplexing, and the transmitter 104 transmits the modulated data (S104).

FIG. 15 is a block diagram showing a configuration example of receiving apparatus 200, and is a diagram showing in detail the configuration of demultiplexing section 203 shown in FIG. 7 and the subsequent stages. As shown in FIG. 15, the receiving device 200 further includes a decoding instruction unit 206. The demultiplexing unit 203 also includes a type determination unit 211, a control information acquisition unit 212, a slice information acquisition unit 213, and a decoded data generation unit 214.

The flow of operation of the receiving device 200 will be described below. FIG. 16 is a flowchart showing an operation example of the receiving device 200. Here, the operation for one access unit is shown. When the decoding processing of a plurality of access units is executed, the processing of this flowchart is repeated.

First, the receiving device 200 receives, for example, a plurality of packets (MMT packets) generated by the transmitting device 100 (S201).

Next, the type determination unit 211 acquires the type of the encoded data stored in the received packet by analyzing the header of the received packet (S202).

Next, the type determination unit 211 determines whether the data stored in the received packet is slice segment pre-data or slice segment data, based on the type of the acquired encoded data (S203). ..

When the data stored in the received packet is the slice segment pre-data (Yes in S203), the control information acquisition unit 212 acquires the slice segment pre-data of the access unit to be processed from the payload of the received packet, and the slice The pre-segment data is stored in the memory (S204).

On the other hand, when the data stored in the received packet is the data of the slice segment (No in S203), the receiving apparatus 200 uses the header information of the received packet to determine that the data stored in the received packet is plural. It is determined which one of the areas is the encoded data. Specifically, the slice information acquisition unit 213 acquires the index number Idx of the slice segment stored in the received packet by analyzing the header of the received packet (S205). Specifically, the index number Idx is an index number in the Movie Fragment of the access unit (sample in MMT).

The process of step S205 may be collectively performed in step S202.

Next, the decoded data generation unit 214 determines the decoding unit that decodes the slice segment (S206). Specifically, the index number Idx is associated with a plurality of decoding units in advance, and the decoded data generation unit 214 causes the decoding unit corresponding to the index number Idx acquired in step S205 to decode the slice segment. Decide which decoding unit to use.

Note that the decoded data generation unit 214, as described in the example of FIG. 12, includes the resolution of the access unit (picture), the method of dividing the access unit into a plurality of slice segments (tiles), and the plurality of units included in the reception device 200. The decoding unit that decodes the slice segment may be determined based on at least one of the processing capabilities of the decoding unit. For example, the decoded data generation unit 214 determines the access unit division method based on the identification information in the descriptor such as the MMT message or the TS section.

Next, the decoded data generation unit 214 includes control information, which is included in any of the plurality of packets and is commonly used for all decoding units in the picture, and the plurality of encoded data of the plurality of slice segments. By combining and, a plurality of input data (combined data) input to the plurality of decoding units is generated. Specifically, the decoded data generation unit 214 acquires slice segment data from the payload of the received packet. The decoded data generation unit 214 generates the input data to the decoding unit determined in step S206 by combining the slice segment previous data stored in the memory in step S204 and the acquired slice segment data. (S207).

After step S204 or S207, if the data of the received packet is not the final data of the access unit (No in S208), the processes of step S201 and subsequent steps are performed again. That is, the above process is repeated until the input data to the plurality of decoding units 204A to 204D corresponding to all the slice segments included in the access unit is generated.

The timing at which the packets are received is not limited to the timing shown in FIG. 16, and a plurality of packets may be received in advance or sequentially and stored in a memory or the like.

On the other hand, when the data of the received packet is the final data of the access unit (Yes in S208), the decoding command unit 206 outputs the plurality of input data generated in step S207 to the corresponding decoding units 204A to 204D. (S209).

Next, the plurality of decoding units 204A to 204D generate a plurality of decoded images by decoding a plurality of input data in parallel according to the DTS of the access unit (S210).

Finally, the display unit 205 generates a display image by combining the plurality of decoded images generated by the plurality of decoding units 204A to 204D, and displays the display image according to the PTS of the access unit (S211).

The receiving device 200 acquires the DTS and PTS of the access unit by analyzing the payload information of the MMT packet storing the header information of the MPU or the header information of the Movie Fragment. In addition, the receiving apparatus 200 acquires the DTS and PTS of the access unit from the header of the PES packet when TS is used as the multiplexing method. When RTP is used as the multiplexing method, the receiving device 200 acquires the DTS and PTS of the access unit from the header of the RTP packet.

Further, the display unit 205 may perform a filtering process such as a deblocking filter at the boundary between adjacent division units when integrating the decoding results of the plurality of decoding units. Note that when displaying the decoding result of a single decoding unit, since the filtering process is unnecessary, the display unit 205 performs the process depending on whether to perform the filtering process on the boundary of the decoding results of the plurality of decoding units. You may switch. Whether or not the filtering process is necessary may be defined in advance depending on the presence or absence of division. Alternatively, information indicating whether or not filtering is necessary may be separately stored in the multiplexing layer. Further, information necessary for filtering such as a filter coefficient may be stored in SPS, PPS, SEI, or slice segment. The decoding units 204</b>A to 204</b>D or the demultiplexing unit 203 acquires these information by analyzing the SEI, and outputs the acquired information to the display unit 205. The display unit 205 performs filter processing using these pieces of information. Note that when these pieces of information are stored in the slice segment, it is desirable that the decoding units 204A to 204D acquire these pieces of information.

In the above description, the example in which the types of data stored in the fragment are two types, that is, the data before slice segment and the slice segment, has been described, but the types of data may be three or more types. In this case, case classification according to the type is performed in step S203.

Further, the transmission device 100 may fragment the slice segment and store it in the MMT packet when the data size of the slice segment is large. That is, the transmitting apparatus 100 may fragment the pre-slice segment data and the slice segment. In this case, if the access unit and the Data unit are set equal to each other as in the packetization example shown in FIG. 11, the following problems occur.

For example, when slice segment 1 is divided into three fragments, slice segment 1 is divided into three packets having Fragment counter values 1 to 3 and transmitted. Further, in slice segment 2 and later, the Fragment counter value becomes 4 or more, and the Fragment counter value and the data stored in the payload cannot be associated with each other. Therefore, the receiving device 200 cannot specify the packet storing the head data of the slice segment from the information of the header of the MMT packet.

In such a case, the receiving apparatus 200 may analyze the payload data of the MMT packet to identify the start position of the slice segment. Here, H. H.264 or H.264. As a format for storing the NAL unit in the multiplexing layer in H.265, a format called a byte stream format in which a start code consisting of a specific bit string is added immediately before the NAL unit header, and a field indicating the size of the NAL unit are added. There are two types, a format called NAL size format.

The byte stream format is used in the MPEG-2 system and RTP. The NAL size format is used in MP4, DASH and MMT using MP4, and the like.

When the byte stream format is used, the receiving apparatus 200 analyzes whether the head data of the packet matches the start code. If the head data of the packet matches the start code, the receiving device 200 acquires the type of the NAL unit from the NAL unit header that follows the packet so that the data included in the packet is the slice segment data. Can detect.

On the other hand, in the case of the NAL size format, the receiving device 200 cannot detect the start position of the NAL unit based on the bit string. Therefore, in order to obtain the start position of the NAL unit, the receiving device 200 needs to shift the pointer by reading the data by the size of the NAL unit in order from the head NAL unit of the access unit.

However, if the size of each sub-sample is indicated in the MPU or the Movie Fragment header in the MMT, and the sub-sample corresponds to the pre-slice data or the slice segment, the receiving apparatus 200 determines based on the size information of the sub-sample. The start position of each NAL unit can be specified. Therefore, the transmission device 100 may include information indicating whether or not the information on a sub-sample basis exists in the MPU or the Movie Fragment in the information acquired by the reception device 200 at the start of data reception, such as the MPT in MMT. ..

The data of the MPU is expanded based on the MP4 format. In MP4, H.264. H.264 or H.264. There are a mode in which a parameter set such as SPS and PPS of 265 can be stored as sample data and a mode in which they cannot be stored. Further, information for specifying this mode is shown as an entry name of SampleEntry. When the storable mode is used and the parameter set is included in the sample, the reception device 200 acquires the parameter set by the method described above.

On the other hand, when the mode that cannot be stored is used, the parameter set is stored as the Decoder Specific Information in the SampleEntry, or is stored using the stream for the parameter set. Here, since the stream for the parameter set is not generally used, it is preferable that the transmitting device 100 store the parameter set in the Decoder Specific Information. In this case, the receiving device 200 analyzes the Sample Entry transmitted as the MPU metadata or the Movie Fragment metadata in the MMT packet, and acquires the parameter set referred to by the access unit.

When the parameter set is stored as sample data, the receiving apparatus 200 can acquire the parameter set required for decoding by referring to only the sample data without referring to SampleEntry. At this time, the transmission device 100 does not have to store the parameter set in the SampleEntry. By doing so, the transmitting apparatus 100 can use the same SampleEntry in different MPUs, and thus the processing load of the transmitting apparatus 100 at the time of MPU generation can be reduced. Further, there is an advantage that the receiving device 200 does not need to refer to the parameter set in the SampleEntry.

Alternatively, the transmission device 100 may store one default parameter set in SampleEntry and store the parameter set referenced by the access unit in the sample data. In the conventional MP4, it is general to store the parameter set in the SampleEntry. Therefore, when the parameter set does not exist in the SampleEntry, there is a possibility that there is a receiving device that stops the reproduction. This problem can be solved by using the above method.

Alternatively, the transmitting device 100 may store the parameter set in the sample data only when the parameter set different from the default parameter set is used.

In both modes, it is possible to store the parameter set in the SampleEntry, so the transmitting device 100 may always store the parameter set in the VisualSampleEntry, and the receiving device 200 may always obtain the parameter set from the VisualSampleEntry.

Note that in the MMT standard, MP4 header information such as Moov and Moof is transmitted as MPU metadata or movie fragment metadata, but the transmission device 100 does not necessarily transmit MPU metadata and movie fragment metadata. You don't have to. Further, the receiving device 200 determines whether the SPS and PPS are stored in the sample data based on the service of the ARIB (Association of Radio Industries and Businesses) standard, the asset type, the presence or absence of transmission of the MPU meta, and the like. It is also possible to judge.

FIG. 17 is a diagram showing an example of a case where the slice segment pre-data and each slice segment are set to different Data units.

In the example shown in FIG. 17, the data size before slice segment and the data size from slice segment 1 to slice segment 4 are Length #1 to Length #5, respectively. The field values of Fragmentation indicator, Fragment counter, and Offset included in the header of the MMT packet are as shown in the figure.

Here, Offset indicates the bit length (offset) from the beginning of the encoded data of the sample (access unit or picture) to which the payload data belongs to the beginning byte of the payload data (encoded data) included in the MMT packet. It is offset information. The value of Fragment counter is described as starting from the value obtained by subtracting 1 from the total number of fragments, but it may be started from another value.

FIG. 18 is a diagram illustrating an example of a case where the Data unit is fragmented. In the example shown in FIG. 18, the slice segment 1 is divided into three fragments, which are respectively stored in MMT packets #2 to #4. Also at this time, assuming that the data size of each fragment is Length #2_1 to Length #2_3, the value of each field is as shown in the figure.

In this way, when a data unit such as a slice segment is set to Data unit, the head of the access unit and the head of the slice segment can be determined as follows based on the field value of the MMT packet header.

The head of the payload in the packet whose Offset value is 0 is the head of the access unit.

The head of the slice segment is the head of the payload of the packet whose Offset value is different from 0 and whose Fragmentation indicator value is 00 or 01.

If fragmentation of the data unit does not occur and packet loss does not occur, the receiving device 200 stores the MMT packet in the MMT packet based on the number of slice segments acquired after detecting the head of the access unit. The index number of the slice segment can be specified.

Also, when the Data unit of the data before the slice segment is fragmented, the receiving device 200 can similarly detect the access unit and the head of the slice segment.

Also, when packet loss occurs, or when SPS, PPS, and SEI included in slice segment previous data are set to different Data units, the receiving device 200 slices based on the analysis result of the MMT header. The start position of the slice segment or tile in the picture (access unit) can be specified by specifying the MMT packet storing the start data of the segment and then analyzing the header of the slice segment. The processing amount related to the slice header analysis is small, and the processing load does not matter.

In this way, each of the plurality of encoded data of the plurality of slice segments is in one-to-one correspondence with a basic data unit (Data unit) which is a unit of data stored in one or more packets. Also, each of the plurality of encoded data is stored in one or more MMT packets.

The header information of each MMT packet includes Fragmentation indicator (identification information) and Offset (offset information).

The receiving device 200 determines that the beginning of the payload data included in the packet having the header information that includes the Fragmentation indicator whose value is 00 or 01 is the beginning of the encoded data of each slice segment. .. Specifically, the beginning of the payload data included in the packet having the header information that includes the Offset whose value is not 0 and the Fragmentation indicator whose value is 00 or 01 is the beginning of the encoded data of each slice segment. To determine.

Further, in the example of FIG. 17, the beginning of the Data unit is either the beginning of the access unit or the beginning of the slice segment, and the value of Fragmentation indicator is 00 or 01. Further, the receiving device 200 determines whether the head of the Data Unit is the access unit delimiter or the slice segment by referring to the type of the NAL unit, thereby referring to the Offset without referring to the Offset. It is also possible to detect the beginning or the beginning of the slice segment.

In this way, the transmission device 100 performs packetization so that the head of the NAL unit always starts from the head of the payload of the MMT packet, and thus includes the case where the pre-slice segment data is divided into a plurality of Data units. Then, the receiving device 200 can detect the head of the access unit or the slice segment by analyzing the Fragmentation indicator and the NAL unit header. The type of NAL unit exists in the first byte of the NAL unit header. Therefore, the receiving device 200 can acquire the type of the NAL unit by additionally analyzing 1-byte data when analyzing the header part of the MMT packet.

In the case of audio, the receiving apparatus 200 only needs to be able to detect the head of the access unit, and may make a determination based on whether the value of Fragmentation indicator is 00 or 01.

Further, as described above, when storing the encoded data encoded so that the division decoding can be performed in the PES packet of the MPEG-2 TS, the transmission device 100 can use the data alignment descriptor. is there. Hereinafter, an example of a method of storing encoded data in a PES packet will be described in detail.

For example, in HEVC, the transmission device 100 can indicate whether the data stored in the PES packet is an access unit, a slice segment, or a tile by using the data alignment descriptor. The alignment type in HEVC is defined as follows.

Alignment type=8 indicates a slice segment of HEVC. Alignment type=9 indicates a slice segment or access unit of HEVC. Alignment type=12 indicates a slice segment or tile of HEVC.

Therefore, the transmission device 100 can indicate that the data of the PES packet is either the slice segment or the data before the slice segment by using the type 9, for example. Since a type indicating a slice instead of a slice segment is separately defined, the transmitting apparatus 100 may use the type indicating a slice instead of the slice segment.

Further, the DTS and PTS included in the header of the PES packet are set only in the PES packet including the head data of the access unit. Therefore, in the receiving device 200, if the type is 9 and the DTS or PTS field is present in the PES packet, the entire access unit or the first division unit in the access unit is stored in the PES packet. You can judge.

In addition, the transmitting device 100 uses a field such as transport_priority indicating the priority of the TS packet that stores the PES packet including the head data of the access unit so that the receiving device 200 can distinguish the data included in the packet. Good. Further, the receiving device 200 may determine the data included in the packet by analyzing whether or not the payload of the PES packet is the access unit delimiter. Further, data_alignment_indicator of the PES packet header indicates whether or not data is stored in the PES packet according to these types. When this flag (data_alignment_indicator) is set to 1, it is guaranteed that the data stored in the PES packet follows the type indicated in the data alignment descriptor.

Moreover, the transmission device 100 may use the data alignment descriptor only when PES packetizing is performed in units that can be divided and decoded such as slice segments. As a result, when the data alignment descriptor is present, the receiving apparatus 200 can determine that the encoded data is PES-packetized in units that can be divided and decoded, and if the data alignment descriptor is not present, the code can be obtained. It can be determined that the encrypted data is PES packetized in access unit units. It is specified in the MPEG-2 TS standard that the unit of PES packetization is an access unit when data_alignment_indicator is set to 1 and no data alignment descriptor is present.

If the PMT includes a data alignment descriptor, the receiving apparatus 200 determines that PES packetization is performed in units that can be divided and decoded, and the input to each decoding unit is based on the packetized units. Data can be generated. Also, when the receiving apparatus 200 does not include the data alignment descriptor in the PMT and it is determined that parallel decoding of encoded data is necessary based on program information or information of other descriptors. , Input data to each decoding unit is generated by analyzing the slice header of the slice segment. Further, when the encoded data can be decoded by a single decoding unit, the receiving device 200 decodes the data of the entire access unit by the decoding unit. If the information indicating whether or not the encoded data is composed of a unit that can be divided and decoded, such as a slice segment or a tile, is separately indicated by a descriptor of the PMT or the like, the receiving apparatus 200 determines that the descriptor Whether or not the encoded data can be decoded in parallel may be determined based on the analysis result.

In addition, since the DTS and PTS included in the header of the PES packet are set only in the PES packet including the head data of the access unit, when the access unit is divided into PES packets, the second and subsequent PES packets are set. The packet does not include information indicating the DTS and PTS of the access unit. Therefore, when performing the decoding process in parallel, each of the decoding units 204A to 204D and the display unit 205 uses the DTS and PTS stored in the header of the PES packet including the head data of the access unit.

(Embodiment 2)
In the second embodiment, a method of storing NAL size format data in an MP4 format-based MPU in MMT will be described. In the following, as an example, a storage method in the MPU used in the MMT will be described, but such a storage method is also applicable to the same MP4 format-based DASH.

[How to store in MPU]
In the MP4 format, a plurality of access units are collected and stored in one MP4 file. In the MPU used in the MMT, data for each medium is stored in one MP4 file, and the data can include an arbitrary number of access units. Since the MPU is a unit that can be decoded by itself, an access unit in GOP units is stored in the MPU, for example.

FIG. 19 is a diagram showing the configuration of the MPU. The heads of the MPUs are ftyp, mmpu, and moov, which are collectively defined as MPU metadata. Initialization information common to files and MMT hint tracks are stored in moov.

Further, in the moof, initialization information and size for each sample or subsample, information (sample_duration, sample_size, sample_composition_time_offset) that can specify the presentation time (PTS) and the decoding time (DTS), and data_offset indicating the position of the data, and the like. Is stored.

The plurality of access units are stored in mdat (mdat box) as samples. Data excluding samples from moof and mdat is defined as movie fragment metadata (hereinafter, referred to as MF metadata), and mdat sample data is defined as media data.

FIG. 20 is a diagram showing the structure of MF metadata. As shown in FIG. 20, more specifically, the MF metadata includes a type, a length, and a data of the moof box (moof), and a type and a length of the mdat box (mdat).

When storing the access unit in MP4 data, the H.264 standard is used. H.264 or H.264. There are a mode in which a parameter set such as SPS and PPS of 265 can be stored as sample data and a mode in which they cannot be stored.

Here, in the mode that cannot be stored, the parameter set is stored in the Decoder Specific Information of the SampleEntry in moov. Also, in the storable mode, the parameter set is included in the sample.

The MPU metadata, the MF metadata, and the media data are respectively stored in the MMT payload, and the fragment type (FT) is stored in the header of the MMT payload as an identifier that can identify these data. FT=0 indicates MPU metadata, FT=1 indicates MF metadata, and FT=2 indicates media data.

Note that FIG. 19 illustrates an example in which the MPU metadata unit and the MF metadata unit are stored in the MMT payload as data units, but units such as ftyp, mmpu, moov, and moof are data units, and It may be stored in the MMT payload in units. Similarly, FIG. 19 illustrates an example in which the sample unit is stored in the MMT payload as a data unit. However, a data unit may be configured in sample units or NAL unit units, and such data units may be stored in the MMT payload in data unit units. Such data units may be further fragmented and stored in the MMT payload.

[Conventional transmission method and problems]
Conventionally, when encapsulating a plurality of access units in the MP4 format, moov and moof are created when all the samples stored in the MP4 are prepared.

When transmitting the MP4 format in real time using broadcasting or the like, for example, if the samples stored in one MP4 file are GOP units, moov and moof are created after GOP unit time samples are accumulated. There is a delay associated with encapsulation. Due to such encapsulation on the transmission side, the end-to-end delay is always lengthened by the GOP unit time. As a result, it becomes difficult to provide services in real time, and in particular, when live content is transmitted, this leads to deterioration of services for viewers.

FIG. 21 is a diagram for explaining the data transmission order. When MMT is applied to broadcasting, as shown in (a) of FIG. 21, it is carried in MMT packets in the order of MPU configuration and transmitted (MMT packets #1, #2, #3, #4, #5, #6). In the order of)), a delay due to encapsulation occurs in the transmission of the MMT packet.

In order to prevent the delay due to this encapsulation, as shown in (b) of FIG. 21, MPU header information such as MPU metadata and MF metadata is not transmitted (packets #1 and #2 are not transmitted, and packet #1 is not transmitted). 3-#6 is transmitted in this order) is proposed. Further, as shown in (c) of FIG. 20, the media data is transmitted first without waiting for the creation of the MPU header information, and the MPU header information is transmitted after the transmission of the media data (#3-#6, #. 1 and #2 may be transmitted in this order).

The receiving device decodes without using the MPU header information when the MPU header information is not transmitted, and when the MPU header information is sent after the MPU header information, the receiving device decodes the MPU header information. Decrypt after waiting for information.

However, the conventional MP4-compliant receiving apparatus is not guaranteed to decode without using the MPU header information. Further, when the receiving device performs decoding by using a special process without using the MPU header, if the conventional transmission method is used, the decoding process becomes complicated and real-time decoding is likely to be difficult. Also, when the receiving device waits for the acquisition of MPU header information and then decodes, buffering of media data is necessary until the receiving device acquires the header information, but a buffer model is not specified. No, decryption was not guaranteed.

Therefore, the transmission device according to the second embodiment transmits MPU metadata before media data by storing only common information in MPU metadata, as shown in (d) of FIG. Then, the transmitting apparatus according to the second embodiment transmits the MF metadata whose generation is delayed after the media data. This provides a transmission method or a reception method that can guarantee the decoding of media data.

Hereinafter, a receiving method when each of the transmitting methods of (a) to (d) of FIG. 21 is used will be described.

In each transmission method shown in FIG. 21, first, MPU data is constructed in the order of MPU metadata, MFU metadata, and media data.

When the transmitting device transmits data in the order of MPU metadata, MF metadata, and media data after forming the MPU data as shown in (a) of FIG. 21, the receiving device is ) And (A-2).

(A-1) After receiving the MPU header information (MPU metadata and MF metadata), the receiving device decodes the media data using the MPU header information.

(A-2) The receiving device decodes the media data without using the MPU header information.

All of these methods cause a delay due to encapsulation on the transmission side, but have an advantage that the receiving apparatus does not need to buffer the media data to acquire the MPU header. When the buffering is not performed, it is not necessary to mount a memory for the buffering, and further, the buffering delay does not occur. In addition, the method (A-1) is applicable to a conventional receiving device because the decoding is performed using MPU header information.

When the transmitting device transmits only the media data as shown in (b) of FIG. 21, the receiving device can perform decoding by the method of (B-1) below.

(B-1) The receiving device decodes the media data without using the MPU header information.

Further, although not shown, when the MPU metadata is transmitted prior to the transmission of the media data of FIG. 21(b), the decoding can be performed by the method of (B-2) below.

(B-2) The receiving device decrypts the media data using the MPU metadata.

Both of the methods (B-1) and (B-2) are advantageous in that no delay occurs due to encapsulation on the transmission side, and it is not necessary to buffer media data to acquire the MPU header. Is. However, since the methods (B-1) and (B-2) do not perform decoding using MPU header information, there is a possibility that special processing may be required for decoding.

When the transmission device transmits data in the order of media data, MPU metadata, and MF metadata as shown in (c) of FIG. 21, the reception device has the following (C-1) and (C-2). Decoding can be performed by either method.

(C-1) The receiving device decodes the media data after acquiring the MPU header information (MPU metadata and MF metadata).

(C-2) The receiving device decodes the media data without using the MPU header information.

When the above method (C-1) is used, it is necessary to buffer the media data in order to acquire the MPU header information. On the other hand, when the method (C-2) is used, it is not necessary to perform buffering for acquiring MPU header information.

In addition, in any of the above methods (C-1) and (C-2), a delay due to encapsulation does not occur on the transmission side. In addition, the method (C-2) does not use MPU header information, and thus may require special processing.

When the transmission device transmits data in the order of MPU metadata, media data, and MF metadata as shown in (d) of FIG. 21, the reception device has the following (D-1) and (D-2). ) Can be used for decryption.

(D-1) The receiving device further acquires the MF metadata after acquiring the MPU metadata, and then decodes the media data.

(D-2) The receiving device, after acquiring the MPU metadata, decodes the media data without using the MF metadata.

When the method of (D-1) is used, it is necessary to buffer the media data for obtaining MF metadata, but in the case of the method of (D-2), for obtaining MF metadata. Need not be buffered.

The method (D-2) does not perform the decoding using the MF metadata, and therefore may require special processing.

As described above, if the MPU metadata and the MF metadata can be used for decoding, there is an advantage that the conventional MP4 receiving device can also perform decoding.

Note that, in FIG. 21, MPU data is configured in the order of MPU metadata, MFU metadata, and media data, and in moof, position information (offset) for each sample or subsample is determined based on this configuration. ing. The MF metadata also includes data (size and type of box) other than the media data in the mdat box.

Therefore, when the receiving device specifies the media data based on the MF metadata, the receiving device reconstructs the data in the order in which the MPU data was composed, regardless of the order in which the data was transmitted. , MPU metadata moov or MF metadata moof is used for decoding.

Note that, in FIG. 21, the MPU data is configured in the order of MPU metadata, MFU metadata, and media data, but even if the MPU data is configured in a different order from FIG. 21 and the position information (offset) is determined. Good.

For example, MPU data may be configured in the order of MPU metadata, media data, and MF metadata, and negative position information (offset) may be indicated in the MF metadata. In this case also, regardless of the order in which the data is transmitted, the receiving apparatus reconstructs the data in the order in which the MPU data was configured on the transmitting side, and then performs decoding using moov or moof.

The transmitting device may signal the information indicating the order in which the MPU data is configured, and the receiving device may reconfigure the data based on the signaled information.

As described above, as shown in (d) of FIG. 21, the receiving device receives the packetized MPU metadata, the packetized media data (sample data), and the packetized MF metadata. Receive in order. Here, the MPU metadata is an example of the first metadata, and the MF metadata is an example of the second metadata.

Next, the receiving apparatus reconstructs MPU data (MP4 format file) including the received MPU metadata, the received MF metadata, and the received sample data. Then, the sample data included in the reconstructed MPU data is decoded using the MPU metadata and the MF metadata. The MF metadata is metadata including data (for example, length stored in mbox) that can be generated only after generation of sample data on the transmission side.

It should be noted that the operation of the above-described receiving device is performed in more detail by each component that constitutes the receiving device. For example, the reception device includes a reception unit that receives the data, a reconstruction unit that reconstructs the MPU data, and a decoding unit that decodes the MPU data. Each of the receiving unit, the generating unit, and the decoding unit is realized by a microcomputer, a processor, a dedicated circuit, or the like.

[Method to decrypt without using header information]
Next, a method of decoding without using header information will be described. Here, a method of decoding without using header information in the receiving device will be described regardless of whether the transmitting side sends or does not send the header information. That is, this method is applicable to any of the transmission methods described with reference to FIG. However, some decoding methods are decoding methods applicable only to a specific transmission method.

FIG. 22 is a diagram showing an example of a method for decoding without using header information. In FIG. 22, only MMT payloads and MMT packets containing only media data are shown, and MMT payloads and MMT packets containing MPU metadata and MF metadata are not shown. Further, in the following description of FIG. 22, it is assumed that media data belonging to the same MPU are continuously transmitted. Further, a case where a sample is stored in the payload as media data will be described as an example. However, in the following description of FIG. 22, it goes without saying that a NAL unit may be stored or a fragmented NAL unit is stored. May be.

In order to decode the media data, the receiving device must first obtain the initialization information necessary for the decoding. If the medium is video, the receiving device must acquire the initialization information for each sample, specify the start position of the MPU, which is a random access unit, and acquire the start positions of the sample and the NAL unit. .. Further, the receiving device needs to specify the decoding time (DTS) and the presentation time (PTS) of each sample.

Therefore, the receiving device can perform decoding without using header information, for example, by using the following method. When the payload stores a NAL unit unit or a unit in which a NAL unit is fragmented, “sample” may be replaced with “NAL unit in sample” in the following description.

<Random access (=specify the first sample of MPU)>
In the case where the header information is not transmitted, there are the following method 1 and method 2 for the receiving device to specify the head sample of the MPU. Note that when header information is transmitted, method 3 can be used.

[Method 1] The receiving device acquires the sample included in the MMT packet with RAP_flag=1 in the MMT packet header.

[Method 2] The receiving device acquires a sample with “sample number=0” in the MMT payload header.

[Method 3] When at least one of MPU metadata and MF metadata is transmitted to at least one of the front and the rear of the media data, the receiving device determines that the receiving device has a fragment type in the MMT payload header. (FT) acquires the sample included in the MMT payload that has been switched to the media data.

In method 1 and method 2, when a plurality of samples belonging to different MPUs are mixed in one payload, it is impossible to determine which NAL unit is a random access point (RAP_flag=1 or sample number=0). For this reason, there is a restriction that samples of different MPUs are not mixed in one payload, or when samples of different MPUs are mixed in one payload, if the last (or first) sample is a random access point, RAP_flag It is necessary to set a constraint such that

Further, in order for the receiving device to acquire the start position of the NAL unit, it is necessary to shift the data read pointer by the size of the NAL unit in order from the head NAL unit of the sample.

When the data is fragmented, the receiving device can specify the data unit by referring to the fragment_indicator and the fragment_number.

<DTS determination of sample>
There are the following method 1 and method 2 for determining the DTS of the sample.

[Method 1] The receiving device determines the DTS of the first sample based on the prediction structure. However, this method requires analysis of encoded data, and decoding in real time may be difficult. Therefore, the following method 2 is preferable.

[Method 2] The receiving apparatus separately transmits the DTS of the leading sample and acquires the transmitted DTS of the leading sample. As a method of transmitting the DTS of the first sample, for example, there is a method of transmitting the DTS of the MPU first sample using MMT-SI, a method of transmitting the DTS for each sample using the MMT packet header extension area, and the like. The DTS may be an absolute value or a relative value with respect to PTS. Further, the transmitting side may signal whether or not the DTS of the first sample is included.

In both method 1 and method 2, the DTS of the subsequent sample is calculated as a fixed frame rate.

As a method of storing the DTS of each sample in the packet header, there is a method of storing the DTS of the sample included in the MMT packet in the NTP time stamp field of 32 bits in the MMT packet header, in addition to using the extension area. When the DTS cannot be represented by the number of bits (32 bits) of one packet header, the DTS may be represented by using a plurality of packet headers. Also, the DTS may be expressed by combining the NTP time stamp field of the packet header and the extension area. If the DTS information is not included, the value is a known value (for example, ALL0).

<Determination of PTS of sample>
The receiving device acquires the PTS of the first sample from the MPU time stamp descriptor for each asset included in the MPU. The receiving device calculates the subsequent sample PTS as a fixed frame rate from parameters such as POC indicating the display order of samples. Thus, in order to calculate DTS and PTS without using header information, transmission at a fixed frame rate is essential.

Further, when the MF metadata is transmitted, the receiving apparatus determines the relative time information of the DTS or PTS from the start sample indicated in the MF metadata and the time stamp of the MPU start sample indicated in the MPU time stamp descriptor. The absolute value of DTS and PTS can be calculated from the absolute value.

Note that when the DTS and PTS are calculated by analyzing the encoded data, the receiving device may use the SEI information included in the access unit to calculate.

<Initialization information (parameter set)>
[For video]
For video, the parameter set is stored in the sample data. Further, when MPU metadata and MF metadata are not transmitted, it is guaranteed that the parameter set necessary for decoding can be acquired by referring to only the sample data.

Further, as in (a) and (d) of FIG. 21, when the MPU metadata is transmitted before the media data, it may be specified that the parameter entry is not stored in the SampleEntry. In this case, the receiving device does not refer to the parameter set of SampleEntry but only the parameter set in the sample.

Further, when the MPU metadata is transmitted before the media data, the SampleEntry stores the parameter set common to the MPU and the default parameter set, and the receiving device stores the parameter set of the SampleEntry and the parameter set in the sample. You may refer. By storing the parameter set in the SampleEntry, it is possible to perform decoding even with a conventional receiving device that cannot be played back unless the parameter set exists in the SampleEntry.

[For audio]
For audio, the LATM header is required for decoding, and for MP4 it is essential that the LATM header be included in the sample entry. However, when the header information is not transmitted, it is difficult for the receiving device to acquire the LATM header, and therefore the LATM header is separately included in the control information such as SI. Note that the LATM header may be included in the message, table, or descriptor. The LATM header may be included in the sample.

The receiving apparatus acquires the LATM header from SI or the like before starting the decoding and starts decoding the audio. Alternatively, as shown in (a) of FIG. 21 and (d) of FIG. 21, when the MPU metadata is transmitted before the media data, the receiving device receives the LATM header before the media data. It is possible. Therefore, when the MPU metadata is transmitted before the media data, the decoding can be performed even by using the conventional receiving device.

<Other>
The transmission order and the type of the transmission order may be notified as control information such as MMT packet header and payload header, MPT and other tables, messages, and descriptors. The types of transmission order here are, for example, four types of transmission orders of (a) to (d) of FIG. 21, and an identifier for identifying each type can be acquired before the start of decoding. It just needs to be stored in a place.

Also, as the type of transmission order, different types may be used for audio and video, or a common type may be used for audio and video. Specifically, for example, audio is transmitted in the order of MPU metadata, MF metadata, and media data, as shown in (a) of FIG. 21, and video is shown in (d) of FIG. Thus, MPU metadata, media data, and MF metadata may be transmitted in this order.

With the method described above, the receiving device can perform decoding without using header information. Further, when the MPU metadata is transmitted before the media data ((a) of FIG. 21 and (d) of FIG. 21), the conventional receiving device can also perform decoding.

In particular, since the MF metadata is transmitted after the media data ((d) in FIG. 21), the delay due to the encapsulation does not occur and the conventional receiving device can also perform the decoding.

[Configuration and operation of transmitter]
Next, the configuration and operation of the transmitter will be described. FIG. 23 is a block diagram of a transmission apparatus according to Embodiment 2, and FIG. 24 is a flowchart of a transmission method according to Embodiment 2.

As shown in FIG. 23, the transmission device 15 includes an encoding unit 16, a multiplexing unit 17, and a transmission unit 18.

The encoding unit 16 encodes the video or audio to be encoded into, for example, H.264. Encoding data is generated by encoding according to H.265 (S10).

The multiplexing unit 17 multiplexes (packets) the encoded data generated by the encoding unit 16 (S11). Specifically, the multiplexing unit 17 packetizes each of the sample data, the MPU metadata, and the MF metadata that form the MP4 format file. The sample data is data in which a video signal or an audio signal is coded, the MPU metadata is an example of the first metadata, and the MF metadata is an example of the second metadata. Both the first metadata and the second metadata are metadata used for decoding the sample data, but the difference between them is that the second metadata can be generated only after the sample data is generated. Is to be included.

Here, the data that can be generated only after the generation of the sample data is, for example, data other than the sample data stored in mdat in the MP4 format (data in the header of mdat. That is, type and length shown in FIG. 20. ). Here, the second metadata may include at least a part of the length of the data.

The transmission unit 18 transmits the packetized MP4 format file (S12). The transmission unit 18 transmits the MP4 format file by the method shown in (d) of FIG. 21, for example. That is, the packetized MPU metadata, the packetized sample data, and the packetized MF metadata are transmitted in this order.

Each of the encoding unit 16, the multiplexing unit 17, and the transmitting unit 18 is realized by a microcomputer, a processor, a dedicated circuit, or the like.

[Receiver configuration]
Next, the configuration and operation of the receiving device will be described. FIG. 25 is a block diagram of the receiving apparatus according to the second embodiment.

As illustrated in FIG. 25, the receiving device 20 includes a packet filtering unit 21, a transmission order type determination unit 22, a random access unit 23, a control information acquisition unit 24, a data acquisition unit 25, PTS, and DTS calculation. A unit 26, an initialization information acquisition unit 27, a decoding instruction unit 28, a decoding unit 29, and a presentation unit 30 are provided.

[Operation 1 of receiving device]
First, the operation for the receiving device 20 to specify the MPU head position and the NAL unit position when the medium is video will be described. FIG. 26 is a flowchart of such an operation of the receiving device 20. Here, it is assumed that the transmission order type of MPU data is stored in the SI information by the transmission device 15 (multiplexing unit 17).

First, the packet filtering unit 21 performs packet filtering on the received file. The transmission order type determination unit 22 analyzes the SI information obtained by packet filtering and acquires the transmission order type of MPU data (S21).

Next, the transmission order type determination unit 22 determines (determines) whether or not the data after packet filtering includes MPU header information (at least one of MPU metadata and MF metadata) (S22). When the MPU header information (is included (Yes in S22), the random access unit 23 identifies the MPU head sample by detecting that the fragment type of the MMT payload header is switched to the media data (S23). ).

On the other hand, when the MPU header information is not included (No in S22), the random access unit 23 specifies the MPU head sample based on the RAP_flag of the MMT packet header or the sample number of the MMT payload header (S24).

Further, the transmission order type determination unit 22 determines whether the packet-filtered data includes MF metadata (S25). When it is determined that the MF metadata is included (Yes in S25), the data acquisition unit 25 reads the NAL unit based on the sample, the subsample offset, and the size information included in the MF metadata. Thus, the NAL unit is acquired (S26). On the other hand, when it is determined that the MF metadata is not included (No in S25), the data acquisition unit 25 reads the data of the NAL unit size in order from the head NAL unit of the sample, thereby determining the NAL unit. It is acquired (S27).

Even when it is determined that the MPU header information is included in step S22, the receiving device 20 may specify the MPU head sample by using the process of step S24 instead of step S23. Further, when it is determined that the MPU header information is included, the process of step S23 and the process of step S24 may be used together.

Further, the receiving device 20 may acquire the NAL unit by using the process of step S27 without using the process of step S26 even when it is determined in step S25 that the MF metadata is included. Further, when it is determined that the MF metadata is included, the process of step S23 and the process of step S24 may be used together.

Further, it is assumed that it is determined in step S25 that the MF metadata is included, and that the MF data is transmitted after the media data. In this case, the receiving device 20 may buffer the media data and wait until the MF metadata is acquired before performing the process of step S26. Alternatively, the receiving device 20 does not wait for the acquisition of the MF metadata. It may be determined whether or not the process of step S27 is performed.

For example, the reception device 20 may determine whether to wait for acquisition of the MF metadata based on whether or not the reception device 20 has a buffer having a buffer size capable of buffering the media data. In addition, the receiving device 20 may determine whether to wait for acquisition of the MF metadata based on whether the end-to-end delay becomes small. The receiving device 20 may mainly perform the decoding process by using the process of step S26, and may use the process of step S27 in the case of the processing mode when a packet loss or the like occurs.

If the transmission order type is predetermined, steps S22 and S26 may be omitted, and in this case, the reception device 20 considers the buffer size and the End-to-End delay and the MPU. The identification method of the leading sample and the identification method of the NAL unit may be determined.

When the transmission order type is known in advance, the transmission order type determination unit 22 is unnecessary in the receiving device 20.

Although not described in FIG. 26, the decoding instruction unit 28 acquires data based on PTS, PTS and DTS calculated by the DTS calculation unit 26, and initialization information acquired by the initialization information acquisition unit 27. The data acquired by the unit is output to the decoding unit 29. The decryption unit 29 decrypts the data, and the presentation unit 30 presents the decrypted data.

[Operation 2 of receiving device]
Next, an operation in which the receiving device 20 acquires the initialization information based on the transmission order type and decodes the media data based on the initialization information will be described. FIG. 27 is a flowchart of such an operation.

First, the packet filtering unit 21 performs packet filtering on the received file. The transmission order type determination unit 22 analyzes the SI information obtained by packet filtering and acquires the transmission order type (S301).

Next, the transmission order type determination unit 22 determines whether MPU metadata is transmitted (S302). When it is determined that the MPU metadata is transmitted (Yes in S302), the transmission order type determination unit 22 determines whether the MPU metadata is transmitted before the media data as a result of the analysis in step S301. Yes (S303). When the MPU metadata has been transmitted before the media data (Yes in S303), the initialization information acquisition unit 27 is based on the common initialization information included in the MPU metadata and the initialization information of the sample data. And decrypts the media data (S304).

On the other hand, when it is determined that the MPU metadata is transmitted after the media data (No in S303), the data acquisition unit 25 buffers the media data until the MPU metadata is acquired (S305). , MPU metadata is acquired, and then the process of step S304 is performed.

If it is determined in step S302 that MPU metadata has not been transmitted (No in S302), the initialization information acquisition unit 27 decodes the media data based on only the initialization information of the sample data. (S306).

Note that if the decoding of the media data is guaranteed only on the basis of the initialization information of the sample data on the transmission side, the process of step S306 is used without performing the process based on the determination of steps S302 and S303.

In addition, the receiving apparatus 20 may determine whether to buffer the media data before step S305. In this case, if the receiving device 20 determines that the media data is buffered, the process proceeds to step S305, and if it is determined that the media data is not buffered, the receiving device 20 proceeds to the process of step S306. The determination as to whether or not to buffer the media data may be performed based on the buffer size and the occupancy of the receiving device 20. For example, the one with a smaller End-to-End delay is selected, or the End. The determination may be performed in consideration of the -to-End delay.

[Operation 3 of receiving device]
Here, the details of the transmission method and the reception method in the case where the MF metadata is transmitted after the media data ((c) of FIG. 21 and (d) of FIG. 21) will be described. In the following, the case of FIG. 21D will be described as an example. It should be noted that only the method of (d) of FIG. 21 is used for transmission, and transmission order type signaling is not performed.

As described above, as shown in (d) of FIG. 21, when data is transmitted in the order of MPU metadata, media data, and MF metadata,
(D-1) The receiving device 20 decodes the media data after acquiring the MPU metadata and further acquiring the MF metadata.
(D-2) The receiving device 20, after acquiring the MPU metadata, decodes the media data without using the MF metadata.
The following two decoding methods are possible.

Here, D-1 requires buffering of media data for obtaining MF metadata, but since decoding can be performed using MPU header information, it can be decoded by a conventional MP4 compliant receiving device. Becomes Further, D-2 does not require buffering of media data for obtaining MF metadata, but cannot be decrypted using MF metadata, so special processing is required for decryption.

Further, the method of (d) of FIG. 21 has an advantage that the MF metadata is transmitted after the media data, so that the delay due to encapsulation does not occur and the end-to-end delay can be reduced.

The receiving device 20 can select the above two decoding methods according to the capability of the receiving device 20 and the quality of service provided by the receiving device 20.

In the decoding operation in the receiving device 20, the transmitting device 15 must ensure that the decoding can be performed while reducing the occurrence of buffer overflow and underflow. The following parameters, for example, can be used as the elements for defining the decoder model when decoding is performed using the D-1 method.

-Buffer size for reconfiguring MPU (MPU buffer)
For example, buffer size=maximum rate×maximum MPU time×α, and the maximum rate is the profile of encoded data, the upper limit rate of the level+the overhead of the MPU header. The maximum MPU time is the maximum time length of GOP when 1 MPU=1 GOP (video).

Here, the audio may be a GOP unit common to the above video or another unit. α is a margin for preventing overflow, and may be multiplied or added to the maximum rate×maximum MPU time. In case of multiplication, α≧1, and in case of addition, α≧0.

-Upper limit of decoding delay time from data input to MPU buffer to decoding (TSTD_delay in STD of MPEG-TS)
For example, at the time of transmission, the DTS is set so that the MPU data acquisition completion time <=DTS at the receiver, taking into consideration the maximum MPU time and the upper limit of the decoding delay time.

Further, the transmission device 15 may add DTS and PTS according to the decoder model when decoding is performed using the method of D-1. As a result, the transmission device 15 guarantees the decoding of the reception device that performs the decoding using the method of D-1, and at the same time transmits the auxiliary information necessary when the decoding is performed using the method of D-2. May be.

For example, the transmitter 15 can guarantee the operation of the receiver that decodes using the method of D-2 by signaling the pre-buffering time in the decoder buffer when decoding using the method of D-2.

The pre-buffering time may be included in SI control information such as a message, a table and a descriptor, or may be included in a header of an MMT packet and an MMT payload. Further, the SEI in the encoded data may be overwritten. The DTS and PTS for decoding using the method of D-1 are stored in the MPU time stamp descriptor, SampleEntry, and the DTS and PTS for decoding using the method of D-2, or the pre-buffering time. It may be described in SEI.

When the receiving device 20 supports only the MP4 compliant decoding operation using the MPU header, the receiving device 20 selects the decoding method D-1 and supports both D-1 and D-2. If either of them is present, either one may be selected.

The transmission device 15 may add DTS and PTS so as to guarantee the decoding operation of one (D-1 in this description), and may further transmit auxiliary information for assisting the decoding operation of one.

In addition, when the method of D-2 is used, compared with the case of using the method of D-1, the delay caused by the prebuffering of the MF metadata may increase the end-to-end delay. Is high. Therefore, when it is desired to reduce the End-to-End delay, the receiving device 20 may select the method of D-2 and perform decoding. For example, the receiving device 20 may always use the method of D-2 when always wanting to reduce the end-to-end delay. Further, the receiving apparatus 20 may use the method of D-2 only when it is operating in the low-delay presentation mode in which it is desired to present with low delay such as live content, channel selection, and zapping operation.

FIG. 28 is a flowchart of such a receiving method.

First, the receiving device 20 receives an MMT packet and acquires MPU data (S401). Then, the reception device 20 (transmission order type determination unit 22) determines whether to present the program in the low delay presentation mode (S402).

When the program is not presented in the low delay presentation mode (No in S402), the reception device 20 (random access unit 23 and initialization information acquisition unit 27) acquires random access and initialization information using the header information (S405). ). Further, the reception device 20 (PTS, DTS calculation unit 26, decoding instruction unit 28, decoding unit 29, presentation unit 30) performs decoding and presentation processing based on the PTS and DTS provided on the transmission side (S406).

On the other hand, when presenting the program in the low-delay presentation mode (Yes in S402), the receiving device 20 (random access unit 23 and initialization information acquisition unit 27) uses the decoding method that does not use header information to perform random access, Initialization information is acquired (S403). In addition, the receiving device 20 performs the decoding and presentation process based on the auxiliary information for decoding without using the PTS, DTS, and header information added on the transmitting side (S404). Note that in steps S403 and S404, processing may be performed using MPU metadata.

[Transmission/reception method using auxiliary data]
The transmission/reception operation when the MF metadata is transmitted after the media data ((c) of FIG. 21 and (d) of FIG. 21) has been described above. Next, a method will be described in which the transmission device 15 transmits auxiliary data having a part of the function of MF metadata so that decoding can be started earlier and End-to-End delay can be reduced. Here, an example will be described in which auxiliary data is further transmitted based on the transmission method shown in FIG. 21D, but the method using auxiliary data is shown in FIGS. 21A to 21C. It is also applicable to the transmission method described above.

FIG. 29A is a diagram showing an MMT packet transmitted using the method shown in FIG. 21D. That is, the data is transmitted in the order of MPU metadata, media data, and MF metadata.

Here, sample #1, sample #2, sample #3, and sample #4 are samples included in the media data. It should be noted that although an example in which the media data is stored in the MMT packet in units of samples will be described here, the media data may be stored in the MMT packet in units of NAL units or may be stored in units of divided NAL units. May be done. Note that a plurality of NAL units may be aggregated and stored in the MMT packet.

As described in D-1 above, in the case of the method shown in (d) of FIG. 21, that is, when data is transmitted in the order of MPU metadata, media data, and MF metadata, MPU metadata is transmitted. After the acquisition, there is a method of further acquiring the MF metadata and then decoding the media data. In such a method of D-1, buffering of media data for obtaining MF metadata is required, but since decoding is performed using MPU header information, a conventional MP4 compliant receiving device can also perform D The -1 method has the advantage of being applicable. On the other hand, the receiving device 20 has a drawback that it has to wait for the decoding start until the acquisition of the MF metadata.

On the other hand, as shown in (b) of FIG. 29, in the method using the auxiliary data, the auxiliary data is transmitted before the MF metadata.

The MF metadata includes information indicating DTS, PTS, offset, and size of all samples included in the movie fragment. On the other hand, the auxiliary data includes DTS and PTS of some of the samples included in the movie fragment, and information indicating the offset and size.

For example, the MF metadata includes information on all samples (sample #1 to sample #4), while the auxiliary data includes information on some samples (sample #1 to #2). ..

In the case shown in (b) of FIG. 29, sample #1 and sample #2 can be decoded by using the auxiliary data, and therefore End-to-End is different from the transmission method of D-1. The delay is small. Note that the auxiliary data may include sample information in any combination, and the auxiliary data may be repeatedly transmitted.

For example, in (c) of FIG. 29, when the auxiliary information is transmitted at the timing of A, the transmitting device 15 includes the information of sample #1 in the auxiliary information, and when transmitting the auxiliary information at the timing of B, Information of sample #1 and sample #2 is included in the auxiliary information. When the transmitter 15 transmits the auxiliary information at the timing of C, the auxiliary information includes the information of sample #1, sample #2, and sample #3.

It should be noted that the MF metadata includes information on sample #1, sample #2, sample #3, and sample #4 (information on all samples in the movie fragment).

The auxiliary data does not necessarily have to be sent immediately after being generated.

The header of the MMT packet or MMT payload specifies the type indicating that auxiliary data is stored.

For example, when the auxiliary data is stored in the MMT payload using the MPU mode, the data type indicating the auxiliary data is specified as the fragment_type field value (for example, FT=3). The auxiliary data may be data based on the structure of moof, or may have another structure.

When the auxiliary data is stored in the MMT payload as a control signal (descriptor, table, message), a descriptor tag indicating the auxiliary data, a table ID, a message ID, and the like are specified.

Further, the PTS or DTS may be stored in the header of the MMT packet or the MMT payload.

[Example of generating auxiliary data]
Hereinafter, an example in which the transmission device generates auxiliary data based on the moof configuration will be described. FIG. 30 is a diagram for explaining an example in which the transmission device generates auxiliary data based on the moof configuration.

In normal MP4, moof is created for a movie fragment as shown in FIG. The moof includes information indicating DTS, PTS, offset, and size of the sample included in the movie fragment.

Here, the transmission device 15 forms an MP4 (MP4 file) by using only a part of the sample data included in the MPU, and generates auxiliary data.

For example, as illustrated in (a) of FIG. 30, the transmission device 15 generates MP4 using only sample #1 among samples #1 to #4 constituting the MPU, and includes the moof+mdat header among them. Use as auxiliary data.

Next, as illustrated in (b) of FIG. 30, the transmission device 15 generates MP4 using sample #1 and sample #2 of samples #1 to #4 forming the MPU, and among them, The header of moof+mdat is used as the next auxiliary data.

Next, as illustrated in (c) of FIG. 30, the transmission device 15 uses the sample #1, the sample #2, and the sample #3 among the samples #1 to #4 forming the MPU to perform the MP4. The header of moof+mdat is generated as the next auxiliary data.

Next, as illustrated in (d) of FIG. 30, the transmission device 15 generates all MP4's of the samples #1 to #4 constituting the MPU, of which the moof+mdat header is the movie fragment metadata. Becomes

Note that here, the transmission device 15 generates the auxiliary data for each sample, but the auxiliary data may be generated for every N samples. The value of N is an arbitrary number, and for example, when transmitting auxiliary data M times when transmitting one MPU, N may be set to N=all samples/M.

The information indicating the offset of the sample in the moof may be the offset value after the sample entry area of the subsequent sample number is secured as the NULL area.

The auxiliary data may be generated so that the MF metadata is fragmented.

[Example of reception operation using auxiliary data]
Reception of the auxiliary data generated as described with reference to FIG. 30 will be described. FIG. 31 is a diagram for explaining reception of auxiliary data. Note that in FIG. 31A, the number of samples forming the MPU is 30, and auxiliary data is generated and transmitted every 10 samples.

In FIG. 30A, samples #1 to #10 are used as auxiliary data #1, samples #1 to #20 are used as auxiliary data #2, and samples #1 to #30 are used as MF metadata. Each information is included.

Note that the samples #1 to #10, the samples #11 to #20, and the samples #21 to #30 are stored in one MMT payload, but they may be stored in sample units or NAL units, or in fragments. Alternatively, the data may be stored in an aggregated unit.

The receiving device 20 receives the packets of MPU meta, sample, MF meta, and auxiliary data, respectively.

The receiving device 20 concatenates the sample data in the order of reception (backward), and after receiving the latest auxiliary data, updates the auxiliary data so far. Further, the receiving device 20 can configure a complete MPU by finally replacing the auxiliary data with the MF metadata.

When receiving the auxiliary data #1, the receiving device 20 concatenates the data as shown in the upper part of FIG. As a result, the receiving apparatus 20 can parse the samples #1 to #10 using the information of the MPU metadata and the auxiliary data #1, and the information of the PTS, DTS, offset, and size included in the auxiliary data. Decoding can be performed based on

Further, when the receiving device 20 receives the auxiliary data #2, the receiving device 20 concatenates the data as shown in the middle part of FIG. As a result, the receiving device 20 can parse the samples #1 to #20 by using the information of the MPU metadata and the auxiliary data #2, and the information of PTS, DTS, offset, and size included in the auxiliary data can be obtained. Decoding can be performed based on this.

Further, when the receiving device 20 receives the MF metadata, the receiving device 20 concatenates the data as shown in the lower part of FIG. As a result, the receiving device 20 can parse the samples #1 to #30 using the MPU metadata and the MF metadata, and based on the PTS, DTS, offset, and size information included in the MF metadata. Can be decrypted.

If there is no auxiliary data, the receiving device 20 can obtain the sample information only after receiving the MF metadata, and thus it is necessary to start the decoding after receiving the MF metadata. However, since the transmitting device 15 generates and transmits the auxiliary data, the receiving device 20 can acquire the sample information using the auxiliary data without waiting for the reception of the MF metadata, so that the decoding start time is shortened. be able to. Furthermore, since the transmission device 15 generates auxiliary data based on the moof described with reference to FIG. 30, the reception device 20 can use the conventional MP4 parser as it is and perform parsing.

In addition, the auxiliary data and the MF metadata that are newly generated include information of a sample that overlaps with the auxiliary data transmitted in the past. Therefore, even if past auxiliary data cannot be acquired due to packet loss or the like, MP4 is reconfigured by using newly acquired auxiliary data and MF metadata, and sample information (PTS, DTS, size, And offset) can be obtained.

The auxiliary data does not necessarily need to include information on past sample data. For example, the auxiliary data #1 may correspond to the sample data #1 to #10, and the auxiliary data #2 may correspond to the sample data #11 to #20. For example, as illustrated in (c) of FIG. 31, the transmission device 15 may sequentially transmit complete MF metadata as a data unit and a unit in which the data unit is fragmented as auxiliary data.

Further, the transmission device 15 may repeatedly transmit the auxiliary data or the MF metadata repeatedly as a measure against packet loss.

The MMT packet and the MMT payload in which the auxiliary data is stored include the MPU sequence number and the asset ID, like the MPU metadata, the MF metadata, and the sample data.

The reception operation using the above auxiliary data will be described with reference to the flowchart of FIG. FIG. 32 is a flowchart of the receiving operation using the auxiliary data.

First, the receiving device 20 receives the MMT packet and analyzes the packet header and the payload header (S501). Next, the receiving device 20 analyzes whether the fragment type is auxiliary data or MF metadata (S502), and when the fragment type is auxiliary data, overwrites and updates the past auxiliary data (S503). .. At this time, if there is no past auxiliary data of the same MPU, the receiving device 20 uses the received auxiliary data as new auxiliary data as it is. Then, the reception device 20 acquires a sample and decodes it based on the MPU metadata, the auxiliary data, and the sample data (S507).

On the other hand, when the fragment type is MF metadata, the receiving device 20 overwrites the past auxiliary data with the MF metadata in step S505 (S505). Then, the receiving device 20 acquires the sample in the form of a complete MPU based on the MPU metadata, the MF metadata, and the sample data, and performs decoding (S506).

Although not shown in FIG. 32, in step S502, the receiving device 20 stores the data in the buffer when the fragment type is MPU metadata, and when the fragment type is sample data, the receiving device 20 adds the data to the back of each sample. Store the concatenated data in the buffer.

When the auxiliary data cannot be acquired due to packet loss, the receiving apparatus 20 can overwrite the latest auxiliary data or decode the sample by using past auxiliary data.

The auxiliary data transmission cycle and the number of times of transmission may be predetermined values. Information on the transmission cycle and the number of times (count, countdown) may be transmitted together with the data. For example, the data unit header may store a sending cycle, a sending count, and a time stamp such as initial_cpb_removal_delay.

By transmitting the auxiliary data including the information of the first sample of the MPU at least once before the initial_cpb_removal_delay, it becomes possible to follow the CPB buffer model. At this time, a value based on the picture timing SEI is stored in the MPU time stamp descriptor.

It should be noted that the transmission method in the receiving operation using such auxiliary data is not limited to the MMT method, and is applicable to cases such as MPEG-DASH for streaming transmission of packets configured in the ISOBMFF file format.

[Transmission method when one MPU is composed of multiple movie fragments]
In the description of FIG. 19 and subsequent figures, one MPU is composed of one movie fragment, but here, a case where one MPU is composed of a plurality of movie fragments will be described. FIG. 33 is a diagram showing the structure of an MPU including a plurality of movie fragments.

In FIG. 33, the samples (#1 to #6) stored in one MPU are stored separately in two movie fragments. The first movie fragment is generated based on the samples #1 to #3, and the corresponding moof box is generated. The second movie fragment is generated based on samples #4-#6, and the corresponding moof box is generated.

The headers of the moof box and mdat box in the first movie fragment are stored in the MMT payload and MMT packet as movie fragment metadata #1. On the other hand, the headers of the moof box and mdat box in the second movie fragment are stored in the MMT payload and MMT packet as movie fragment metadata #2. Note that, in FIG. 33, the MMT payload in which the movie fragment metadata is stored is hatched.

Note that the number of samples that make up the MPU and the number of samples that make up the movie fragment are arbitrary. For example, two movie fragments may be configured with the number of samples constituting the MPU as the number of samples in GOP units and the number of samples as half as the number of GOP units as movie fragments.

Although an example in which one MPU includes two movie fragments (moof box and mdat box) is shown here, one MPU may include two or more movie fragments instead of two. Also, the samples stored in the movie fragment may be divided into any number of samples instead of the equally divided number of samples.

Note that, in FIG. 33, the MPU metadata unit and the MF metadata unit are stored as data units in the MMT payload. However, the transmitter 15 may store units such as ftyp, mmpu, moov, and moof as data units in the MMT payload in data unit units, or may store the data units in fragmented units in the MMT payload. Good. Further, the transmission device 15 may store the data units in the MMT payload in units of aggregation.

Further, in FIG. 33, samples are stored in the MMT payload in sample units. However, the transmission device 15 may configure a data unit in NAL unit units or in a unit in which a plurality of NAL units are collected, instead of in sample units, and store the data unit in the MMT payload in data unit units. Further, the transmission device 15 may store the data unit in the MMT payload in a fragmented unit, or may store the data unit in the MMT payload in an aggregated unit.

In FIG. 33, the MPU is configured in the order of moof#1, mdat#1, moof#2, and mdat#2, and offset is given to moof#1 assuming that the corresponding mdat#1 is behind. There is. However, offset may be added as if mdat#1 precedes moof#1. However, in this case, movie fragment metadata cannot be generated in the form of moof+mdat, and the headers of moof and mdat are transmitted separately.

Next, the transmission order of MMT packets when the MPU having the configuration described in FIG. 33 is transmitted will be described. FIG. 34 is a diagram for explaining the transmission order of MMT packets.

FIG. 34A shows the transmission order when the MMT packets are transmitted in the MPU configuration order shown in FIG. 34A, specifically, MPU meta, MF meta #1, media data #1 (samples #1 to #3), MF meta #2, media data #2 (samples #4 to #6). ) Is transmitted in this order.

34B, MPU meta, media data #1 (samples #1 to #3), MF meta #1, media data #2 (samples #4 to #6), and MF meta #2 are transmitted in this order. Here is an example:

In FIG. 34C, media data #1 (samples #1 to #3), MPU meta, MF meta #1, media data #2 (samples #4 to #6), and MF meta #2 are transmitted in this order. Here is an example:

The MF meta #1 is generated using samples #1 to #3, and the MF meta #2 is generated using samples #4 to #6. Therefore, when the transmission method of FIG. 34(a) is used, a delay due to encapsulation occurs in the transmission of the sample data.

On the other hand, when the transmission methods of (b) in FIG. 34 and (c) in FIG. 34 are used, the sample can be transmitted without waiting for the generation of the MF meta, so that the delay due to the encapsulation does not occur. The end-to-end delay can be reduced without occurring.

Also, in the transmission order of (a) of FIG. 34, one MPU is divided into a plurality of movie fragments, and the number of samples stored in the MF meta is smaller than that in the case of FIG. The amount of delay due to encapsulation can be made smaller than in the case.

In addition to the method described here, for example, the transmission device 15 may concatenate the MF meta #1 and the MF meta #2 and collectively transmit the MF meta #1 and the MF meta #2 at the end. In this case, MF metas of different movie fragments may be aggregated and stored in one MMT payload. Further, the MF metas of different MPUs may be aggregated together and stored in the MMT payload.

[Reception method when one MPU is composed of multiple movie fragments]
Here, an operation example of the receiving device 20 that receives and decodes the MMT packets transmitted in the transmission order described in FIG. 34B will be described. 35 and 36 are diagrams for explaining such an operation example.

The reception device 20 receives the MMT packets including the MPU meta, the sample, and the MF meta, which are transmitted in the transmission order shown in FIG. 35. The sample data is concatenated in the order of reception.

The receiving device 20 concatenates the data as shown in (1) of FIG. 36 at T1, which is the time when the MF meta #1 is received, to form the MP4. As a result, the receiving device 20 can acquire the samples #1 to #3 based on the MPU metadata and the information of the MF meta #1, and the information of the PTS, DTS, offset, and size included in the MF meta. Decoding can be performed based on

Further, the receiving device 20 concatenates the data to T2, which is the time when the MF meta #2 is received, as shown in (2) of FIG. 36, and configures MP4. Accordingly, the receiving device 20 can acquire the samples #4 to #6 based on the MPU metadata and the information on the MF meta #2, and based on the information on the PTS, DTS, offset, and size of the MF meta. Can be decrypted. In addition, the receiving device 20 concatenates the data as shown in (3) of FIG. 36 and configures MP4 to obtain the samples #1 to #6 based on the information of the MF meta #1 and the MF meta #2. You may get it.

By dividing one MPU into a plurality of movie fragments, the time until the first MF meta is acquired in the MPU is shortened, so that the decoding start time can be shortened. Also, the buffer size for accumulating the samples before decoding can be reduced.

It should be noted that the transmission device 15 has a time period from the transmission (or reception) of the first sample in the movie fragment to the transmission (or reception) of the MF meta corresponding to the movie fragment shorter than the initial_cpb_removal_delay designated by the encoder. The division unit of the movie fragment may be set so that By setting in this way, the receiving buffer can follow the cpb buffer, and low-delay decoding can be realized. In this case, absolute time based on initial_cpb_removal_delay can be used for PTS and DTS.

Further, the transmitting device 15 may divide the movie fragment into equal intervals, or may divide the subsequent movie fragment into shorter intervals than the previous movie fragment. As a result, the receiving apparatus 20 can receive the MF meta including the information of the sample without fail before decoding the sample, and the continuous decoding is possible.

The following two methods can be used to calculate the absolute time of PTS and DTS.

(1) The absolute time of PTS and DTS is determined based on the reception time (T1 or T2) of MF meta #1 or MF meta #2, and the relative time of PTS and DTS included in MF meta.

(2) The absolute time of PTS and DTS is determined based on the absolute time signaled from the transmission side such as MPU time stamp descriptor and the relative time of PTS and DTS included in the MF meta.

Further, (2-A) the absolute time signaled by the transmission device 15 may be an absolute time calculated based on initial_cpb_removal_delay specified by the encoder.

Further, (2-B) the absolute time signaled by the transmission device 15 may be the absolute time calculated based on the predicted value of the reception time of the MF meta.

The MF meta #1 and the MF meta #2 may be repeatedly transmitted. By repeatedly transmitting the MF meta #1 and the MF meta #2, the receiving device 20 can acquire the MF meta again even if the MF meta cannot be acquired due to a packet loss or the like.

An identifier indicating the order of movie fragments can be stored in the payload header of the MFU containing the samples that make up the movie fragment. On the other hand, the MMT payload does not include an identifier indicating the order of MF metas that form a movie fragment. Therefore, the receiving device 20 identifies the order of the MF meta with the packet_sequence_number. Alternatively, the transmission device 15 stores an identifier indicating to which movie fragment the MF meta belongs in the control information (message, table, descriptor), MMT header, MMT payload header, or data unit header to perform signaling. You may.

The transmitter 15 transmits the MPU meta, the MF meta, and the sample in a predetermined predetermined transmission order, and the reception device 20 performs the reception process based on the predetermined predetermined transmission order. May be. Further, the transmission device 15 may signal the transmission order, and the reception device 20 may select (determine) the reception process based on the signaling information.

The receiving method as described above will be described with reference to FIG. FIG. 37 is a flowchart of the operation of the receiving method described with reference to FIGS.

First, the receiving device 20 determines (identifies) whether the data included in the payload is MPU metadata, MF metadata, or sample data (MFU) based on the fragment type indicated in the MMT payload (S601). , S602). If the data is sample data, the receiving device 20 buffers the sample, and waits for reception of MF metadata corresponding to the sample and start of decoding (S603).

On the other hand, in step S602, when the data is MF metadata, the reception device 20 acquires the sample information (PTS, DTS, position information, and size) from the MF metadata, and uses the acquired sample information as the acquired sample information. A sample is acquired based on this, and the sample is decoded and presented based on PTS and DTS (S604).

Although not shown, when the data is MPU metadata, the MPU metadata includes initialization information necessary for decoding. Therefore, the receiving device 20 accumulates this and uses it for decoding the sample data in step S604.

When the receiving device 20 stores the received MPU data (MPU metadata, MF metadata, and sample data) in the storage device, the receiving device 20 rearranges the MPU configuration described in FIG. 19 or FIG. 33. And then accumulate.

On the transmitting side, a packet sequence number is given to the MMT packet for packets having the same packet ID. At this time, the MMT packets including the MPU metadata, the MF metadata, and the sample data may be assigned the packet sequence numbers after being sorted in the transmission order, or the packet sequence numbers may be assigned in the order before the sorting. Good.

When the packet sequence numbers are assigned in the order before the rearrangement, the receiving device 20 can rearrange the data in the MPU configuration order based on the packet sequence numbers, which facilitates the storage.

[Method to detect the beginning of access unit and the beginning of slice segment]
A method of detecting the head of the access unit or the head of the slice segment based on the information of the MMT packet header and the MMT payload header will be described.

Here, when non-VCL NAL units (access unit delimiter, VPS, SPS, PPS, SEI, etc.) are collectively stored in the MMT payload as data units, and when non-VCL NAL units are data units, respectively, Two examples of aggregating and storing in one MMT payload are shown.

FIG. 38 is a diagram showing a case where non-VCL NAL units are individually used as data units and are aggregated.

In the case of FIG. 38, the head of the access unit is the head data of the MMT payload including the data unit whose fragment_type value is MFU, whose aggregation_flag value is 1 and whose offset value is 0. At this time, the Fragmentation_indicator value is 0.

Further, in the case of FIG. 38, the head of the slice segment is the MMT packet whose fragment_type value is MFU, and the head data of the MMT payload whose aggregation_flag value is 0 and whose fragmentation_indicator value is 00 or 01.

FIG. 39 is a diagram showing a case where non-VCL NAL units are collectively used as a data unit. The field value of the packet header is as shown in FIG. 17 (or FIG. 18).

In the case of FIG. 39, at the head of the access unit, the head data of the payload in the packet whose Offset value is 0 becomes the head of the access unit.

Further, in the case of FIG. 39, the head of the slice segment is a value different from 0, and the head data of the payload of the packet having the fragmentation indicator value of 00 or 01 is the head of the slice segment.

[Reception processing when packet loss occurs]
Normally, when transmitting MP4 format data in an environment where packet loss occurs, the receiving device 20 restores a packet by ALFEC (Application Layer FEC), packet retransmission control, or the like.

However, if packet loss occurs when AL-FEC is not used in streaming such as broadcasting, the packet cannot be restored.

After the data is lost due to the packet loss, the receiving device 20 needs to restart the decoding of video and audio again. For that purpose, the receiving device 20 needs to detect the head of the access unit or the NAL unit and start decoding from the head of the access unit or the NAL unit.

However, since the start code is not attached to the head of the MP4 format NAL unit, the receiving device 20 cannot detect the head of the access unit or the NAL unit even if the stream is analyzed.

FIG. 40 is a flowchart of the operation of the receiving device 20 when a packet loss occurs.

The receiving device 20 detects packet loss by Packet sequence number in the header of the MMT packet or MMT payload, packet counter, fragment counter, etc. (S701), and determines which packet has disappeared from the front-back relationship (S702). ..

When it is determined that the packet loss has not occurred (No in S702), the receiving device 20 configures the MP4 file and decrypts the access unit or the NAL unit (S703).

When it is determined that the packet loss has occurred (Yes in S702), the receiving device 20 generates an NAL unit corresponding to the NAL unit in which the packet loss has occurred from the dummy data, and configures an MP4 file (S704). When receiving the dummy data in the NAL unit, the receiving device 20 indicates that the type of the NAL unit is the dummy data.

In addition, the receiving device 20 detects the head of the next access unit or NAL unit based on the method described in FIGS. 17, 18, 38, and 39, and inputs it to the decoder from the head data, Decoding can be restarted (S705).

When a packet loss occurs, the receiving device 20 may restart the decoding from the head of the access unit and the NAL unit based on the information detected based on the packet header, or may receive the dummy data NAL unit. The decoding may be restarted from the beginning of the access unit and the NAL unit based on the header information of the reconstructed MP4 file including the.

When accumulating an MP4 file (MPU), the receiving device 20 may separately acquire (accumulate (replace)) packet data (NAL unit, etc.) lost due to packet loss from broadcast or communication.

At this time, when the receiving device 20 acquires the lost packet from the communication, the receiving device 20 obtains information on the lost packet (packet ID, MPU sequence number, packet sequence number, IP data flow number, IP address, etc.) from the server. Is notified and the packet is acquired. The receiving device 20 may acquire not only the lost packets but also the packet groups before and after the lost packets at the same time.

[How to configure movie fragment]
Here, a method of configuring a movie fragment will be described in detail.

As described in FIG. 33, the number of samples forming a movie fragment and the number of movie fragments forming one MPU are arbitrary. For example, the number of samples that make up a movie fragment and the number of movie fragments that make up one MPU may be a fixed number that is fixed or may be dynamically determined.

Here, since the movie fragment is configured on the transmitting side (transmitting device 15) so as to satisfy the following conditions, low-delay decoding in the receiving device 20 can be guaranteed.

The conditions are as follows.

The transmitting device 15 receives the sample data so that the receiving device 20 can receive the MF meta including the information of the sample without fail before the decoding time (DTS(i)) of the arbitrary sample (Sample(i)). The MF meta is generated and transmitted with the divided unit as a movie fragment.

Specifically, the transmission device 15 forms a movie fragment by using samples (including the i-th sample) that have been encoded before DTS(i).

As a method for dynamically determining the number of samples forming a movie fragment and the number of movie fragments forming one MPU so as to guarantee low-delay decoding, for example, the following method is used.

(1) At the start of decoding, the decoding time DTS(0) of the sample Sample(0) at the beginning of the GOP is the time based on initial_cpb_removal_delay. The transmitting device configures the first movie fragment by using the already-encoded samples at a time before DTS(0). The transmitter 15 also generates MF metadata corresponding to the first movie fragment and transmits the MF metadata at a time before DTS(0).

(2) The transmission device 15 configures the movie fragment so that the following samples also satisfy the above condition.

For example, when the first sample of the movie fragment is the kth sample, the MF meta of the movie fragment including the kth sample is transmitted by the decoding time DTS(k) of the kth sample. When the coding completion time of the l-th sample is before DTS(k) and the coding completion time of the (l+1)-th sample is after DTS(k), the transmission device 15 determines the k-th The movie fragment is constructed by using the l-th sample from the sample.

Note that the transmission device 15 may configure a movie fragment by using the kth sample to the lth less sample.

(3) After the encoding of the last sample of the MPU is completed, the transmission device 15 forms a movie fragment using the remaining samples, generates MF metadata corresponding to the movie fragment, and transmits the MF metadata.

Note that the transmission device 15 may configure the movie fragment by using some of the samples that have been encoded, instead of configuring the movie fragment by using all the samples that have been encoded.

In the above, an example is shown in which the number of samples constituting a movie fragment and the number of movie fragments constituting one MPU are dynamically determined based on the above conditions so as to guarantee low-delay decoding. It was However, the method of determining the number of samples and the number of movie fragments is not limited to such a method. For example, the number of movie fragments constituting one MPU may be fixed to a predetermined value, and the number of samples may be determined so as to satisfy the above condition. Further, the number of movie fragments forming one MPU and the time (or the code amount of the movie fragment) at which the movie fragment is divided may be fixed to predetermined values, and the number of samples may be determined so as to satisfy the above condition.

Further, when the MPU is divided into a plurality of movie fragments, information indicating whether the MPU is divided into a plurality of movie fragments, the attribute of the divided movie fragment, or the attribute of the MF meta for the divided movie fragment. May be transmitted.

Here, the attribute of the movie fragment is information indicating whether the movie fragment is the first movie fragment of the MPU, the last movie fragment of the MPU, or another movie fragment.

The MF meta attribute is the MF meta corresponding to the first movie fragment of the MPU, the MF meta corresponding to the last movie fragment of the MPU, or any other movie fragment. It is information indicating whether or not the MF meta is to be executed.

Note that the transmission device 15 may store the number of samples forming a movie fragment and the number of movie fragments forming one MPU as control information and transmit the control information.

[Operation of receiving device]
The operation of the receiving device 20 based on the movie fragment configured as described above will be described.

The reception device 20 determines the absolute time of each of the PTS and DTS based on the absolute time signaled from the transmission side such as the MPU time stamp descriptor and the relative time of the PTS and DTS included in the MF meta.

The receiving device 20 performs the following processing based on the information of whether the MPU is divided into a plurality of movie fragments, and when the MPU is divided, based on the attributes of the divided movie fragments. ..

(1) When the movie fragment is the first movie fragment of the MPU, the receiving device 20 determines the absolute time of the PTS of the first sample included in the MPU time stamp descriptor and the relative times of the PTS and DTS included in the MF meta. It is used to generate absolute times for PTS and DTS.

(2) When the movie fragment is not the first movie fragment of the MPU, the receiving device 20 uses the relative time of the PTS and DTS included in the MF meta, without using the information of the MPU time stamp descriptor, and uses the PTS and DTS. Generates the absolute time of.

(3) When the movie fragment is the last movie fragment of the MPU, the receiving device 20 resets the calculation process (addition process of relative time) of PTS and DTS after calculating the absolute time of PTS and DTS of all samples. To do. The reset process may be performed on the movie fragment at the head of the MPU.

The receiving device 20 may determine whether the movie fragment is divided as described below. The receiving device 20 may also acquire the attribute information of the movie fragment as described below.

For example, the receiving device 20 may determine whether or not the division is performed based on the identifier movie_fragment_sequence_number field value indicating the order of movie fragments indicated in the MMTP (MMT Protocol) payload header.

Specifically, when the number of movie fragments included in one MPU is 1, the movie_fragment_sequence_number field value is 1, and the field value has a value of 2 or more, the receiving device 20 has one. It may be determined that the MPU is divided into a plurality of movie fragments.

Further, when the number of movie fragments included in one MPU is 1, the movie_fragment_sequence_number field value is 0, and the field value is a value other than 0, the receiving device 20 receives the MPU. May be determined to be divided into a plurality of movie fragments.

Similarly, the attribute information of a movie fragment may be determined based on movie_fragment_sequence_number.

It should be noted that, without using movie_fragment_sequence_number, whether a movie fragment is divided or attribute information of a movie fragment may be determined by counting the transmission of a movie fragment or MF meta included in one MPU.

With the configurations of the transmitting device 15 and the receiving device 20 as described above, the receiving device 20 can receive the movie fragment metadata at an interval shorter than that of the MPU, and can start decoding with low delay. In addition, it is possible to perform decoding with low delay by using the decoding process based on the MP4 parse method.

The reception operation when the MPU is divided into a plurality of movie fragments as described above will be described using a flowchart. FIG. 41 is a flowchart of the receiving operation when the MPU is divided into a plurality of movie fragments. It should be noted that this flowchart illustrates the operation of step S604 of FIG. 37 in more detail.

First, the receiving device 20 acquires MF metadata based on the data type indicated in the MMTP payload header when the data type is MF meta (S801).

Next, the receiving device 20 determines whether the MPU is divided into a plurality of movie fragments (S802), and when the MPU is divided into a plurality of movie fragments (Yes in S802), the received MF is received. It is determined whether the metadata is the MPU first metadata (S803). When the received MF metadata is the MF metadata at the head of the MPU (Yes in S803), the reception device 20 receives the absolute time of the PTS indicated by the MPU time stamp descriptor and the PTS indicated by the MF metadata. The absolute time of PTS and DTS is calculated from the relative time of DTS (S804), and it is determined whether it is the last metadata of MPU (S805).

On the other hand, if the received MF metadata is not the MF metadata at the head of the MPU (No in S803), the receiving device 20 does not use the information of the MPU time stamp descriptor and the relative PTS and DTS indicated in the MF metadata. The absolute time of PTS and DTS is calculated using the time (S808), and the process proceeds to step S805.

When it is determined in step S805 that the MF metadata is the last MF metadata (Yes in S805), the receiving device 20 resets the calculation process of PTS and DTS after calculating the absolute time of PTS and DTS of all samples. To do. When it is determined in step S805 that it is not the last MF metadata of the MPU (No in S805), the reception device 20 ends the process.

Further, when it is determined in step S802 that the MPU is not divided into a plurality of movie fragments (No in S802), the reception device 20 acquires the sample data based on the MF metadata transmitted after the MPU. Then, PTS and DTS are determined (S807).

Then, although not shown, the receiving device 20 finally performs the decoding process and the presentation process based on the determined PTS and DTS.

[Problems that occur when movie fragments are divided and their solutions]
So far, the method of dividing the movie fragment to shorten the end-to-end delay has been described. From here, a problem newly generated when a movie fragment is divided and its solution will be described.

First, as a background, a picture structure in encoded data will be described. FIG. 42 is a diagram illustrating an example of a prediction structure of a picture in each TemporalId when realizing temporal scalability.

In a coding method such as MPEG-4 AVC or HEVC (High Efficiency Video Coding), temporal scalability (temporal scalability) is realized by using B pictures (bidirectional reference prediction pictures) that can be referenced from other pictures. it can.

The TemporalId shown in (a) of FIG. 42 is an identifier of the hierarchy of the coding structure, and the TemporalId indicates that the larger the value, the deeper the hierarchy. A square block indicates a picture, and Ix in the block is an I picture (intra-picture prediction picture), Px is a P picture (forward reference prediction picture), and Bx and bx are B pictures (bidirectional reference prediction pictures). Show. The x of Ix/Px/Bx indicates the display order, and represents the order of displaying the pictures. Arrows between pictures indicate a reference relationship, and for example, a picture of B4 indicates that a predicted image is generated using I0 and B8 as reference images. Here, one picture is prohibited from using another picture having a TemporalId larger than its own TemporalId as a reference image. Hierarchies are defined for the purpose of providing temporal scalability. For example, in FIG. 42, when all pictures are decoded, 120 fps (frame per second) video is obtained, but only layers from TemporalId 0 to 3 are obtained. Is decoded, a 60 fps image is obtained.

FIG. 43 is a diagram showing the relationship between the decoding time (DTS) and the display time (PTS) in each picture of FIG. For example, the picture I0 shown in FIG. 43 is displayed after the decoding of B4 is completed so that a gap does not occur in the decoding and the display.

As shown in FIG. 43, when the prediction structure includes a B picture, the decoding order and the display order are different, so that the decoding processing and the picture rearrangement after decoding the pictures in the receiving apparatus 20 are performed. (Reorder) processing is required.

The example of the picture prediction structure in the temporal scalability has been described above. However, even when the temporal scalability is not used, depending on the prediction structure, a picture delay process and a reorder process may be necessary. is there. FIG. 44 is a diagram illustrating an example of a picture prediction structure that requires picture delay processing and picture reorder processing. The numbers in FIG. 44 indicate the decoding order.

As shown in FIG. 44, depending on the prediction structure, the first sample in the decoding order may be different from the first sample in the presentation order. In FIG. 44, the first sample in the presentation order is the decoding order. Is the 4th sample. Note that FIG. 44 shows an example of the prediction structure, and the prediction structure is not limited to such a structure. Also in other prediction structures, the first sample in the decoding order may be different from the first sample in the presentation order.

FIG. 45 is a diagram showing an example in which an MPU configured in the MP4 format is divided into a plurality of movie fragments and stored in an MMTP payload and an MMTP packet, similar to FIG. Note that the number of samples that make up the MPU and the number of samples that make up the movie fragment are arbitrary. For example, two movie fragments may be configured with the number of samples constituting the MPU as the number of samples in GOP units and the number of samples as half as the number of GOP units as movie fragments. One sample may be one movie fragment, or the samples forming the MPU may not be divided.

Although FIG. 45 shows an example in which one MPU includes two movie fragments (moof box and mdat box), one MPU does not have to include two movie fragments. The number of movie fragments included in one MPU may be three or more, or may be the number of samples included in the MPU. Further, the samples stored in the movie fragment may be divided into any number of samples instead of the equally divided number of samples.

The movie fragment metadata (MF metadata) includes PTS, DTS, offset, and size information of the sample included in the movie fragment, and the receiving device 20 decodes the sample when decoding the sample. The PTS and DTS are extracted from the MF meta containing the information of (1) to determine the decoding timing and the presentation timing.

From here, for the sake of detailed description, the absolute value of the decoding time of the i sample is described as DTS(i), and the absolute value of the presentation time is described as PTS(i).

The information of the i-th sample in the time stamp information stored in the moof in the MF meta is specifically the relative value of the decoding time of the i-th sample and the (i+1)-th sample, and the i-th sample. Is a relative value of the decoding time and the presentation time of the sample of, and these will be referred to as DT(i) and CT(i) hereinafter.

Movie fragment metadata #1 includes DT(i) and CT(i) of samples #1-#3, and movie fragment metadata #2 includes DT(i of samples #4-#6. ) And CT(i) are included.

Also, the PTS absolute value of the access unit at the head of the MPU is stored in the MPU time stamp descriptor or the like, and the receiving device 20 determines the PTS and DTS based on the PTS_MPU of the access unit at the head of the MPU and CT and DT. calculate.

FIG. 46 is a diagram for explaining the PTS and DTS calculation method and problems when the MPU is configured by the samples #1 to #10.

46A shows an example in which the MPU is not divided into movie fragments, FIG. 46B shows an example in which the MPU is divided into two movie fragments in units of 5 samples, and FIG. ) Shows an example in which the MPU is divided into 10 movie fragments for each sample.

As described in FIG. 45, in the case where PTS and DTS are calculated using the MPU time stamp descriptor and the time stamp information (CT and DT) in MP4, the first sample in the presentation order in FIG. 44. Is the fourth in decoding order. Therefore, the PTS stored in the MPU time stamp descriptor becomes the PTS (absolute value) of the fourth sample in the decoding order. Hereinafter, this sample will be referred to as A sample. Also, the first sample in the decoding order is called a B sample.

Since the absolute time information related to the time stamp is only the information of the MPU time stamp descriptor, the receiving device 20 calculates the PTS (absolute time) and DTS (absolute time) of other samples until the A sample arrives. Can not. The receiving device 20 cannot calculate the PTS and DTS of the B sample.

In the example of FIG. 46A, the A sample is included in the same movie fragment as the B sample and is stored in one MF meta. Therefore, the receiving device 20 can determine the BTS DTS immediately after receiving the MF meta.

In the example of (b) of FIG. 46, the A sample is included in the same movie fragment as the B sample and is stored in one MF meta. Therefore, the receiving device 20 can determine the BTS DTS immediately after receiving the MF meta.

In the example of (c) of FIG. 46, the A sample is included in a movie fragment different from the B sample. Therefore, the receiving apparatus 20 can determine the DTS of the B sample only after receiving the MF meta including the CT and DT of the movie fragment including the A sample.

Therefore, in the case of the example in (c) of FIG. 46, the receiving device 20 cannot start decoding immediately after the arrival of B samples.

As described above, when the A sample is not included in the movie fragment including the B sample, the reception device 20 receives the MF meta related to the movie fragment including the A sample and then decodes the B sample. I can't start.

In the case where the first sample in the presentation order and the first sample in the decoding order do not match, the movie fragment is divided until the A sample and the B sample are no longer stored in the same movie fragment, which causes this problem. To do. In addition, this problem occurs regardless of whether the MF meta is post-feed or post-feed.

In this way, in the case where the first sample in the presentation order does not match the first sample in the decoding order, if the A sample and the B sample are not stored in the same movie fragment, immediately after receiving the B sample, Cannot determine DTS. Therefore, the transmitting device 15 separately transmits information that allows the DTS (absolute value) of the B sample or the DTS (absolute value) of the B sample to be calculated on the receiving side. Such information may be transmitted using control information, a packet header, or the like.

The receiving device 20 calculates the DTS (absolute value) of the B sample using such information. FIG. 47 is a flowchart of the receiving operation when the DTS is calculated using such information.

The receiving device 20 receives the movie fragment at the head of the MPU (S901) and determines whether the A sample and the B sample are stored in the same movie fragment (S902). If they are stored in the same movie fragment (Yes in S902), the reception apparatus 20 does not use the DTS (absolute time) of the B sample, calculates the DTS using only the information of the MF meta, and starts decoding. (S904). Note that in step S904, the receiving apparatus 20 may determine the DTS using the BTS DTS.

On the other hand, when the A sample and the B sample are not stored in the same movie fragment in step S902 (No in S902), the reception device 20 acquires the DTS (absolute time) of the B sample, determines the DTS, and decodes the DTS. Is started (S903).

In the above description, the absolute value of the decoding time and the absolute value of the presentation time of each sample are calculated using MF meta (timestamp information stored in moof in MP4 format) in the MMT standard. Although an example has been described, it goes without saying that the MF meta may be implemented by replacing the absolute value of the decoding time of each sample with arbitrary control information that can be used for calculation of the absolute value of the presentation time. As an example of such control information, the relative value CT(i) of the decoding time of the above-mentioned i-th sample and (i+1)-th sample is calculated as the presentation time of the i-th sample and (i+1)-th sample. The control information replaced with the relative value, the relative value CT(i) of the decoding time of the i-th sample and the (i+1)th sample, and the relative value of the presentation time of the i-th sample and the (i+1)th sample There is control information including both.

(Embodiment 3)
[Overview]
In the third embodiment, a content transmission method and a data structure in the case of transmitting content such as video, audio, subtitles, and data broadcasting by broadcasting will be described. That is, a content transmission method and a data structure specialized for reproduction of a broadcast stream will be described.

In the third embodiment, an example in which the MMT method (hereinafter also simply referred to as MMT) is used as the multiplexing method will be described, but other multiplexing methods such as MPEG-DASH or RTP may be used. ..

First, the details of the method of storing the data unit (DU: Data Unit) in the payload in the MMT will be described. FIG. 48 is a diagram for explaining a method of storing a data unit in a payload in MMT.

In the MMT, the transmitting device stores a part of the data forming the MPU in the MMTP payload as a data unit, attaches a header and transmits. The header includes an MMTP payload header and an MMTP packet header. The unit of the data unit may be the unit of NAL unit or the unit of sample.

FIG. 48A shows an example in which the transmission device aggregates a plurality of data units and stores them in one payload. In the example of (a) of FIG. 48, a data unit header (DUH: Data Unit Header) and a data unit length (DUL: Data Unit Length) are added to the head of each of the plurality of data units, and the data unit header and A plurality of data units to which a data unit length has been added are collectively stored in the payload.

FIG. 48B shows an example in which one data unit is stored in one payload. In the example of FIG. 48B, a data unit header is added to the beginning of the data unit and stored in the payload. FIG. 48(c) shows an example in which one data unit is divided and a data unit header is added to the divided data unit and stored in the payload.

The data unit includes timed-MFU, which is a medium that includes information related to synchronization such as video, audio, or subtitles, non-timed-MFU, which is a medium that does not include information related to synchronization such as a file, MPU metadata, MF metadata, etc. , And the data unit header is determined according to the type of data unit. There is no data unit header in MPU metadata and MF metadata.

In addition, the transmission device cannot, in principle, aggregate data units of different types, but may be defined so that data units of different types can be aggregated. For example, when the size of MF metadata is small, such as when divided into movie fragments for each sample, the number of packets can be reduced by aggregating MF metadata and media data, and further the transmission capacity can be reduced. You can also

When the data unit is MFU, a part of information of MPU such as information for configuring MPU (MP4) is stored as a header.

For example, the header of timed-MFU includes movie_fragment_sequence_number, sample_number, offset, priority, and dependency_counter, and the header of non-timed-MFU includes item_iD. The meaning of each field is shown in standards such as ISO/IEC 23008-1 or ARIB STD-B60. The meaning of each field defined in such a standard will be described below.

movie_fragment_sequence_number indicates the sequence number of the movie fragment to which the MFU belongs, and is also indicated in ISO/IEC14496-12.

sample_number indicates the sample number to which the MFU belongs, and is also shown in ISO/IEC 14496-12.

The offset indicates the offset amount of the MFU in bytes in the sample to which the MFU belongs.

The priority indicates the relative importance of the MFU in the MPU to which the MFU belongs, and the MFU having a large priority number is more important than the MFU having a small priority number.

Dependency_counter indicates the number of MFUs for which the decoding process depends on the MFU (that is, the number of MFUs that cannot be decoded without decoding this MFU). For example, when the BFU or P picture refers to an I picture when the MFU is HEVC, the B picture or P picture cannot be decoded unless the I picture is decoded.

Therefore, when the MFU is a sample unit, the dependency_counter in the MFU of the I picture indicates the number of pictures that refer to the I picture. When the MFU is in units of NAL units, the dependency_counter in the MFU belonging to the I picture indicates the number of NAL units belonging to the picture that refers to the I picture. Furthermore, in the case of a time-direction hierarchically encoded video signal, the MFU of the enhancement layer depends on the MFU of the base layer, so the dependency_counter in the MFU of the base layer indicates the number of MFUs of the enhancement layer. This field can only be created after the number of MFUs it depends on has been determined.

item_iD indicates an identifier that uniquely identifies the item.

[MP4 non-support mode]
As described with reference to FIGS. 19 and 21, as a method for the transmitting device to transmit the MPU in the MMT, a method of transmitting the MPU metadata or the MF metadata before or after the media data, and only the media data are transmitted. There is a way to send. In addition, in the receiving device, there are a method of decoding using a receiving device and a receiving method conforming to MP4, and a method of decoding without using a header.

As a data transmission method specialized for broadcast stream reproduction, for example, there is a transmission method that does not support MP4 reconstruction in the receiving device.

As a transmission method that does not support MP4 reconstruction in the receiving device, for example, there is a method that does not transmit metadata (MPU metadata and MF metadata) as shown in (b) of FIG. In this case, the field value of the fragment type (information indicating the type of data unit) included in the MMTP packet is fixed to 2 (=MFU).

If metadata is not sent, as described above, MP4 compliant receivers cannot decrypt the received data as MP4, but they can be decrypted without using metadata (header). Is.

Therefore, the metadata is not necessarily essential information for decoding and reproducing the broadcast stream. Similarly, the information of the data unit header in the timed-MFU described in FIG. 48 is information for reconfiguring MP4 in the receiving device. Since it is not necessary to reconfigure MP4 in playing the broadcast stream, the information of the data unit header in the timed-MFU (hereinafter, also referred to as the timed-MFU header) is not necessarily the information necessary for playing the broadcast stream.

The receiving device can easily reconfigure MP4 by using the metadata and the information for reconfiguring MP4 in the data unit header (hereinafter, also referred to as MP4 configuration information). However, the receiving device cannot reconfigure MP4 even if only one of the metadata and the MP4 configuration information in the data unit header is transmitted. The merit of transmitting only one of the metadata and the information for reconstructing the MP4 is small, and generating and transmitting unnecessary information causes an increase in processing and a decrease in transmission efficiency.

Therefore, the transmission device controls the data structure and transmission of the MP4 configuration information using the following method. The transmitter determines whether to indicate the MP4 configuration information in the data unit header based on whether the metadata is transmitted. Specifically, the transmitting device indicates the MP4 configuration information in the data unit header when the metadata is transmitted, and does not indicate the MP4 configuration information in the data unit header when the metadata is not transmitted.

As a method of not showing the MP4 configuration information in the data unit header, for example, the following method can be used.

1. The transmission device sets the MP4 configuration information as reserved and does not operate it. As a result, it is possible to reduce the processing amount (the processing amount of the transmission device) on the sending side that generates the MP4 configuration information.

2. The transmitting device deletes the MP4 configuration information and compresses the header. As a result, it is possible to reduce the processing amount on the sending side that generates the MP4 configuration information and reduce the transmission capacity.

When deleting the MP4 configuration information and compressing the header, the transmitting device may indicate a flag indicating that the MP4 configuration information has been deleted (compressed). The flag is indicated in a header (MMTP packet header, MMTP payload header, data unit header) or control information.

Further, the information as to whether or not the metadata is transmitted may be determined in advance, or may be separately signaled to the header or control information and transmitted to the receiving device.

For example, the MFU header may store information as to whether or not the metadata corresponding to the MFU is transmitted.

On the other hand, the receiving device can determine whether the MP4 configuration information is indicated based on whether the metadata is transmitted.

Here, when the transmission order of data (for example, the order of MPU metadata, MF metadata, media data) is determined, the receiving device determines whether the metadata is received before the media data. You may judge it.

When the MP4 configuration information is shown, the receiving apparatus can use the MP4 configuration information for MP4 reconfiguration. Alternatively, the receiving device can use the MP4 configuration information for detecting the heads of other access units and NAL units.

The MP4 configuration information may be the entire timed-MFU header or a part thereof.

Similarly, in the non-timed-MFU header, the transmitter also determines whether the metadata is transmitted in the non-timed-MFU header.
You may decide whether to show id.

The transmitting device may indicate the MP4 configuration information only in one of the timed-MFU and the non-timed-MFU. If only one of them indicates the MP4 configuration information, the transmitting device determines whether to indicate the MP4 configuration information based on whether the metadata is transmitted and whether the metadata is timed-MFU or non-timed-MFU. To do. The receiving device can determine whether the metadata is transmitted and whether the MP4 configuration information is indicated based on the timed/non-timed flag.

In the above description, the transmission device determines whether to show the MP4 configuration information based on whether the metadata (both MPU metadata and MF metadata) is transmitted. However, the transmitting device may not show the MP4 configuration information when a part of the metadata (either one of the MPU metadata and the MF metadata) is not transmitted.

Further, the transmitting device may determine whether to indicate the MP4 configuration information based on information other than the metadata.

For example, a mode such as MP4 support mode/MP4 non-support mode is defined, and the transmission device indicates MP4 configuration information in the data unit header in the case of the MP4 support mode, and data in the case of the MP4 non-support mode. The MP4 configuration information may not be indicated in the unit header. Also, the transmitting device transmits metadata in the MP4 support mode and indicates MP4 configuration information in the data unit header, and in the MP4 non-support mode, transmits the data unit header without transmitting the metadata. In the above, the MP4 configuration information may not be shown.

[Operation flow of transmitter]
Next, the operation flow of the transmission device will be described. FIG. 49 is an operation flow of the transmission device.

The transmitting device first determines whether or not to transmit the metadata (S1001). When the transmitting device determines to transmit the metadata (Yes in S1002), the transmitting device proceeds to step S1003, generates MP4 configuration information, stores it in a header, and transmits it (S1003). In this case, the transmitting device also generates and transmits the metadata.

On the other hand, when the transmitting device determines not to transmit the metadata (No in S1002), the transmitting device does not generate the MP4 configuration information and stores it in the header (S1004). In this case, the transmitting device does not generate the metadata and does not transmit it.

Whether or not the metadata is transmitted in step S1001 may be determined in advance, and it may be determined whether or not the metadata is generated inside the transmitting device and whether the metadata is transmitted inside the transmitting device. It may be determined based on.

[Operation flow of receiving device]
Next, an operation flow of the receiving device will be described. FIG. 50 is an operation flow of the receiving device.

The receiving device first determines whether or not the metadata is transmitted (S1101). Whether or not the metadata is transmitted can be determined by monitoring the fragment type in the MMTP packet payload. Further, whether or not the data is transmitted may be predetermined.

When the receiving device determines that the metadata is transmitted (Yes in S1102), the receiving device reconfigures MP4 and executes the decoding process using the MP4 configuration information (S1103). On the other hand, when it is determined that the metadata has not been transmitted (No in S1102), the decoding process is executed without performing the MP4 reconfiguration process and using the MP4 configuration information (S1104).

The receiving apparatus can detect a random access point, detect the head of an access unit, detect the head of a NAL unit, etc. without using the MP4 configuration information by using the method described above. , Packet loss detection, and packet loss recovery processing can be performed.

For example, the head of the access unit is the head data of the MMT payload having an aggregation_flag value of 1. At this time, the Fragmentation_indicator value is 0.

Further, the head of the slice segment is the head data of the MMT payload having an aggregation_flag value of 0 and a fragmentation_indicator value of 00 or 01.

The receiving device can detect the head of the access unit and the slice segment based on the above information.

In addition, the receiving device analyzes the NAL unit header in the packet including the beginning of the data unit whose fragmentation_indicator value is 00 or 01, the NAL unit type is the AU delimiter, and the NAL unit type is the slice segment. May be detected.

[Broadcast simple mode]
Up to now, a method of not supporting MP4 configuration information in the receiving device has been described as a data transmission method specialized for broadcast stream reproduction, but the data transmission method specialized for broadcast stream reproduction is not limited to this. Absent.

For example, the following method may be used as a method of transmitting data specialized for playing a broadcast stream.

-The transmitter does not have to use AL-FEC in a fixed broadcast reception environment. When AL-FEC is not used, FEC_type in the MMTP packet header is always fixed to 0.

-The transmitter may always use AL-FEC in the mobile reception environment of broadcasting and the communication UDP transmission mode. When AL-FEC is used, FEC_type in the MMTP packet header is always 0 or 1.

-The transmitter does not have to perform bulk transfer of assets. When the bulk transmission of assets is not performed, location_infolocation indicating the number of transmission locations of assets in the MPT may be fixed to 1.

-The transmitter does not have to perform hybrid transmission of assets, programs, and messages.

In addition, for example, when the broadcast simple mode is defined and the transmission device is in the broadcast simple mode, the transmission device sets the MP4 non-support mode, or uses the above-described data transmission method specialized for broadcast stream reproduction. May be Whether to be in the broadcast simple mode may be determined in advance, or the transmitting device may store a flag indicating the broadcast simple mode as control information and transmit it to the receiving device.

In addition, the transmitting apparatus determines, as described above with reference to FIG. 49, whether or not the MP4 non-support mode is used (whether the metadata is transmitted). The data transmission method specialized for the reproduction of the broadcast stream shown in (1) may be used.

In the broadcast simple mode, the receiving device determines that it is in the MP4 non-support mode and can perform the decoding process without reconfiguring MP4.

Further, when in the broadcast simple mode, the receiving device can determine that the function is specialized for broadcasting, and can perform reception processing specialized for broadcasting.

As a result, in the simple broadcast mode, it is possible to reduce unnecessary processing for the transmitting device and the receiving device by using only the function specialized for broadcasting, and to not compress unnecessary information for transmission. As a result, transmission overhead can be reduced.

When the MP4 non-support mode is used, hint information supporting a storage method other than the MP4 configuration may be displayed.

Storage methods other than the MP4 configuration include, for example, a method of directly storing MMT packets and IP packets, a method of converting MMT packets into MPEG-2 TS packets, and the like.

In addition, in the MP4 non-support mode, a format that does not comply with the MP4 configuration may be used.

For example, in the MP4 non-support mode, the data stored in the MFU may be in a format with a byte start code instead of a format with the NAL unit size at the beginning of the NAL unit in MP4 format. Good.

In MMT, an asset type indicating an asset type is described in 4CC registered in MP4REG (http://www.mp4ra.org), and when HEVC is used as a video signal,'HEV1' or'HVC1' is used. To be “HVC1” is a format that may include a parameter set in the sample, and “HEV1” is a format that does not include the parameter set in the sample but includes the parameter set in the sample entry in the MPU metadata.

In the case of the broadcast simple mode or the MP4 non-support mode, when the MPU metadata and the MF metadata are not transmitted, it may be specified that the parameter set should be included in the sample. Further, it may be specified that the asset type always takes the format of “HVC1” regardless of whether “HEV1” or “HVC1” is indicated in the asset type.

[Supplement 1: Transmission device]
As described above, when the metadata is not transmitted, the MP4 configuration information is set to reserved, and the non-operating transmitting device can be configured as shown in FIG. 51. FIG. 51 is a diagram illustrating an example of a specific configuration of the transmission device.

The transmitting device 300 includes an encoding unit 301, an adding unit 302, and a transmitting unit 303. Each of the encoding unit 301, the adding unit 302, and the transmitting unit 303 is realized by, for example, a microcomputer, a processor, a dedicated circuit, or the like.

The encoding unit 301 encodes a video signal or an audio signal to generate sample data. The sample data is specifically a data unit.

The adding unit 302 adds the header information including the MP4 configuration information to the sample data that is the data in which the video signal or the audio signal is encoded. The MP4 configuration information is information for reconstructing the sample data as an MP4 format file on the receiving side, and has different contents depending on whether or not the presentation time of the sample data is set.

As described above, the adding unit 302 adds the movie_fragment_sequence_number, sample_number, offset, priority to the header (header information) of the timed-MFU, which is an example of sample data (sample data including information about synchronization) for which the presentation time is set. , And MP4 configuration information such as dependency_counter.

On the other hand, the assigning unit 302 includes MP4 configuration information such as item_id in the header of the timed-MFU (header information) that is an example of sample data (sample data that does not include information about synchronization) for which the presentation time is not determined. Include.

Then, when the metadata corresponding to the sample data is not transmitted by the transmitting unit 303 (for example, in the case of (b) of FIG. 21), the adding unit 302 determines whether the presentation time of the sample data is set. Depending on whether or not, the header information not including the MP4 configuration information is added to the sample data.

Specifically, when the presentation time of the sample data is set, the adding unit 302 adds the header information not including the first MP4 configuration information to the sample data, and the presentation time of the sample data is set. If not, the header information including the second MP4 configuration information is added to the sample data.

For example, as illustrated in step S1004 of FIG. 49, the adding unit 302 sets the MP4 configuration information to reserved (fixed value) when the metadata corresponding to the sample data is not transmitted by the transmitting unit 303. The MP4 configuration information is not substantially generated and is not substantially stored in the header (header information). Note that the metadata includes MPU metadata and movie fragment metadata.

The transmitting unit 303 transmits the sample data to which the header information is added. More specifically, the transmission unit 303 packetizes the sample data to which the header information is added by the MMT method and transmits the packetized data.

As described above, in the transmission method and the reception method specialized for reproduction of the broadcast stream, it is not necessary to reconfigure the data unit into MP4 in the reception device. When the receiving device does not need to be reconfigured to MP4, processing of the transmitting device is reduced by not generating unnecessary information such as MP4 configuration information.

On the other hand, the transmission device must send necessary information, but it is necessary to maintain consistency with the standard so that extra additional information and the like need not be separately sent.

According to the configuration of the transmitting device 300, by setting the area in which the MP4 configuration information is stored to a fixed value, the MP4 configuration information is not transmitted, and only the necessary information is transmitted based on the standard, which is unnecessary. The effect is that it is not necessary to send additional information. That is, it is possible to reduce the configuration of the transmission device and the processing amount of the transmission device. In addition, since unnecessary data is not transmitted, it is possible to improve transmission efficiency.

[Supplement 2: Receiver]
Further, the receiving device corresponding to the transmitting device 300 may be configured as shown in FIG. 52, for example. FIG. 52 is a diagram illustrating another example of the configuration of the receiving device.

The receiving device 400 includes a receiving unit 401 and a decoding unit 402. The receiving unit 401 and the decoding unit 402 are realized by, for example, a microcomputer, a processor, a dedicated circuit, or the like.

The receiving unit 401 is sample data that is data in which a video signal or an audio signal is coded, and a sample to which header information including MP4 configuration information for reconstructing the sample data as an MP4 format file is added. Receive data.

The decoding unit 402 decodes the sample data without using the MP4 configuration information when the receiving unit does not receive the metadata corresponding to the sample data and the presentation time of the sample data is set. To do.

For example, when the receiving unit 401 does not receive the metadata corresponding to the sample data, as illustrated in step S1104 of FIG. 50, the decoding unit 402 executes the decoding process without using the MP4 configuration information.

As a result, the configuration of the receiving device 400 and the amount of processing in the receiving device 400 can be reduced.

(Embodiment 4)
[Overview]
In the fourth embodiment, a method for storing an asynchronous (non-timed) medium that does not include information on synchronization such as a file in an MPU and a method for transmitting an MMTP packet will be described. In the fourth embodiment, an MPU in MMT will be described as an example, but the same MP4-based DASH is also applicable.

First, details of a method of storing a non-timed medium (hereinafter, also referred to as “asynchronous media data”) in the MPU will be described with reference to FIG. FIG. 53 is a diagram showing a method for storing a non-timed medium in an MPU and a method for transmitting an MMTP packet.

The MPU that stores the non-timed media is composed of boxes such as ftyp, mmpu, moov, and meta, and stores information about files stored in the MPU. A plurality of idat boxes can be stored in the meta box, and one file is stored as an item in the idat box.

Some of the ftyp, mmpu, moov, and meta boxes compose one data unit as MPU metadata, and the item or idat boxes compose a data unit as MFU.

After the data unit is aggregated or fragmented, a data unit header, an MMTP payload header, and an MMTP packet header are added and transmitted as an MMTP packet.

Note that FIG. 53 shows an example in which File #1 and File #2 are stored in one MPU. Although the MPU metadata is not divided and the MFU is divided and stored in the MMTP packet, the MFU metadata is not limited to this, and may be aggregated or fragmented according to the size of the data unit. Also, MPU metadata may not be transmitted, in which case only MFU is transmitted.

The header information such as the data unit header indicates the itemID (identifier that uniquely identifies the item), and the MMTP payload header and the MMTP packet header include the packet sequence number (sequence number for each packet) and the MPU sequence number. (MPU sequence number, unique number in asset.) is included.

The data structure of the MMTP payload header and the MMTP packet header other than the data unit header is the same as that of the timed media (hereinafter, also referred to as “synchronous media data”) described above, and includes aggregation_flag, fragmentation_indicator, fragment_counter, and the like. ..

Next, a specific example of header information when a file (=Item=MFU) is divided and packetized will be described with reference to FIGS. 54 and 55.

54 and 55 are diagrams showing an example in which a plurality of pieces of divided data obtained by dividing a file are packetized and transmitted. 54 and 55, specifically, information (packet sequence number, fragment counter, fragmentation) included in any of the data unit header, the MMTP payload header, and the MMTP packet header, which is the header information of each divided MMTP packet. Indicator, MPU sequence number, item ID). 54 is a diagram showing an example in which File#1 is divided into M (M<=256), and FIG. 55 is an example in which File#2 is divided into N (256<N). FIG.

The divided data number indicates the index of the divided data from the beginning of the file, and this information is not transmitted. That is, the divided data number is not included in the header information. The divided data number is a number given to a packet corresponding to each of a plurality of divided data obtained by dividing a file, and is a number added by 1 in ascending order from the head packet. is there.

The packet sequence number is a sequence number of a packet having the same packet ID. In FIGS. 54 and 55, the divided data at the beginning of the file is A, and consecutive numbers are given up to the divided data at the end of the file. The packet sequence number is a number added by incrementing by 1 from the divided data at the head of the file in ascending order, and is a number corresponding to the divided data number.

The fragment counter indicates the number of a plurality of pieces of divided data after the divided data among a plurality of pieces of divided data obtained by dividing one file. Further, the fragment counter indicates the remainder of dividing the number of divided data by 256 when the number of divided data, which is the number of a plurality of divided data obtained by dividing one file, exceeds 256. In the example of FIG. 54, since the number of divided data is 256 or less, the field value of the fragment counter is (M-divided data number). On the other hand, in the example of FIG. 55, since the number of pieces of divided data exceeds 256, the value is ((N-divided data number)% 256) obtained by dividing (N-divided data number) by 256.

The fragmentation indicator indicates the state of division of the data to be stored in the MMTP packet, and indicates whether the division data is the first division data, the last division data, or other division data in the divided data unit. Alternatively, it is a value indicating whether it is one or more data units that are not divided. Specifically, the fragmentation indicator is “01” for the first divided data, “11” for the last divided data, “10” for the remaining divided data, and “10” for the undivided data unit. 00”.

In the present embodiment, when the number of divided data items exceeds 256, the description will be made assuming that the remainder obtained by dividing the number of divided data items by 256 is shown. However, the number of divided data items is not limited to 256 and may be any other number (predetermined number). May be.

When a file is divided as shown in FIG. 54 and FIG. 55 and conventional header information is added to each of a plurality of pieces of divided data obtained by dividing the file, the received MMTP packet is received by the receiving device. The number of divided data in the original file (divided data number), the number of divided data of the file, or the information that can derive the divided data number and the number of divided data is not available. Therefore, in the conventional transmission method, even if the MMTP packet is received, it is not possible to uniquely detect the division data number or the division data number of the data stored in the received MMTP packet.

For example, when the number of divided data is 256 or less and it is known in advance that the number of divided data is 256 or less as shown in FIG. 54, the divided data number and the number of divided data are referred to by referring to the fragment counter. It is possible to specify. However, if the number of divided data is 256 or more, the divided data number or the number of divided data cannot be specified.

When the number of divided data files is limited to 256 or less, and the data size that can be transmitted in one packet is x [bytes], the maximum size of a file that can be transmitted is limited to x * 256 [bytes]. It For example, x=4k [bytes] is assumed in broadcasting, and in this case, the maximum size of a file that can be transmitted is limited to 4k*256=1M [bytes]. Therefore, when it is desired to transmit a file exceeding 1 [Mbytes], the number of divided data of the file cannot be limited to 256 or less.

Further, for example, since it is possible to detect the first divided data and the last divided data of the file by referring to the fragmentation indicator, the number of MMTP packets is counted until the MMTP packet including the last divided data of the file is received. It is possible to calculate the fragment data number and the number of fragment data by combining with the packet sequence number after receiving the MMTP packet containing the last fragment data of the file. The division data number and the division data number may be signaled by combining with the number. However, when the reception is started from the MMTP packet including the divided data in the middle of the file (that is, the divided data which is neither the first divided data of the file nor the last divided data of the file), the divided data number or the divided data of the divided data The number cannot be specified. The divided data number and the number of divided data of the divided data can be specified only after receiving the MMTP packet including the divided data at the end of the file.

The problem described in FIGS. 54 and 55, that is, the following method for uniquely determining the divided data number and the number of divided data of the divided data of a file when a packet including the divided data of the file is received from the middle To use.

First, the divided data number will be described.

As for the divided data number, the packet sequence number in the first divided data of the file (item) is signaled.

As a signaling method, it is stored in the control information for managing the file. Specifically, the packet sequence number A of the divided data at the head of the file in FIGS. 54 and 55 is stored in the control information. The receiving device acquires the value A from the control information and calculates the divided data number from the packet sequence number indicated in the packet header.

The divided data number of the divided data is obtained by subtracting the packet sequence number A of the first divided data from the packet sequence number of the divided data.

As the control information for managing the file, for example, there is an asset management table defined in ARIB STD-B60. In the asset management table, file size, version information, and the like are shown for each file, which is stored in a data transmission message and transmitted. FIG. 56 is a diagram showing the syntax of a loop for each file in the asset management table.

When the area of the existing asset management table cannot be expanded, signaling may be performed using a partial 32-bit area of the item_info_byte field indicating item information. A flag indicating whether or not the packet sequence number in the first divided data of the file (item) is indicated in a partial area of the item_info_byte may be indicated in, for example, the reserved_future_use field of the control information.

When a file such as a data carousel is repeatedly transmitted, a plurality of packet sequence numbers may be indicated, or a packet sequence number at the beginning of a file to be transmitted immediately after may be indicated.

The information is not limited to the packet sequence number of the divided data at the beginning of the file, and may be information associating the divided data number of the file with the packet sequence number.

Next, the number of pieces of divided data will be described.

The loop order for each file included in the asset management table may be defined as the file transmission order. As a result, since the packet sequence numbers at the heads of two consecutive files in the transmission order are known, by subtracting the head packet sequence number of the file transmitted earlier from the head packet sequence number of the file transmitted later, It is possible to specify the number of pieces of divided data of the transmitted file. That is, for example, when File#1 shown in FIG. 54 and File#2 shown in FIG. 55 are consecutive files in this order, the last packet sequence number of File#1 and the first packet sequence of File#2 are shown. The numbers are consecutive numbers.

Further, the number of pieces of divided data of the file may be specified by defining the method of dividing the file. For example, when the number of divided data is N, the size of each of the 1st to (N-1)th divided data is L, and the size of the Nth divided data is a fraction (item_size-L*(N-1)). By defining, the number of division data can be calculated backward from item_size shown in the asset management table. In this case, the integer value obtained by rounding up (item_size/L) is the number of divided data. The file dividing method is not limited to this.

Further, the number of pieces of divided data may be directly stored in the asset management table.

The receiving device receives the control information by using the above method, and calculates the number of divided data based on the control information. Further, the packet sequence number corresponding to the divided data number of the file can be calculated based on the control information. If the reception timing of the divided data packet is earlier than the reception timing of the control information, the division data number or the division data number may be calculated at the reception timing of the control information.

When the divided data number or the divided data number is signaled using the above method, the divided data number and the divided data number are not specified based on the fragment counter, and the fragment counter becomes unnecessary data. Therefore, in the transmission of asynchronous media, when the divided data number and the information that can specify the number of divided data are signaled using the above method, the fragment counter may not be operated or header compression may be performed. .. As a result, it is possible to reduce the processing amount of the transmission device and the reception device and improve the transmission efficiency. That is, when transmitting asynchronous media, the fragment counter may be reserved (invalidated). Specifically, the value of the fragment counter may be a fixed value of "0", for example. Further, when receiving asynchronous media, the fragment counter may be ignored.

When storing synchronous media such as video and audio, the transmission order of the MMTP packets at the sending device and the arrival order of the MMTP packets at the receiving device match, and the packets are not retransmitted. In such a case, the fragment counter may not be operated if it is not necessary to detect and reconstruct the packet loss. In other words, in this case, the fragment counter may be reserved.

Further, it is possible to detect a random access point, detect the head of an access unit, detect the head of a NAL unit, etc. without using a fragment counter, and perform decoding processing, packet loss detection, and recovery from packet loss. Can be processed.

Further, in the transmission of real-time contents such as live broadcasting, transmission with lower delay is required, and it is required to sequentially packetize and transmit the encoded data. However, in real-time content transmission, the conventional fragment counter cannot determine the number of pieces of divided data when sending the first piece of divided data, so sending of the first piece of divided data is completed after all the data units have been encoded. Only after the number is determined, a delay occurs. Even in such a case, this delay can be reduced by not operating the fragment counter using the above method.

FIG. 57 is an operation flow for identifying a divided data number in the receiving device.

The receiving device acquires the control information in which the file information is described (S1201). The receiving device determines whether the packet sequence number at the beginning of the file is indicated in the control information (S1202), and if the packet sequence number at the beginning of the file is indicated in the control information (Yes in S1202), the file The packet sequence number corresponding to the divided data number of the divided data is calculated (S1203). Then, the receiving device, after acquiring the MMTP packet in which the divided data is stored, specifies the divided data number of the file from the packet sequence number stored in the packet header of the acquired MMTP packet (S1204).

On the other hand, when the packet sequence number at the beginning of the file is not indicated in the control information (No in S1202), the receiving device acquires the MMTP packet including the divided data at the end of the file, and then the packet header of the acquired MMTP packet. The fragment data number is specified by using the fragment indicator stored in (1) and the packet sequence number (S1205).

FIG. 58 is an operation flow for specifying the number of division data in the receiving device.

The receiving device acquires the control information in which the file information is described (S1301). The receiving device determines whether or not the control information includes information capable of calculating the number of divided data items of the file (S1302), and when determining that the control information includes information capable of calculating the number of divided data items, (Yes in S1302), the number of divided data is calculated based on the information included in the control information (S1303). On the other hand, when the receiving device determines that the number of divided data pieces cannot be calculated (No in S1302), the receiving device acquires the MMTP packet including the divided data at the end of the file and then stores the MMTP packet in the packet header of the acquired MMTP packet. The fragmented data number is specified using the fragment indicator and the packet sequence number (S1304).

FIG. 59 is an operation flow for determining whether to operate the fragment counter in the transmission device.

First, the transmission device determines whether the medium to be transmitted (hereinafter, also referred to as “media data”) is a synchronous medium or an asynchronous medium (S1401).

If the result of the determination in step S1401 is synchronous media (synchronous media in S1402), whether the transmitting device has the same MMTP packet order of transmission and reception in the environment for transmitting the synchronous media, and packet reconstruction is not necessary at the time of packet loss. It is determined (S1403). When it is determined that the transmission device is unnecessary (Yes in S1403), the transmission device does not operate the fragment counter (S1404). On the other hand, when the transmitting device determines that it is not necessary (No in S1403), it operates the fragment counter (S1405).

If the result of the determination in step S1401 is asynchronous media (asynchronous media in S1402), the transmitter determines whether the fragment data number or the number of fragment data is signaled using the method described above. Decide whether to operate the counter. Specifically, when the divided data number and the number of divided data are signaled (Yes in S1406), the transmitting device does not operate the fragment counter (S1404). On the other hand, when the divided data number and the divided data number are not signaled (No in S1406), the transmitting device operates the fragment counter (S1405).

When the fragment counter is not operated, the transmitting device may set the value of the fragment counter to reserved or may perform header compression.

Note that the transmitting device may determine whether to signal the above-mentioned divided data number or the number of divided data based on whether to operate the fragment counter.

In addition, when the synchronous medium does not operate the fragment counter, the transmitting device may signal the divided data number or the number of divided data using the method described above in the asynchronous medium. Conversely, the operation of the synchronous media may be determined based on whether the asynchronous media operates the fragment counter. In this case, whether the fragment is operated or not can be the same in the synchronous media and the asynchronous media.

Next, a method of specifying the number of divided data and the divided data number (when utilizing the fragment counter) will be described. FIG. 60 is a diagram for explaining a method of specifying the number of pieces of divided data and the number of divided data (when a fragment counter is used).

As described with reference to FIG. 54, when the number of divided data is 256 or less and it is known in advance that the number of divided data is 256 or less, the fragment data is referred to by the fragment counter and the divided data number is divided. It is possible to specify the number of data.

When the number of divided data of files is limited to 256 or less, when the data size that can be transmitted in one packet is x [bytes], the maximum size of the file that can be transmitted is limited to x*256 [bytes]. For example, x=4k [bytes] is assumed in broadcasting, and in this case, the maximum size of a file that can be transmitted is limited to 4k*256=1M [bytes].

When the file size exceeds the maximum size of the file that can be transmitted, the file is divided in advance so that the size of the divided file becomes x*256 [bytes] or less. Each of the plurality of divided files obtained by dividing the file is treated as one file (item), further divided within 256, and the divided data obtained by the further division are respectively converted into MMTP packets. Stored and transmitted.

Information indicating that the item is a divided file, the number of divided files, and the sequence number of the divided file may be stored in the control information and transmitted to the receiving device. Further, these pieces of information may be stored in the asset management table, or may be indicated by using a part of the existing field item_info_byte.

If the item is one of a plurality of divided files obtained by dividing one file, the receiving device specifies another divided file and reconstructs the original file. You can In addition, the receiving device can uniquely specify the number of divided data and the divided data number by using the number of divided files of the divided file in the control information, the index of the divided file, and the fragment counter. Moreover, the number of divided data and the divided data number can be uniquely specified without using a packet sequence number or the like.

Here, it is desirable that the item_id of each of the plurality of divided files obtained by dividing one file be the same. When another item_id is added, the item_id of the first divided file may be indicated in order to uniquely refer to the file from other control information or the like.

Further, the plurality of divided files may always belong to the same MPU. When storing a plurality of files in the MPU, files of different types may not be stored, and a file obtained by dividing one file may be stored. The receiving device can detect the file update by confirming the version information for each MPU without confirming the version information for each item.

FIG. 61 is an operation flow of the transmission device when utilizing the fragment counter.

First, the transmission device confirms the size of the file to be transmitted (S1501). Next, the transmission device determines whether or not the file size exceeds x*256 [bytes] (x is a data size that can be transmitted in one packet. For example, MTU size) (S1502), and the file size is x. When it exceeds *256 [bytes] (Yes in S1502), the file is divided so that the size of the divided file is less than x*256 [bytes] (S1503). Then, the divided file is transmitted as an item, and information about the divided file (for example, the divided file, the sequence number in the divided file, etc.) is stored in the control information and transmitted (S1504). On the other hand, if the file size is less than x*256 [bytes] (No in S1502), the file is transmitted as an item as usual (S1505).

FIG. 62 is an operation flow of the receiving device when utilizing the fragment counter.

First, the receiving device acquires and analyzes control information related to file transmission such as an asset management table (S1601). Next, the receiving device determines whether the desired item is a divided file (S1602). When the receiving device determines that the desired file is the divided file (Yes in S1602), the receiving device acquires information for reconstructing the file, such as the divided file and the index of the divided file, from the control information (S1603). .. Then, the receiving device acquires the items that form the divided file and reconstructs the original file (S1604). On the other hand, when the receiving device determines that the desired file is not a divided file (No in S1602), the receiving device acquires the file as usual (S1605).

In short, the transmitting device signals the packet sequence number of the divided data at the beginning of the file. Further, the transmission device signals information that can specify the number of pieces of divided data. Alternatively, the transmission device defines a division rule that can specify the number of pieces of divided data. Further, the transmission device performs reserved or header compression without operating the fragment counter.

When the packet sequence number of the data at the beginning of the file is signaled, the receiving device specifies the divided data number and the number of divided data from the packet sequence number of the divided data at the beginning of the file and the packet sequence number of the MMTP packet. ..

From another viewpoint, the transmitting device divides the file, divides the data for each divided file, and transmits the divided data. The information (sequence number, number of divisions, etc.) that links the divided files is signaled.

The receiving device specifies the divided data number and the divided data number based on the fragment counter and the sequence number of the divided file.

Thereby, the divided data number and the divided data can be uniquely specified. Further, since the divided data number of the divided data can be specified when the divided data on the way is received, the waiting time and the memory can be reduced.

In addition, by not operating the fragment counter, the configuration of the transmission/reception device can reduce the processing amount and can improve the transmission efficiency.

FIG. 63 is a diagram showing a service configuration when the same program is transmitted by a plurality of IP data flows. Here, part of the program (video/audio) of the service ID=2 is transmitted by the IP data flow using the MMT method, has the same service ID, and the data different from the part of the data is the advanced BS. An example is shown in which an IP data flow using a data transmission method (in this example, the file transmission protocol is different, but the same protocol may be used) may be used.

The transmitting device multiplexes the IP data so that the receiving device can guarantee that the data composed of the plurality of IP data flows are ready by the decoding time.

The receiving device can realize the guaranteed receiver operation by processing the data based on the decoding time using the data composed of the plurality of IP data flows.

[Supplement: Transmitting device and receiving device]
As described above, the transmission device that transmits data without operating the fragment counter can also be configured as shown in FIG. Also, a receiving device that receives data without operating the fragment counter can be configured as shown in FIG. FIG. 64 is a diagram illustrating an example of a specific configuration of the transmission device. FIG. 65 is a diagram illustrating an example of a specific configuration of the receiving device.

The transmission device 500 includes a division unit 501, a configuration unit 502, and a transmission unit 503. Each of the division unit 501, the configuration unit 502, and the transmission unit 503 is realized by, for example, a microcomputer, a processor, a dedicated circuit, or the like.

The reception device 600 includes a reception unit 601, a determination unit 602, and a configuration unit 603. Each of the reception unit 601, the determination unit 602, and the configuration unit 603 is realized by, for example, a microcomputer, a processor, a dedicated circuit, or the like.

Detailed description of each component of the transmission device 500 and the reception device 600 will be given in the description of the transmission method and the reception method, respectively.

First, the transmission method will be described with reference to FIG. 66. FIG. 66 is an operation flow (transmission method) performed by the transmission device.

First, the division unit 501 of the transmission device 500 divides the data into a plurality of pieces of divided data (S1701).

Next, the configuration unit 502 of the transmission device 500 configures a plurality of packets by adding header information to each of the plurality of divided data and packetizing them (S1702).

Then, the transmission unit 503 of the transmission device 500 transmits the plurality of configured packets (S1703). The transmission unit 503 transmits the divided data information and the invalidated fragment counter value. The divided data information is information for specifying the divided data number and the number of divided data. Further, the divided data number is a number indicating the number of divided data among the plurality of divided data. The number of pieces of divided data is the number of pieces of divided data.

As a result, the processing amount of the transmitting device 500 can be reduced.

Next, the reception method will be described with reference to FIG. FIG. 67 is an operation flow (reception method) by the receiving device.

First, the receiving unit 601 of the receiving device 600 receives a plurality of packets (S1801).

Next, the determination unit 602 of the reception device 600 determines whether the divided data information has been acquired from the plurality of received packets (S1802).

When the determination unit 602 determines that the divided data information has been acquired (Yes in S1802), the configuration unit 603 of the reception device 600 receives the fragment data without using the value of the fragment counter included in the header information. Data is constructed from the plurality of packets thus processed (S1803).

On the other hand, when the determination unit 602 determines that the divided data information has not been acquired (No in S1802), the configuration unit 603 uses the value of the fragment counter included in the header information to receive the plurality of received data. Data may be constructed from packets (S1804).

As a result, the processing amount of the receiving device 600 can be reduced.

(Embodiment 5)
[Overview]
In the fifth embodiment, a method of transmitting a transmission packet (TLV packet) when the NAL unit is stored in the multiplexing layer in the NAL size format will be described.

As described in the first embodiment, H.264. H.264 or H.264. There are two types of formats for storing 265 NAL units in the multiplexing layer. One is a format called a byte stream format in which a start code consisting of a specific bit string is added immediately before the NAL unit header. The other is a format called NAL size format in which a field indicating the size of the NAL unit is added. The byte stream format is used in the MPEG-2 system and RTP, and the NAL size format is used in MP4 or DASH and MMT using MP4.

In the byte stream format, the start code is composed of 3 bytes, and an arbitrary byte (byte whose value is 0) can be further added.

On the other hand, in the general MP4 NAL size format, the size information is indicated by any one of 1 byte, 2 bytes, and 4 bytes. This size information is indicated by the lengthSizeMinusOne field in the HEVC sample entry. When the value of the field is “0”, it indicates 1 byte, when it is “1”, it is 2 bytes, and when it is “3”, it is 4 bytes.

Here, in ARIB STD-B60 “Media transport method by MMT in digital broadcasting” standardized in July 2014, when the NAL unit is stored in the multiplexing layer, the output of the HEVC encoder is a byte stream. In this case, the byte start code is removed, and the size of the NAL unit in bytes shown by 32 bits (an unsigned integer) is added immediately before the NAL unit as length information. The MPU metadata including the HEVC sample entry is not transmitted, and the size information is fixed to 32 bits (4 bytes).

Further, in ARIB STD-B60 “Media transport method by MMT in digital broadcasting”, in the reception buffer model that the transmission device considers at the time of transmission in order to guarantee the buffer operation in the reception device, the buffer before decoding the video signal is used. Is defined as CPB.

However, there are the following issues. CPB in the MPEG-2 system and HRD in HEVC are specified on the assumption that the video signal is in the byte stream format. Therefore, for example, when the rate control of the transmission packet is performed on the assumption that the byte stream format has a start code of 3 bytes, the transmission packet of the NAL size format to which the size area of 4 bytes is added is received. The receiving device may not be able to satisfy the receiving buffer model in ARIB STD-B60. Further, since the receiving buffer model in ARIB STD-B60 does not show a specific buffer size and extraction rate, it is difficult to guarantee the buffer operation in the receiving device.

Therefore, in order to solve the above problem, a reception buffer model for guaranteeing the buffer operation in the receiver is defined as follows.

FIG. 68 shows a reception buffer model based on the reception buffer model defined in ARIB STD-B60, particularly when only a broadcast transmission path is used.

The reception buffer model includes a TLV packet buffer (first buffer), an IP packet buffer (second buffer), an MMTP buffer (third buffer), and a pre-decoding buffer (fourth buffer). It should be noted that the broadcast transmission path does not require a de-jitter buffer or a buffer for FEC, and is therefore omitted.

The TLV packet buffer receives a TLV packet (transmission packet) from a broadcast transmission path, and stores a variable-length packet header (IP packet header, full header when IP packet is compressed, or IP packet compression) stored in the received TLV packet. An IP packet obtained by converting an IP packet composed of a compressed header) and a variable-length payload into an IP packet (first packet) having a fixed-length IP packet header whose header has been expanded, and converting the IP packet. Is output at a constant bit rate.

The IP packet buffer converts the IP packet into an MMTP packet (second packet) having a packet header and a variable-length payload, and outputs the MMTP packet obtained by the conversion at a constant bit rate. The IP packet buffer may be merged with the MMTP buffer.

The MMTP buffer converts the output MMTP packet into a NAL unit, and outputs the NAL unit obtained by the conversion at a constant bit rate.

The pre-decoding buffer sequentially stores the output NAL units, generates an access unit from the stored plurality of NAL units, and outputs the generated access unit to the decoder at the timing of the decoding time corresponding to the access unit.

The receiving buffer model shown in FIG. 68 is characterized in that the MMTP buffer and the pre-decoding buffer, which are buffers other than the TLV packet buffer and the IP packet buffer in the preceding stage, follow the receiving buffer model in MPEG-2 TS.

For example, the MMTP buffer (MMTP B1) in video is composed of a transport buffer (TB) in MPEG-2 TS and a buffer corresponding to a multiplexing buffer (MB). Further, the MMTP buffer (MMTP Bn) for audio is composed of a buffer corresponding to the transport buffer (TB) in MPEG-2 TS.

The buffer size of the transport buffer is the same as that of MPEG-2 TS and is a fixed value. For example, the size is n times the MTU size (n may be a decimal or an integer, and is 1 or more).

Further, the MMTP packet size is specified so that the overhead rate of the MMTP packet header becomes smaller than the overhead rate of the PES packet header. As a result, the extraction rate from the transport buffer can be the same as the extraction rate RX1, RXn, RXs of the transport buffer in the MPEG-2 TS.

The size of the multiplexing buffer and the extraction rate are MPEG-2.
MB size in TS and RBX1.

In addition to the above reception buffer model, the following restrictions are set in order to solve the problems.

The HEVC HRD standard presupposes a byte stream format, and MMT is a NAL size format in which a size area of 4 bytes is added to the head of a NAL unit. Therefore, at the time of encoding, rate control is performed so as to satisfy HRD in the NAL size format.

That is, the transmission device controls the rate of the transmission packet based on the reception buffer model and the constraints described above.

The receiving device can perform a decoding operation that causes an underflow or overflow by performing a reception process using the above signal.

Even if the head size area of the NAL unit is not 4 bytes, the rate control is performed so as to satisfy HRD in consideration of the head size area of the NAL unit.

The extraction rate of the TLV packet buffer (the bit rate at which the TLV packet buffer outputs IP packets) is set in consideration of the transmission rate after decompressing the IP header.

That is, a TLV packet having a variable data size is input, and after the TLV header is removed and the IP header is expanded (restored), the transmission rate of the output IP packet is considered. In other words, the increase/decrease amount of the header is taken into consideration with respect to the input transmission rate.

Specifically, since the data size is variable, packets with IP header compression and packets without IP header compression are mixed, and the size of the IP header differs depending on the packet type such as IPv4 and IPv6. The transmission rate of the output IP packet is not unique. Therefore, the average packet length of variable length data size is determined, and the transmission rate of the IP packet output from the TLV packet is determined.

Here, in order to define the maximum transmission rate after decompressing the IP header, the transmission rate is determined assuming that the IP header is always compressed.

When the packet types of IPv4 and IPv6 coexist, or when the packet types are defined without distinction, the transmission rate is determined by assuming an IPv6 packet having a large header size and a large increase rate after decompressing the header. ..

For example, if the average packet length of the TLV packets input to the TLV packet buffer is S, and all the IP packets stored in the TLV packets are IPv6 packets and the header is compressed, the TLV header removal and The maximum output transmission rate after decompressing the IP header is
Input rate x {S/(S+IPv6 header compression amount)}
Becomes

More specifically, the average packet length S of TLV packets is
S=0.75×1500 (1500 assumes the maximum MTU size)
Set as the standard,
IPv6 header compression amount=TLV header length-IPv6 header length-UDP header length
=3-40-8
, The maximum output transmission rate after removal of the TLV header and decompression of the IP header is
Input rate x 1.0417 ≈ Input rate x 1.05
Becomes

FIG. 69 is a diagram showing an example in which a plurality of data units are aggregated and stored in one payload.

In the MMT method, when a data unit is aggregated, as shown in FIG. 69, the data unit length and the data unit header are added before the data unit.

However, for example, when storing a video signal in the NAL size format as one data unit, as shown in FIG. 70, there are two fields indicating the size for one data unit, and they are duplicated as information. .. FIG. 70 is an example of aggregating a plurality of data units and storing them in one payload, and is a diagram showing an example in which a video signal in the NAL size format is one data unit. Specifically, both the size area at the beginning of the NAL size format (hereinafter referred to as “size area”) and the data unit length field located before the data unit header in the MMTP payload header are both the size. Is a field indicating that the information is duplicated. For example, when the length of the NAL unit is L bytes, L bytes are shown in the size area, and L bytes+“length of size area” (byte) are shown in the data unit length field. Although the values indicated by the size area and the data unit length field do not completely match, it can be said that they overlap because one value can be easily calculated from the other.

In this way, when the data including the size information of the data is stored as a data unit, and when a plurality of the data units are aggregated and stored in one payload, since the size information is duplicated, the overhead is large, There is a problem that the transmission efficiency is poor.

Therefore, in the transmitting device, when data including size information of data inside is stored as a data unit and a plurality of the data units are aggregated and stored in one payload, as shown in FIG. 71 and FIG. 72. Can be stored.

As shown in FIG. 71, it is conceivable that a NAL unit including a size area is stored as a data unit and the data unit length conventionally included in the MMTP payload header is not shown. FIG. 71 is a diagram showing the structure of the payload of an MMTP packet in which the data unit length is not shown.

Further, as shown in FIG. 72, a flag indicating whether the data unit length is indicated or information indicating the length of the size area may be newly stored in the header. The location where the flag or the information indicating the length of the size area is stored may be indicated in a data unit unit such as a data unit header, or may be indicated in a unit (packet unit) in which a plurality of data units are aggregated. FIG. 72 is an example shown in the extend area added in units of packets. The storage location of the above-mentioned new information is not limited to this, and may be an MMTP payload header, an MMTP packet header, or control information.

On the receiving side, if the flag indicating whether or not the data unit length is compressed indicates that the data unit length is compressed, the length information of the size area inside the data unit is acquired and the size area is acquired. By acquiring the size area based on the length information of the data area, the data unit length can be calculated using the acquired length information of the size area and the acquired size area.

By the above method, the data amount can be reduced on the sending side and the transmission efficiency can be improved.

Note that the overhead may be reduced by reducing the size area instead of reducing the data unit length. When reducing the size area, information indicating whether the size area is reduced or information indicating the length of the data unit length field may be stored.

The MMTP payload header also includes length information.

When a NAL unit including a size area is stored as a data unit, the payload size area in the MMTP payload header may be reduced regardless of aggregation/non-aggregation.

Further, even when data that does not include a size area is stored as a data unit, if the data is aggregated and the data unit length is indicated, the payload size area in the MMTP payload header may be reduced.

In the case of reducing the payload size area, similarly to the above, a flag indicating whether or not it has been reduced, length information of the reduced size field, or length information of the non-reduced size field may be indicated.

FIG. 73 shows an operation flow of the receiving device.

In the sending device, as described above, the NAL unit including the size area is stored as a data unit, and the data unit length included in the MMTP payload header is not indicated in the MMTP packet.

In the following, an example will be described in which a flag indicating whether the data unit length is indicated or the length information of the size area is indicated in the MMTP packet.

The receiving device determines whether or not the data unit includes the size area and the data unit length is reduced based on the information transmitted from the transmission side (S1901).

When it is determined that the data unit length has been reduced (Yes in S1902), the length information of the size area inside the data unit is acquired, and then the size area inside the data unit is analyzed to calculate the data unit length. (S1903).

On the other hand, when it is determined that the data unit length is not reduced (No in S1902), the data unit length is calculated from either one of the data unit length and the size area inside the data unit as usual (S1904).

Note that the flag indicating whether the data unit length has been reduced or the length information of the size area need not be transmitted if the receiving device is known in advance. In this case, the receiving device performs the process shown in FIG. 73 based on the predetermined information.

[Supplement: Transmitting device and receiving device]
As described above, the transmission device that performs rate control so as to satisfy the specification of the reception buffer model at the time of encoding can also be configured as shown in FIG. Further, the receiving device that receives and decodes the transmission packet transmitted from the transmitting device can be configured as shown in FIG. FIG. 74 is a diagram illustrating an example of a specific configuration of the transmission device. FIG. 75: is a figure which shows the example of a concrete structure of a receiver.

The transmission device 700 includes a generation unit 701 and a transmission unit 702. Each of the generation unit 701 and the transmission unit 702 is realized by, for example, a microcomputer, a processor, a dedicated circuit, or the like.

The receiving device 800 includes a receiving unit 801, a first buffer 802, a second buffer 803, a third buffer 804, a fourth buffer 805, and a decoding unit 806. Each of the reception unit 801, the first buffer 802, the second buffer 803, the third buffer 804, the fourth buffer 805, and the decoding unit (decoder) 806 is, for example, a microcomputer, a processor, or a dedicated circuit. Is realized by

Detailed description of each component of the transmission device 700 and the reception device 800 will be given in the description of the transmission method and the reception method, respectively.

First, the transmission method will be described with reference to FIG. FIG. 76 is an operation flow (transmission method) performed by the transmission device.

First, the generation unit 701 of the transmission device 700 generates a coded stream by performing rate control so as to satisfy the regulation by a predetermined reception buffer model in order to guarantee the buffer operation of the reception device (S2001).

Next, the transmission unit 702 of the transmission device 700 packetizes the generated encoded stream, and transmits the transmission packet obtained by packetizing (S2002).

Note that the reception buffer model used in the transmission device 700 has a configuration including the first to fourth buffers 802 to 805 of the configuration of the reception device 800, and therefore description thereof will be omitted.

Accordingly, the transmitting device 700 can guarantee the buffer operation of the receiving device 800 when transmitting data using a method such as MMT.

Next, the receiving method will be described with reference to FIG. FIG. 77 is an operation flow (reception method) by the reception device.

First, the receiving unit 801 of the receiving device 800 receives a transmission packet composed of a fixed-length packet header and a variable-length payload (S2101).

Next, the first buffer 802 of the receiving device 800 converts the packet composed of the variable-length packet header and the variable-length payload stored in the received transmission packet into a fixed-length packet header decompressed. The packet is converted into the first packet that the user has, and the first packet obtained by the conversion is output at a constant bit rate (S2102).

Next, the second buffer 803 of the receiving device 800 converts the first packet obtained by the conversion into a second packet including a packet header and a variable-length payload, and obtains the second packet by converting the second packet. The generated second packet is output at a constant bit rate (S2103).

Next, the third buffer 804 of the receiving device 800 converts the output second packet into a NAL unit, and outputs the NAL unit obtained by the conversion at a constant bit rate (S2104).

Next, the fourth buffer 805 of the receiving device 800 sequentially accumulates the output NAL units, generates an access unit from the accumulated NAL units, and decodes the generated access unit at the decoding time corresponding to the access unit. It is output to the decoder at the timing of (S2105).

Then, the decoding unit 806 of the reception device 800 decodes the access unit output by the fourth buffer (S2106).

This allows the receiving device 800 to perform a decoding operation that causes underflow or overflow.

(Embodiment 6)
[Overview]
The sixth embodiment will describe a transmission method and a reception method in the case where leap second adjustment is performed on time information that serves as a reference of a reference clock in the MMT/TLV transmission method.

FIG. 78 is a diagram showing a protocol stack of the MMT/TLV method defined in ARIB STD-B60.

In the MMT method, data such as video and audio is contained in a plurality of MPUs (Media) in a packet.
It is stored for each first data unit such as a Presentation Unit) or an MFU (Media Fragment Unit) and an MMTP packet header is added to generate an MMTP packet as a predetermined packet (MMTP packetization). Also, an MMTP packet header is added to control information such as a control message in MMTP to generate an MMTP packet as a predetermined packet. The MMTP packet header is provided with a field for storing 32-bit short format NTP (Network Time Protocol: defined in IETF RFC 5905), and can be used for QoS control of a communication line.

In addition, a reference clock on the transmission side (transmission device) is synchronized with a 64-bit long format NTP defined in RFC 5905, and based on the synchronized reference clock, PTS (Presentation Time Stamp) or DTS (Decode Time). Time stamp such as Stamp) is added to the synchronized media. Further, the reference clock information of the transmitting side is transmitted to the receiving side, and the receiving device generates a system clock in the receiving device based on the reference clock information received from the transmitting side.

Specifically, the PTS and DTS are stored in the MPU time stamp descriptor or the MPU extended time stamp descriptor, which is the control information of MMTP, are stored in the MP table for each asset, and are MMTP packetized as a control message. And then transmitted.

The MMTP packetized data is encapsulated in an IP packet with a UDP header and an IP header added. At this time, a set of packets having the same source IP address, destination IP address, source port number, destination port number, and protocol type in the IP header or UDP header is defined as an IP data flow. Note that the headers of IP packets of the same IP data flow are redundant, and therefore some IP packets are header-compressed.

Also, as the reference clock information, a 64-bit NTP time stamp is stored in the NTP packet and then in the IP packet. At this time, in the IP packet storing the NTP packet, the source IP address, the destination IP address, the source port number, the destination port number, and the protocol type are fixed values, and the header of the IP packet is not compressed.

FIG. 79 is a diagram showing the structure of a TLV packet.

As shown in FIG. 79, the TLV packet can include, as data, transmission control information such as an IP packet, a compressed IP packet, AMT (Address Map Table), and NIT (Network Information Table). It is identified using an 8-bit data type. In the TLV packet, the data length (byte unit) is indicated using a 16-bit field, and the data value is stored after that. The TLV packet has 1-byte header information before the data type, and the header information is stored in a 4-byte header area in total. Further, the TLV packet is mapped to a transmission slot in the advanced BS transmission method, and mapping information is stored in TMCC (Transmission and Multiplexing Configuration Control) control information.

FIG. 80 is a diagram showing an example of a block diagram of a receiving device.

In the receiving device, first, the broadcast signal received by the tuner is subjected to channel coded data decoding, error correction, etc. in the demodulating means, and a TLV packet is extracted. Then, the TLV/IP DEMUX means performs TLV DEMUX processing and IP DEMUX processing. The TLV DEMUX process is performed according to the data type of the TLV packet. For example, when the TLV packet has a compressed IP packet, the compressed header of the compressed IP packet is restored. The IP DEMUX performs processing such as header analysis of IP packets and UDP packets to extract MMTP packets and NTP packets.

The NTP clock generation means reproduces the NTP clock from the extracted NTP packet. The MMTP DEMUX performs filtering processing of components such as video and audio and control information based on the packet ID stored in the extracted MMTP packet header. The time stamp descriptor stored in the MP table is acquired from the control information acquisition means, and the PTS/DTS for each access unit is calculated by the PTS/DTS calculation means. The time stamp descriptor includes both MPU time stamp descriptor and MPU extended time stamp descriptor.

The access unit reproducing means converts the video or audio filtered from the MMTP packet into data of a unit to be presented. Specifically, the data to be presented is a NAL unit or access unit of a video signal, an audio frame, a presentation unit of caption, or the like. The decoding and presenting means decodes and presents the access unit at the time when the PTS/DTS of the access unit match based on the reference time information of the NTP clock.

The configuration of the receiving device is not limited to this.

Next, the time stamp descriptor will be described.

FIG. 81 is a diagram for explaining the time stamp descriptor.

The PTS and DTS are stored in the MPU time stamp descriptor as the first control information and the MPU extended time stamp descriptor as the second control information, which are the control information of the MMT, are stored in the MP table for each asset, and are controlled. MMTP packetized as a message and then transmitted.

FIG. 81(a) is a diagram showing the structure of the MPU time stamp descriptor defined in ARIB STD-B60. In the MPU time stamp descriptor, for each of the plurality of MPUs, the first (first) AU in the presentation order of the plurality of access units (AUs) as the second data unit stored in the MPU (hereinafter , "Head AU"), the presentation time information (first time information) indicating the PTS (absolute value represented by NTP of 64 bits) is stored. That is, the presentation time information of the MPU given to the MPU is stored in the control information of the MMTP packet and transmitted.

FIG. 81(b) shows the structure of the MPU extension time stamp descriptor. The MPU extension time stamp descriptor stores information for calculating the PTS and DTS of the AU included in each MPU of the plurality of MPUs. The MPU extended time stamp descriptor includes relative information (second time information) from the PTS of the first AU of the MPU stored in the MPU time stamp descriptor, and the PTS of each of the AUs included in the MPU and The DTS can be calculated based on both the MPU time stamp descriptor and the MPU extended time stamp descriptor. That is, the PTS and DTS of AUs other than the head AU included in the MPU are based on the PTS of the head AU stored in the MPU time stamp descriptor and the relative information stored in the MPU extended time stamp descriptor. Can be calculated.

In other words, the second time information is relative time information for calculating the PTS or DTS of each of the plurality of AUs with the first time information. That is, the second time information is information indicating the PTS or DTS of each of the plurality of AUs together with the first time information.

NTP is reference time information based on Coordinated Universal Time (UTC). The UTC adjusts the leap second (hereinafter referred to as “leap second adjustment”) in order to adjust the difference from the astronomical time based on the rotation speed of the earth. The leap second adjustment is specifically an adjustment that is performed at 9 am Japan time and inserts or deletes for 1 second.

FIG. 82 is a diagram for explaining leap second adjustment.

FIG. 82A is a diagram showing an example of leap second insertion in Japan time. As shown in (a) of FIG. 82, in the leap second insertion, after 8:59:59 Japan time, it is 8:59:59 at the time when it should originally be 9:00: 8:59:59 units. Repeated twice.

FIG. 82(b) is a diagram showing an example of leap second deletion in Japan time. As shown in (b) of FIG. 82, in the leap second deletion, after 8:59:58 Japan time, it is 9:00:00 at the timing when originally it is 8:59:59, and 8:59:59 units are left. One second is deleted.

The NTP packet stores a 2-bit leap_indicator in addition to the 64-bit time stamp. The leap_indicator is a flag for notifying that leap second adjustment will be performed in advance, and indicates leap second insertion when leap_indicator=1 and deletion of leap second when leap_indicator=2. The advance notification includes a method of notifying from the beginning of the month for performing leap second adjustment, a method of notifying 24 hours before, and a method of starting notification at any time. Further, the leap_indicator becomes 0 at the time (9:00:00) when the leap second adjustment is completed. For example, if notification is given 24 hours before, the time immediately before the leap second adjustment is performed on the day when the leap second adjustment is performed from 9:00 the day before the leap second adjustment is performed in Japan time (that is, the leap second adjustment is performed). In the case of insertion, the time is 8:59:59 at the first time, and in the case of leap second deletion, the time is 8:59:58 until leap_indicator is "1" or "2". Shown.

Next, the problem when adjusting the leap second will be described.

FIG. 83 is a diagram showing the relationship between the NTP time, the MPU time stamp, and the MPU presentation timing. The NTP time is the time indicated by NTP. The MPU time stamp is a time stamp indicating the PTS of the first AU in the MPU. The MPU presentation timing is the timing at which the receiving device should present the MPU according to the MPU time stamp. Specifically, (a) to (c) of FIG. 83 respectively show the NTP time, the MPU time stamp, and the time when the leap second is inserted, when the leap second is inserted, and when the leap second is deleted. It is a figure which shows the relationship of MPU presentation time.

Here, the case where the NTP time (reference clock) on the sending side is synchronized with the NTP server and the NTP time (system clock) on the receiving side is synchronized with the NTP time on the sending side will be described as an example. To do. In this case, the receiving device reproduces based on the time stamp stored in the NTP packet transmitted from the transmitting side. Further, in this case, both the NTP time on the sending side and the NTP time on the receiving side are synchronized with the NTP server, and therefore the leap second is adjusted by ±1 second. Further, the NTP time in FIG. 83 is common to the NTP time on the sending side and the NTP time on the receiving side. The description will be made assuming that there is no transmission delay.

The MPU time stamp in FIG. 83 shows the time stamp of the head AU in the presentation order among the plurality of AUs included in each of the plurality of MPUs, and is generated (set) based on the NTP time indicated by the arrow. To be done. Specifically, the MPU is presented by adding a predetermined time (for example, 1.5 seconds in FIG. 83) to the NTP time as the reference time information at the timing of generating the presentation time information of the MPU. Generate time information. The generated MPU time stamp is stored in the MPU time stamp descriptor.

The receiving device presents the MPU at the MPU presentation time based on the time stamp stored in the MPU time stamp descriptor.

Note that, in FIG. 83, it is assumed that the reproduction time of one MPU is 1 second, but the reproduction time of one MPU may be another reproduction time, for example, 0.5 seconds. , 0.1 seconds may be sufficient.

In the example of (a) of FIG. 83, the receiving device can sequentially present the MPUs #1 to #5 based on the time stamps stored in the MPU time stamp descriptor.

However, in (b) of FIG. 83, the presentation times of MPU#2 and MPU#3 overlap due to the leap second insertion. Therefore, if the receiving device presents the MPU based on the time stamp stored in the MPU time stamp descriptor, there are two MPUs to present in the same time zone of 9:00:00. Therefore, it is impossible to determine which of the two MPUs should be presented. Further, since the receiving apparatus has the MPU presentation time (8:59:59) indicated by the time stamp of MPU#1 twice, which MPU presentation time should be presented among the two MPU presentation times? Can't judge.

Also, in (c) of FIG. 83, since the leap second is deleted, the receiving device does not have the MPU presentation time (8:59:59) indicated by the MPU time stamp of MPU#3 at the NTP time. 3 cannot be presented.

The receiving device can solve the above problems when performing the decoding process and the presentation process of the MPU not based on the time stamp. However, it is difficult for a receiving device that performs processing based on a time stamp to perform different processing (processing that is not based on a time stamp) only when leap seconds occur.

Next, a method for solving the problem at the time of adjusting the leap second by correcting the time stamp on the transmitting side will be described.

FIG. 84 is a diagram for explaining a correction method for correcting the time stamp on the transmission side. Specifically, (a) of FIG. 84 illustrates an example of leap second insertion, and (b) of FIG. 84 illustrates an example of leap second deletion.

First, the case of leap second insertion will be described.

As shown in (a) of FIG. 84, at the time of leap second insertion, the time immediately before the leap second insertion (that is, up to 8:59:59 at the first time at NTP time) is set as the A area, and after leap second insertion. (That is, after the second time 8:59:59 at the NTP time) is set as the B area. The A region and the B region are temporal regions and are time zones or periods. The MPU time stamp in (a) of FIG. 84 is the same as the time stamp described in FIG. 83, and is a time stamp generated (set) based on the NTP time at the timing of giving the MPU time stamp.

A method of correcting the time stamp on the transmitting side when leap seconds are inserted will be specifically described.

The transmitting side (transmitting device) performs the following processing.

1. The timing at which the MPU time stamp is added (the timing indicated by the origin of the arrow in (a) of FIG. 84) is included in the area A, and the MPU time stamp (the value of the MPU time stamp before correction) is 9:00:00. In the following cases, the MPU time stamp is decremented by 1 second and corrected by 1 second, and the corrected MPU time stamp is stored in the MPU time stamp descriptor. That is, when the MPU time stamp is generated based on the NTP time included in the area A and the MPU time stamp indicates after 9:00:00, the MPU time stamp is corrected by −1 second. Note that "9:00:00" here is the time corresponding to Japan time as the reference time for the leap second adjustment (that is, the time calculated by adding 9 hours to UTC time). Is. Also, correction information, which is information indicating that the correction has been performed separately, is transmitted to the receiving device.

2. When the timing to give the MPU time stamp (timing shown by the origin of the arrow in (a) of FIG. 84) is the B region, the MPU time stamp is not corrected. That is, when the MPU time stamp is generated based on the NTP time included in the B area, the MPU time stamp is not corrected.

The receiving device presents the MPU based on the MPU time stamp and the correction information indicating whether or not the MPU time stamp has been corrected (that is, whether or not the information indicating that the correction has been performed is included).

When the receiving device determines that the MPU time stamp is not corrected (that is, when the information indicating that the correction is performed is not included), the time stamp stored in the MPU time stamp descriptor is the receiving device. The MPU is presented at a time corresponding to the NTP time (including both before and after correction). That is, in the case of the MPU time stamp stored in the MPU time stamp descriptor of the MPU transmitted before the MPU whose MPU time stamp is corrected, the MPU time stamp is before the leap second insertion (that is, the first 8 : 59:59 or earlier), the MPU is presented at the timing matching the NTP time. When the MPU time stamp stored in the MPU time stamp descriptor of the received MPU is the time stamp transmitted after the corrected MPU time stamp, after the leap second insertion (that is, the second 8:59: The MPU is presented at a timing that coincides with the NTP time of 59 units or more).

In addition, when the MPU time stamp stored in the MPU time stamp descriptor of the received MPU is corrected, the receiving device detects that the time stamp stored in the MPU time stamp descriptor is ( That is, the MPU is presented based on the second NTP time of 8:59:59.

Information indicating that the MPU time stamp value has been corrected is stored in a control message, descriptor, table, MPU metadata, MF metadata, MMTP packet header, etc. and transmitted.

Next, the case of leap second deletion will be described.

As shown in (b) of FIG. 84, at the time of deleting a leap second, the time immediately before the deletion of the leap second (that is, immediately before 9:00:00 at the NTP time) is defined as the C area, and the time after the deletion of the leap second (that is, , 9:00 and later at NTP time) is the D area. The C region and the D region are temporal regions and are time zones or periods. The MPU time stamp in (b) of FIG. 84 is the same as the time stamp described in FIG. 83, and is a time stamp generated (set) based on the NTP time at the timing of giving the MPU time stamp.

A method of correcting the time stamp on the transmitting side when the leap second is deleted will be specifically described.

The transmitting side (transmitting device) performs the following processing.

1. The timing at which the MPU time stamp is added (timing indicated by the origin of the arrow in (b) of FIG. 84) is included in the area C, and the MPU time stamp (the value of the MPU time stamp before correction) is 8:59:59. In the following cases, the MPU time stamp is added for 1 second, the correction is performed for +1 second, and the corrected MPU time stamp is stored in the MPU time stamp descriptor. That is, when the MPU time stamp is generated based on the NTP time included in the C area and the MPU time stamp indicates 8:59:59 or later, the MPU time stamp is corrected by +1 second. Note that "8:59:59" here is the time corresponding to Japan time as the reference time for the leap second adjustment (that is, the time calculated by adding 9 hours to UTC time). Is the time obtained by subtracting -1 second from. Also, correction information, which is information indicating that the correction has been performed separately, is transmitted to the receiving device. In this case, the correction information does not necessarily have to be transmitted.

2. When the timing of giving the MPU time stamp (the timing indicated by the origin of the arrow in (b) of FIG. 84) is the D region, the MPU time stamp is not corrected. That is, when the MPU time stamp is generated based on the NTP time included in the D area, the MPU time stamp is not corrected.

The receiving device presents the MPU based on the MPU time stamp. If there is correction information indicating whether or not the MPU time stamp has been corrected, the MPU may be presented based on the MPU time stamp and the correction information.

By the above processing, even when the leap second adjustment is performed on the NTP time, the receiving apparatus can present a normal MPU using the MPU time stamp stored in the MPU time stamp descriptor. ..

The receiving side may be notified by signaling whether the MPU time stamp addition timing is in the A area, the B area, the C area, or the D area. That is, the MPU time stamp is generated based on the NTP time included in the area A, generated based on the NTP time included in the area B, or generated based on the NTP time included in the area C. It may be signaled whether it is generated based on the NTP time included in the D area and notify the receiving side. In other words, the identification information indicating whether or not the MPU time stamp (presentation time) is generated based on the reference time information (NTP time) before leap second adjustment may be transmitted. Since this identification information is added based on the leap_indicator included in the NTP packet, the leap second adjustment is performed from 9:00 the day before the leap second adjustment is performed in Japan time, and immediately before the leap second adjustment is performed on the day. It is set based on the time until the time (that is, the time of the first 8:59:59 in the case of leap second insertion and the time of 8:59:58 in the case of leap second deletion). This is information indicating whether or not it is an MPU time stamp. That is, is the identification information generated based on the NTP time from the time immediately before the time immediately before the leap second adjustment is performed by a predetermined period (for example, 24 hours) before the time immediately before the leap second adjustment is performed? This is information indicating whether or not.

Next, a method for solving the problem at the time of leap second adjustment by correcting the time stamp in the receiving device will be described.

FIG. 85 is a diagram for explaining a correction method for correcting the time stamp in the receiving device. Specifically, (a) of FIG. 85 shows an example of leap second insertion, and (b) of FIG. 85 shows an example of leap second deletion.

First, the case of leap second insertion will be described.

As shown in (a) of FIG. 85, when the leap second is inserted as in the case of (a) of FIG. 84, the time immediately before the leap second insertion (that is, up to the first 8:59:59 at NTP time). Is the area A, and the time after the leap second insertion (that is, after the second time 8:59:59 at the NTP time) is the area B. The A region and the B region are temporal regions and are time zones or periods. The MPU time stamp in (a) of FIG. 85 is the same as the time stamp described in FIG. 83, and is a time stamp generated (set) based on the NTP time at the timing of giving the MPU time stamp.

A method of correcting the time stamp at the receiving device in the case of leap second insertion will be specifically described.

The transmitting side (transmitting device) performs the following processing.

-The generated MPU time stamp is not corrected and the MPU time stamp is stored in the MPU time stamp descriptor and transmitted to the receiving device.

The information indicating whether the timing at which the time stamp of the MPU is added is the A area or the B area is transmitted to the receiving apparatus as identification information. That is, the identification information indicating whether the MPU time stamp is generated based on the NTP time included in the area A or the NTP time included in the area B is transmitted to the receiving device.

The receiving device performs the following processing.

The receiving apparatus corrects the MPU time stamp based on the MPU time stamp and the identification information indicating whether the timing to which the MPU time stamp is added is the A area or the B area.

Specifically, the following processing is performed.

1. The timing at which the MPU time stamp is added (the timing indicated by the origin of the arrow in (a) of FIG. 85) is included in the area A, and the MPU time stamp (the value of the MPU time stamp before correction) is 9:00:00. In the following cases, the MPU time stamp is reduced by 1 second and corrected by -1 second. That is, when the MPU time stamp is generated based on the NTP time included in the area A and the MPU time stamp indicates after 9:00:00, the MPU time stamp is corrected by −1 second. Note that "9:00:00" here is the time corresponding to Japan time as the reference time for the leap second adjustment (that is, the time calculated by adding 9 hours to UTC time). Is.

2. When the timing of giving the MPU time stamp (the timing indicated by the origin of the arrow in (a) of FIG. 85) is the B region, the MPU time stamp is not corrected. That is, when the MPU time stamp is generated based on the NTP time included in the B area, the MPU time stamp is not corrected.

When the MPU time stamp is not corrected, the MPU is presented at the time when the MPU time stamp stored in the MPU time stamp descriptor matches the NTP time (including before and after correction) of the receiving device.

That is, in the case of the MPU time stamp received before the MPU time stamp to be corrected, the MPU is presented at the time that matches the NTP time before the leap second insertion (that is, before the first 8:59:59). Further, in the case of the MPU time stamp received after the MPU time stamp to be corrected, the MPU is presented at the time corresponding to the NTP time after the leap second insertion (after the second 8:59:59).

When the MPU time stamp is corrected, the corrected MPU time stamp presents the MPU based on the NTP time after the leap second insertion in the receiving device (that is, after the second 8:59:59).

On the transmitting side, the identification information indicating whether the timing to which the MPU time stamp is added is in the A area or the B area is a control message, descriptor, table, MPU metadata, MF metadata, MMTP packet header. Store it in a file and transmit it.

Next, the case of leap second deletion will be described.

As shown in (b) of FIG. 85, at the time of deleting a leap second, as in (a) of FIG. 84, the time until immediately before the deletion of the leap second (that is, immediately before 9:00:00 in the NTP time) is C The area is defined as the area, and the time after the leap second deletion (that is, 9:00:0:00 or later at the NTP time) is defined as the area D. The C region and the D region are temporal regions and are time zones or periods. The MPU time stamp in (b) of FIG. 85 is the same as the time stamp described in FIG. 83, and is a time stamp generated (set) based on the NTP time at the timing of giving the MPU time stamp.

A method for correcting the time stamp in the receiving device in the case of the leap second deletion will be specifically described.

The transmitting side (transmitting device) performs the following processing.

-The generated MPU time stamp is not corrected and the MPU time stamp is stored in the MPU time stamp descriptor and transmitted to the receiving device.

The information indicating whether the timing at which the time stamp of the MPU is added is the C area or the D area is transmitted to the receiving apparatus as identification information. That is, the identification information indicating whether the MPU time stamp is generated based on the NTP time included in the C area or the NTP time included in the B area is transmitted to the receiving device.

The receiving device performs the following processing.

In the receiving device, the MPU time stamp is corrected based on the MPU time stamp and the identification information indicating whether the timing to which the MPU time stamp is added is the C area or the D area.

Specifically, the following processing is performed.

1. The timing (the timing indicated by the origin of the arrow in FIG. 85B) of adding the MPU time stamp is included in the C area, and the MPU time stamp (the value of the MPU time stamp before correction) is 8:59:59. In the following cases, the MPU time stamp value is added by 1 second and corrected by +1 second. That is, when the MPU time stamp is generated based on the NTP time included in the C area and the MPU time stamp indicates 8:59:59 or later, the MPU time stamp is corrected by +1 second. Note that "8:59:59" here is the time corresponding to Japan time as the reference time for the leap second adjustment (that is, the time calculated by adding 9 hours to UTC time). Is the time obtained by subtracting -1 second from.

2. When the timing of giving the MPU time stamp (timing shown by the origin of the arrow in (b) of FIG. 85) is the D region, the MPU time stamp is not corrected. That is, when the MPU time stamp is generated based on the NTP time included in the D area, the MPU time stamp is not corrected.

The receiving device presents the MPU based on the MPU time stamp and the corrected MPU time stamp.

By the above processing, even when the leap second adjustment is performed on the NTP time, the receiving apparatus can present a normal MPU using the MPU time stamp stored in the MPU time stamp descriptor. ..

Even in this case, as in the case of correcting the MPU time stamp on the transmission side described with reference to FIG. 84, the MPU time stamp application timing is in the A area, the B area, or the C area. It may be signaled whether it is present or in the D area and notified to the receiving side. Since the details of the notification are the same, the description is omitted.

It should be noted that the information indicating whether or not the MPU time stamp has been corrected, and the timing at which the MPU time stamp is added, described in FIGS. 84 and 85, is the A area, the B area, or the C area. , The information indicating whether it is the D area or the like (identification information) may be validated when the leap_indicator of the NTP packet indicates leap second deletion (leap_indicator=2) or insertion (leap_indicator=1). It may be valid from an arbitrary predetermined time (for example, 3 seconds before the leap second adjustment), or may be valid dynamically.

Also, the timing at which the valid period of the additional information ends and the timing at which the signaling of the additional information ends may be matched to the leap_indicator of the NTP packet, or at an arbitrary predetermined time (for example, 3 seconds before the leap second adjustment). May be enabled from the beginning, or may be enabled dynamically.

Since the 32-bit time stamp stored in the MMTP packet and the time stamp information stored in the TMCC are also generated and added based on the NTP time, the same problem occurs. Therefore, in the case of a 32-bit time stamp or time stamp information stored in TMCC, the time stamp is corrected using the same method as in FIGS. 84 and 85, and the processing based on the time stamp is performed in the receiving device. can do. For example, when the above-mentioned additional information for the 32-bit time stamp stored in the MMTP packet is stored, the extension area of the MMTP packet header may be used. In this case, the extended type of the multi-header type indicates additional information. In addition, the time when the leap_indicator is standing may indicate additional information by using some of the bits in the time stamp of 32 bits or 64 bits.

In the example of FIG. 84, the corrected MPU time stamp is stored in the MPU time stamp descriptor. However, the MPU time stamps before and after the correction (that is, the uncorrected MPU time stamp and the corrected MPU time stamp). ) May be transmitted to both receiving devices. For example, the MPU time stamp descriptor before correction and the MPU time stamp descriptor after correction may be stored in the same MPU time stamp descriptor, or may be stored in two MPU time stamp descriptors respectively. Good. In this case, the MPU time stamp before the correction or the MPU time stamp after the correction is identified by the arrangement order of the two MPU time stamp descriptors, the description order in the MPU time stamp descriptor, and the like. Good. In addition, the MPU time stamp before correction is always stored in the MPU time stamp descriptor, and the corrected time stamp may be stored in the MPU extended time stamp descriptor when correction is performed.

In the present embodiment, the time in Japan time (9:00 am) has been described as an example of the NTP time, but it is not limited to the time in Japan time. The leap second adjustment is based on UTC time and is corrected world-wide. Japan time is a time advanced by 9 hours with respect to UTC time, and is represented by a value of time (+9) with respect to UTC time. In this way, the times adjusted to different times according to the time difference depending on the place may be adopted.

As described above, by correcting the time stamp at the transmitting side or the receiving device based on the information indicating the timing to which the time stamp is added, the normal reception process using the time stamp becomes possible.

A receiving device that continues the decoding process well before the leap second adjustment time may be able to perform the decoding process and the presentation process without using the time stamp, but the receiving device selected immediately before the leap second adjustment time is , The time stamp cannot be determined, and the time stamp may not be presented until after the leap second adjustment is completed. Even in that case, by using the correction method according to the present embodiment, the reception process using the time stamp can be performed, and the channel can be selected immediately before the leap second adjustment time.

FIG. 86 shows an operation flow of the transmission side (transmission apparatus) and FIG. 87 shows an operation flow of the reception apparatus when the MPU time stamp is corrected on the transmission side (transmission apparatus) described in FIG.

First, the operation flow of the transmission side (transmission device) will be described using FIG. 86.

When the leap second is inserted or deleted, it is determined whether the MPU time stamp addition timing is in the A area, the B area, the C area, or the D area (S2201). The case where leap seconds are not inserted or deleted is not shown.

Here, the areas A to D are defined as follows.

Area A: Time until just before leap second insertion (up to 8:59:59 for the first time)
Area B: Time after leap second insertion (after the second 8:59:59)
Area C: Time until just before leap second deletion (until 9:00: 00)
Area D: Time after deletion of leap seconds (after 9: 00: 00)

If it is determined in step S2201 that the area is the A area and the MPU time stamp indicates 9:00 or later, the MPU time stamp is corrected by -1 second, and the corrected MPU time stamp is set to the MPU time. It is stored in the stamp descriptor (S2202).

Then, the correction information indicating that the MPU time stamp has been corrected is signaled and transmitted to the receiving device (S2203).

If it is determined in step S2201 that the area is the C area and the MPU time stamp indicates 8:59:59 or later, the MPU time stamp is corrected by +1 second and the corrected MPU time stamp is set to MPU. It is stored in the time stamp descriptor (S2205).

If it is determined in step S2201 that the area is the B area or the D area, the MPU time stamp is not corrected and stored in the MPU time stamp descriptor (S2204).

Next, the operation flow of the receiving device will be described with reference to FIG.

Based on the information signaled from the transmitting side, it is determined whether the MPU time stamp has been corrected (S2301).

When it is determined that the MPU time stamp has been corrected (Yes in S2301), the MPU is presented based on the MPU time stamp at the NTP time after the leap second adjustment is performed in the receiving device (S2302).

When it is determined that the MPU time stamp has not been corrected (No in S2301), the MPU is presented based on the MPU time stamp at the NTP time.

When correcting the MPU time stamp, the MPU corresponding to the MPU time stamp is presented in the section in which the leap second is inserted.

On the contrary, when the MPU time stamp is not corrected, the MPU corresponding to the MPU time stamp is not presented in the section in which the leap second is inserted, but is presented in the section in which the leap second is not inserted.

FIG. 88 shows an operation flow of the transmission side and FIG. 89 shows an operation flow of the reception device when the MPU time stamp is corrected in the reception device described in FIG.

First, the operation flow of the transmission side (transmission device) will be described using FIG. 88.

Based on the leap_indicator of the NTP packet, it is determined whether the leap second is adjusted (inserted or deleted) (S2401).

When it is determined that the leap second is adjusted (Yes in S2401), the timing of adding the MPU time stamp is determined, the identification information is signaled, and transmitted to the receiving device (S2402).

On the other hand, when it is determined that the leap second is not adjusted (No in S2041), the process ends without changing the normal operation.

Next, the operation flow of the receiving apparatus will be described using FIG. 89.

Based on the identification information signaled by the transmission side (transmission device), it is determined whether the timing of giving the MPU time stamp is the A region, the B region, the C region, or the D region. (S2501). Here, since the areas A to D are the same as those defined above, the description thereof will be omitted. Note that, similarly to FIG. 87, the case where the leap second is not inserted or deleted is not shown.

When it is determined in step S2501 that the area is the A area and the MPU time stamp indicates 9:00 or later, the MPU time stamp is corrected by -1 second (S2502).

When it is determined in step S2501 that the area is the C area and the MPU time stamp indicates 8:59:59 or later, the MPU time stamp is corrected by +1 second (S2504).

When it is determined in step S2501 that the region is the B region or the D region, the MPU time stamp is not corrected (S2503).

The receiver further presents the MPU based on the corrected MPU time stamp in a process (not shown).

When correcting the MPU time stamp, the MPU corresponding to the MPU time stamp is presented based on the MPU time stamp at the NTP time after the leap second adjustment is performed.

On the contrary, when the MPU time stamp is not corrected, the MPU corresponding to the MPU time stamp is not presented in the section in which the leap second is inserted, but is presented in the section in which the leap second is not inserted.

In short, the transmission side (transmission device) determines, for each of the plurality of MPUs, the timing of giving the MPU time stamp corresponding to the MPU. Then, as a result of the determination, when the timing is the time immediately before the insertion of the leap second and the MPU time stamp indicates 9:00:00 or later, the MPU time stamp is corrected by −1 second. Also, correction information indicating that the MPU time stamp has been corrected is signaled and transmitted to the receiving device. Further, as a result of the determination, when the timing is a time immediately before the leap second deletion and the MPU time stamp indicates 8:59:59 or later, the MPU time stamp is corrected by +1 second.

Further, if the receiving device indicates that the MPU time stamp is corrected based on whether or not the MPU time stamp indicated by the correction information signaled by the transmission side (transmission device) is corrected, , Presents the MPU based on the MPU time stamp at the NTP time after the leap second adjustment is performed. When the MPU time stamp is not corrected, the MPU is presented based on the MPU time stamp at the NTP time before leap second adjustment.

In addition, the transmission side (transmission device) determines the timing of giving the MPU time stamp corresponding to the MPU for each of the plurality of MPUs, and signals the determined result. The receiving device 1 performs the following processing based on the information signaled by the transmitting side and indicating the timing of adding the MPU time stamp. Specifically, when the information indicating the timing indicates the time immediately before the leap second insertion, and the MPU time stamp indicates 9:00:00 or later, the MPU time stamp is set to − Correct for 1 second. Further, when the information indicating the timing is the time immediately before the leap second deletion and the MPU time stamp is 8:59:59 or later, the MPU time stamp is corrected by +1 second.

When the MPU time stamp is corrected, the receiving device presents the MPU based on the MPU time stamp at the NTP time after the leap second adjustment is performed. When the MPU time stamp is not corrected, the MPU is presented based on the MPU time stamp at the NTP time before leap second adjustment.

As described above, by determining the timing of giving the MPU time stamp in the leap second adjustment, the MPU time stamp can be corrected, the receiving device can judge which MPU should be presented, and the MPU time stamp description. Appropriate reception processing using a child or MPU extended time stamp descriptor becomes possible. That is, even if the leap second adjustment is performed on the NTP time, the receiving apparatus can present a normal MPU using the MPU time stamp stored in the MPU time stamp descriptor.

[Supplement: Transmitting device and receiving device]
As described above, the transmission device that stores the data forming the encoded stream in the MPU and transmits the data can also be configured as shown in FIG. 90. Further, the receiving device that receives the MPU in which the data forming the encoded stream is stored can be configured as shown in FIG. 91. FIG. 90 is a diagram illustrating an example of a specific configuration of the transmission device. FIG. 91 is a diagram illustrating an example of a specific configuration of the receiving device.

The transmission device 900 includes a generation unit 901 and a transmission unit 902. Each of the generation unit 901 and the transmission unit 902 is realized by, for example, a microcomputer, a processor, a dedicated circuit, or the like.

The receiving device 1000 includes a receiving unit 1001 and a reproducing unit 1002. Each of the reception unit 1001 and the reproduction unit 1002 is realized by, for example, a microcomputer, a processor, a dedicated circuit, or the like.

Detailed description of each component of the transmitting device 900 and the receiving device 1000 will be given in the description of the transmitting method and the receiving method, respectively.

First, the transmission method will be described with reference to FIG. FIG. 92 is an operation flow (transmission method) performed by the transmission device.

First, the generation unit 901 of the transmission device 900 generates presentation time information (MPU time stamp) indicating the presentation time of the MPU as the first data unit based on the NTP time as the reference time information received from the outside. (S2601).

Next, whether the transmission unit 902 of the transmission device 900 is generated based on the MPU, the presentation time information generated by the generation unit 901, and the presentation time information (MPU time stamp) based on the NTP time before leap second adjustment. The identification information indicating whether or not it is transmitted (S2602).

As a result, the receiving device that receives the information transmitted from the transmitting device 900 can reproduce the MPU based on the identification information even when the leap second adjustment is performed, and thus reproduce the MPU at the intended time. it can.

Next, the receiving method will be described with reference to FIG. FIG. 93 is an operation flow (reception method) by the reception device.

First, the receiving unit 1001 of the receiving apparatus 1000 determines whether the MPU, the presentation time information indicating the presentation time of the MPU, and the presentation time information (MPU time stamp) are generated based on the NTP time before leap second adjustment. And the identification information indicating (S2701).

Next, the reproduction unit 1002 of the reception device 1000 reproduces the MPU received by the reception unit 1001 based on the presentation time information (MPU time stamp) and the identification information received by the reception unit 1001 (S2702).

Accordingly, the receiving apparatus 1000 can reproduce the MPU at the intended time even when the leap second adjustment is performed.

In the sixth embodiment, reproduction means a process including at least one of decoding and presentation.

(Modification of Embodiment 6)
Next, a specific example of a specific signaling method of additional information (identification information) described in FIGS. 84 and 85 will be described.

Here, a method of signaling using a combination of the MPU time stamp descriptor shown in (a) of FIG. 81 and the MPU extended time stamp descriptor shown in (b) of FIG. 81 will be described.

FIG. 94 is a diagram showing an extension example of the MPU extension time stamp descriptor. Fields underlined in FIG. 94 are fields newly added to the MPU extended time stamp descriptor shown in FIG. 81(b).

NTP_leap_indicator in FIG. 94 indicates a flag indicating whether or not the MPU extended time stamp descriptor as the second control information indicates additional information (identification information) related to leap second adjustment of NTP. When this flag is set, mpu_presentation_time_type in the loop for each of the plurality of MPUs becomes valid.

When mpu_presentation_time_type in FIG. 94 is corrected on the transmitting side (transmitting device) as described in FIG. 84, whether mpu_presentation_time of the same sequence number described in the MPU time stamp stamp descriptor is corrected. Show me how. In addition, in the case of mpu_presentation_time_type, when the receiving device corrects the MPU time stamp as described in FIG. 85, the timing at which mpu_presentation_time having the same sequence number described in the MPU time stamp stamp descriptor is given is the A area. It indicates whether it is the B area, the C area, or the D area.

That is, the MPU extended time stamp descriptor stores identification information indicating whether the MPU time stamp was generated based on the time information (NTP time) before leap second adjustment. The MPU extension time stamp descriptor stores identification information corresponding to the MPU in each loop of the plurality of MPUs.

In step S2602, the transmitter 902 of the transmitter 900 receives the MPU, the MPU time stamp descriptor as the first control information in which the presentation time information (MPU time stamp) generated by the generator 901 is stored, and The MPU extension time stamp descriptor as second control information, which stores identification information indicating whether the presentation time information is generated based on the time information (NTP time) before leap second adjustment, is transmitted.

In step S2701, the receiving unit 1001 of the receiving device 1000 receives the MPU, the MPU time stamp descriptor, and the MPU extended time stamp descriptor. Then, the reproduction unit 1002 of the reception device 1000 reproduces the MPU received by the reception unit 1001 based on the MPU time stamp descriptor and the MPU extended time stamp descriptor received by the reception unit 1001. In this way, the receiving apparatus 1000 can perform the decoding process using the MPU time stamp when adjusting the leap second by analyzing both the MPU time stamp descriptor and the MPU extended time stamp descriptor. Therefore, the receiving apparatus 1000 can reproduce the MPU at the intended time even when the leap second adjustment is performed. Further, since the identification information is stored in the MPU extended time stamp descriptor, the MPU time stamp descriptor has a configuration corresponding to the existing standard (MPEG), and the time information before the leap second adjustment (NTP time) of the MPU time stamp is kept. ), it is possible to extend the function of signaling whether or not it is generated based on. As described above, since the conventional configuration can be used, even if the function of signaling is expanded, the design change can be minimized and the manufacturing cost of the transmission side (transmission device) and the reception device can be reduced. ..

Note that the timing for setting NTP_leap_indicator may be the same as that for leap_indicator in the NTP packet, or may be any time while the leap_indicator is standing. That is, the NTP_leap_indicator is determined to be a value corresponding to the value of the leap_indicator in the NTP packet corresponding to the reference NTP time when the MPU time stamp was generated.

(Modification 2 of Embodiment 6)
The problem related to the leap second adjustment described in the present embodiment is not limited to the case of using the time based on the NTP time, but is the same problem even when the time based on the UTC time is used. is there.

When the time system is a time system based on UTC, and when the presentation time information (MPU time stamp) is generated and added based on the time system, use the same method as the method shown so far. Thus, even when the leap second is adjusted, it is possible to correct and process the presentation time information using the presentation time information and the signaling information (additional information, identification information) on the receiving device side.

For example, in ARIB STD-B60, an event message descriptor is transmitted when an event message is transmitted for the purpose of notifying a time (time designation information) for designating an operation from a broadcasting station to an application operating in a receiver. Is stored in the MMTP packet, and the time when the event message occurs is shown in the event message descriptor. In this case, the data forming the application is stored in the MPU as the first data unit and transmitted. Regarding the time at which the event message occurs, there are a plurality of methods for specifying the time, which is indicated by time_mode. In the time_mode, when the time at which the event message occurs is expressed in UTC, the UTC time is added based on the time information given based on the time information before the leap second adjustment or the time information after the leap second adjustment. Signaling identification information indicating whether it is time information. That is, like the identification information described in the present embodiment, the identification information is information indicating whether or not the presentation time information was generated based on the time information before the leap second adjustment (NTP time). Such identification information (signaling information) is preferably indicated in the event message descriptor. For example, signaling can be performed using the reserved_future_use field of the event message descriptor.

That is, in step S2601, the generation unit 901 of the transmission device 900 generates time designation information indicating the operation time of the application based on the NTP time as the reference time information received from the outside. Then, in step S2602, the transmission unit 902 of the transmission device 900 stores the MPU, the time designation information indicating the operation time of the application (the time when the event message occurs) generated by the generation unit 901, and the time information. And an event message descriptor as control information in which identification information indicating whether or not the time information before leap second adjustment is stored is transmitted.

It should be noted that the transmitting side (transmitting device) determines the time zone and programs that require leap second adjustment, prohibits the use of time_mode expressed in UTC in time zones and programs that require leap second adjustment, and It may be specified to use no time_mode.

Alternatively, the transmitter side (transmitting device) calculates the UTC time at which the event message occurs, determines whether the time at which the event message occurs is after leap second adjustment, and if the time is after leap second adjustment, then leap second adjustment is performed. The UTC time after second adjustment may be calculated and stored in the event message descriptor. Further, it may be separately signaled whether or not the presentation time information is adjusted for leap seconds.

Similarly, a UTC (Universal Time Coordinated)-NPT (Normal Play Time) reference descriptor indicates the relationship between UTC and NPT by using the UTC_Reference field and the NPT_Reference field. Whether the time information indicated in the UTC_Reference field or the NPT_Reference field is the time information given based on the reference time information before the leap second adjustment (NTP time) or the time information given based on the reference time information after the leap second adjustment. Signal. That is, the UTC-NPT reference descriptor includes presentation time information generated by the generation unit 901 of the transmission device 900 and identification information indicating whether the presentation time information is time information before leap second adjustment (NTP time). Is the control information that stores The identification information can be signaled by using a reserved field or the like in the UTC-NPT reference descriptor.

Further, the method of designating the presentation time of the subtitle or the superimposition of characters described in ARIB STD-B60 or ARIB STD-B62 is indicated by TMD (time control mode) in the additional identification information (Additional_Arib_Subtitle_Info). The additional identification information is stored in the asset loop of MPT.

TMD (time control mode) uses MPU time stamp and NPT (Normal Play).
In the case of the mode in which the time is from (Time) as the starting point, the presentation time of the subtitle or the superimposition of the character is indicated in the MPU extended time stamp descriptor or the UTC-NPT reference descriptor. With the above signaling information, it is possible to correct and process the time information using the time information and the signaling information on the receiving device side even when adjusting the leap second.

When the time control mode='0000' (UTC_TTML description), the time code in the ARIB-TTML subtitle indicates the presentation time corresponding to UTC. In this case, it signals whether the UTC time indicated by the time code is the time information given based on the time information before the leap second adjustment or the time information given based on the time information after the leap second adjustment. .. It is desirable that the signaling information transmitted at this time be shown in ARIB-TTML subtitles.

When the time control mode='0010' (UTC_TTML description), it is the time starting from the UTC time indicated in the reference_start_time field in the additional identification information. In this case, in the additional identification information, signaling is performed as in the case of time control mode='0000' (UTC_TTML description). In this case, on the transmission side (transmission device), the UTC time indicated in the reference_start_time field is a predetermined period (eg, a time until 9:00:00) immediately before the leap second adjustment is performed. Information indicating whether or not it is the time from the time 24 hours before (that is, 9:00:00 the day before the leap second adjustment is performed) to the time immediately before the time is transmitted as the identification information.

It is specified that the transmitting side (transmitting device) determines the time zone or program that requires leap second adjustment, prohibits the use of the time control mode expressed in UTC, and uses the time control mode not related to UTC. Good.

The same applies to other MH-Expire descriptors.

(Embodiment 7)
A general broadcasting station facility has a plurality of transmission systems, and one system is used as an actual facility for an actual broadcasting service and the other system is operated as a redundant system for backup. As a result, when an error occurs in the service provided to the viewer due to the transmission facility or the like, it is possible to switch to the redundant system facility. Here, in the MMT/TLV system, the following phenomenon occurs at the time of switching the redundant system equipment.

1. Adjustment of MPU sequence number The MPU sequence number is adjusted as a pre-process for switching the redundant system equipment. As shown in FIG. 95, when the MPU sequence number is adjusted, discontinuity may occur in the MPU sequence number before and after the adjustment. FIG. 95 shows an example in which the MPU sequence numbers overlap.

The access unit and the MPU time stamp for the access unit are the same before and after the adjustment of the MPU sequence number, but the same MPU sequence number is given to different MPUs, so that the two MPUs are given. The same MPU sequence number is given to all over.

As a result, the MPU sequence number stored in the MMTP packet header and the MPU sequence number stored in the MPU time stamp descriptor and the MPU extended time stamp descriptor overlap in different MPUs. Similarly, the MPU sequence number may be skipped or returned.

2. Discontinuity of packet sequence numbers Since packet sequence numbers and packet counters are different between normal equipment and redundant equipment, when switching from normal equipment to redundant equipment, packet sequence numbers and packet counters become discontinuous. May be. As shown in FIG. 96, the packet sequence numbers are continuous in normal operation, but may be skipped, duplicated, or returned. 96 is a diagram for explaining a case where the packet sequence numbers are discontinuous at the timing of switching from the normal equipment to the redundant equipment.

When the MPU sequence number or the packet sequence number becomes discontinuous like 1 or 2, there is a problem that a receiving device that operates by believing in the continuity of these sequence numbers may malfunction.

It should be noted that the access unit and the MPU time stamp for the access unit maintain continuity even when the MPU sequence number or the packet sequence number is discontinuous, and continuous data is sent to the receiving device.

A reception process when the MPU sequence number or the packet sequence number discontinuity as described with reference to FIGS. 95 and 96 is described.

When the MPU sequence number or the packet sequence number is discontinuous, the receiving device that performs the process using the MPU sequence number or the packet sequence number as described below may malfunction.

An example of the process using the MPU sequence number is a process of detecting the MPU boundary by switching the MPU sequence number in the MMTP packet header.

Further, as the processing using the MPU sequence number, for example, the MPU time stamp descriptor or the MPU extended time stamp descriptor describes the time stamp for the MPU sequence number, and decoding and/or decoding is performed using the time stamp. Present.

As the processing using the packet sequence number, for example, the packet sequence number is used for detecting packet loss, if the packet sequence number is skipped, it is determined as packet loss, and if the packet sequence number is returned, it is determined as packet replacement. There is processing.

In order to prevent malfunctions caused by performing processing using these MPU sequence numbers and packet sequence numbers, the receiving device determines the discontinuity of the MPU sequence numbers and packet sequence numbers. Perform processing that does not use numbers or packet sequence numbers.

Here, for example, the following method can be used as a method of determining the discontinuity of the MPU sequence number and the packet sequence number.

First, one of the above determination methods may be to signal additional information indicating switching to redundant equipment. Thereby, the receiving device can perform the receiving process based on the additional information without using the MPU sequence number or the packet sequence number.

As a method of signaling the additional information, there is a method of storing the additional information in a control message, a descriptor, a table, MPU metadata, MF metadata, an MMTP packet header, an extension header of an MMTP packet, etc. and transmitting the additional information.

Next, a method of detecting discontinuity by using other information in the receiving device will be described.

For example, as the information indicating the MPU boundary, there are a method of detecting switching of the MPU sequence number in the MMTP payload header and a method of using RAP_flag which is a flag indicating whether or not the MMTP packet header is a random access point. Besides, it is also possible to detect the MPU boundary by switching packets whose Sample_number is 0, switching fragment types, and the like. It is also possible to detect the MPU boundary by counting the number of access units.

Here, when there is a possibility that the switching of the redundant system equipment is performed, the MPU boundary using the switching of the MPU sequence number is not determined, and the MPU boundary is determined using another method. For example, the packet whose RAP_flag is 1 is determined to be the first packet in the MPU. Here, when the MPU sequence number of the packet after the MPU boundary does not change from the MPU sequence number of the previous packet, it is determined that the MPU sequence number duplication occurs in the MMTP packet.

Alternatively, the packet whose RAP_flag is 1 is determined to be the first packet in the MPU. Here, when the MPU sequence number of the packet determined to be the head packet is skipped or returned from the MPU sequence number of the preceding packet, it is determined that the MPU sequence number discontinuity occurs in the MMTP packet.

If the discontinuity of the MPU sequence number in the MMTP payload header always precedes the discontinuity of the MPU sequence number or the packet sequence number in the MPU time stamp descriptor or the MPU extended time stamp descriptor, based on the judgment result. As a result, a reception process that does not use a time stamp or a process that does not use a packet sequence number may be performed.

Further, the discontinuity of the MPU sequence number in the MMTP payload header of the video packet is the discontinuity of the MPU sequence number in the MMTP payload header of the audio packet, the MPU sequence number in the audio MPU time stamp descriptor or the MPU extended time stamp descriptor. And the packet sequence number discontinuity always occurs based on the detection result of the MPU sequence number discontinuity of the video packet, the reception process that does not use the audio time stamp and the process that does not use the packet sequence number. May be carried out. The determination result may be determined as the timing at which the switching to the redundant equipment is started.

Note that the receiving device does not use the MPU sequence number or the packet sequence number only when the additional information indicating the switching to the redundant equipment is indicated by using the method of signaling the additional information indicating the switching to the redundant equipment. You may process.

Note that the receiving device detects the discontinuity of RAP_flag and the like and the MPU sequence number only when the additional information indicating the switching to the redundant equipment is indicated by using the method of signaling the additional information indicating the switching to the redundant equipment. Processing may be carried out.

Here, the determination using the time stamp descriptor in the receiving device will be described. Normally, the time stamp for the MPU sequence number is described and often transmitted multiple times. However, when the time stamp value is updated despite the fact that the MPU sequence number is the same, the receiving device notifies the MPU sequence number. Are determined to be duplicates.

If the time stamp value is updated by the presentation time of one MPU, but the MPU sequence number is updated by 2 or more, and the relationship between the time stamp value and the MPU sequence number does not correspond, the reception is performed. The apparatus determines that the MPU sequence number discontinuity has occurred.

When updating the MPU time stamp or the MPU extension time stamp, the version in the MP table may not be updated. Alternatively, version information may be separately defined, and the version of the time stamp information and the version of information other than the time stamp information may be transmitted separately. As a result, the version frequency of information other than the time stamp information can be reduced, and reception processing can be reduced.

If the MPU sequence numbers are discontinuous, the relationship between the MPU time stamp and the time stamp information cannot be trusted. Therefore, the descriptor may signal that the time stamp information is unreliable information. For example, a field similar to NTP_leap_indicator or mpu_presentation_time_type described in FIG. 94 may be used to indicate that the timing at which the MPU time stamp is added is the timing before and after the redundant system switching and the time stamp cannot be trusted. Further, it may be indicated in the MPU extended time stamp descriptor on the assumption that it is used together with the MPU time stamp descriptor.

By using the above method, by determining that the MPU sequence number or the packet sequence number is discontinuous in the receiving device, it is possible to perform processing without using the MPU sequence number or the packet sequence number, and to prevent malfunction in synchronous reproduction. It becomes possible to prevent it. An operation flow of the receiving device when the discontinuity of the MPU sequence number or the packet sequence number as described above occurs will be described. FIG. 97 is an operation flow by the receiving device when discontinuity of MPU sequence number or packet sequence number occurs.

The receiving device determines the MPU boundary from the packet header or payload header of the MMTP packet. Specifically, the MPU boundary is determined by determining that the packet having RAP_flag of 1 is the MPU head packet. Further, the MPU sequence number is determined and it is determined whether the MPU sequence number of the previous packet is incremented by 1 (S2801).

When it is determined that RAP_flag is 1 and the MPU sequence number has increased by 1 (Yes in S2801), it is assumed that the MPU sequence number discontinuity has not occurred, and the reception process using the MPU sequence number is performed. (S2803).

On the other hand, if it is determined that RAP_flag is 0 or RAP_flag is 1 and the MPU sequence number of the previous packet is not increased by 1 (No in S2801), discontinuity of MPU sequence numbers occurs. If the MPU sequence number is not used, the reception process is performed (S2802).

(Embodiment 8)
Although the sixth embodiment has mainly described the correction of the MPU time stamp, a method of correcting the PTS or DTS for each AU included in the MPU will be described. That is, a correction method for correcting the time stamps indicating the PTS or DTS of each of a plurality of AUs other than the head AU among the plurality of AUs included in the MPU will be described.

FIG. 98 is a diagram for explaining a correction method for correcting the time stamp in the receiving device when the leap second is inserted. FIG. 98 shows the same case as FIG. 85(a).

Here, the NTP time on the transmitting side is synchronized with the NTP server, and the NTP time on the receiving side is reproduced based on the time stamp stored in the NTP packet transmitted from the transmitting side. A case where the NTP time is synchronized with that of the sender will be described as an example. In this case, both the NTP time on the transmitting side and the NTP time on the receiving side are adjusted by +1 second when the leap second is inserted.

It is assumed that the NTP time in FIG. 98 is common to the NTP time on the transmitting side and the NTP time on the receiving side. In the following description, it is assumed that there is no transmission delay. Further, only the method for correcting the PTS of the AU in the MPU will be described below, but the DTS of the AU in the MPU can also be corrected by using the same method. In addition, in FIG. 98, it is described that the number of AUs included in one MPU is 5, but the number of AUs included in the MPU is not limited to this.

Similar to the transmission method described in (a) of FIG. 85, the NTP time immediately before the leap second insertion (up to 8:59:59 for the first time) is the area A, and the NTP time after the leap second insertion (the second time). 8:59:59 and later) is set as the B area.

The transmitting side (transmitting device) performs the following processing.

As described with reference to FIG. 83 in the sixth embodiment, the MPU time stamp which is the presentation time information (first time information) of the MPU is generated.

-The generated MPU time stamp is not corrected and the MPU time stamp is stored in the MPU time stamp descriptor and transmitted to the receiving device. That is, the first time information is time information that is not corrected on the transmission side of the MPU during leap second adjustment. Also, the identification information indicating whether the timing at which the MPU time stamp is added is the A area or the B area is transmitted to the receiving device. Specifically, the MMTP packet in which a plurality of MPUs are stored is transmitted. This MMTP packet includes, as control information, an MPU time stamp descriptor including an MPU time stamp as first time information, an MPU extended time stamp descriptor including relative information as second time information, and identification information. ..

In FIG. 98, the timing at which the time stamp of the MPU is added is indicated by an arrow.

The receiving device performs the following processing.

As described in the sixth embodiment, in the MPU extended time stamp descriptor, the second time information for calculating the PTS or DTS for each AU is stored in the MPU time stamp descriptor of the head AU of the MPU. It is stored as relative information from the PTS. At this time, the relative information stored in the MPU extended time stamp descriptor is not a value considering the insertion of leap seconds.

The receiving device first receives the MMTP packet transmitted by the transmitting side (transmitting device) and including the MPU, the first time information, the second time information, and the identification information. As shown in PTS, all AUs included in the MPU based on both the first time information stored in the MPU time stamp descriptor and the second time information stored in the MPU extended time stamp descriptor. The PTS (DTS) is calculated for each.

Next, the receiving device corrects the PTS for each AU. In the AU to be corrected, the timing of giving the MPU time stamp of the MPU to which the AU belongs (the timing indicated by the arrow in FIG. 98) is the A region, and the PTS of each AU before correction is 9:00: It is AU which shows 00 and after. The AUs to be corrected are the AUs underlined in FIG. 98, which are AU#5 in MPU#1 and AU#1-AU#5 in MPU#2. When the receiving device determines that it corresponds to the PTS correction target AU, it corrects the PTS of the AU by −1 second.

That is, in the correction, for each of the plurality of AUs, the MPU timestamp of the MPU that stores the AU is generated based on the reference time information before leap second adjustment, and the calculated PTS of the AU or It is determined whether the DTS satisfies a correction condition after a predetermined time (9:00:00 in the case of leap second insertion). Then, in the correction, the PTS or DTS of the AU that has been determined to satisfy the above correction conditions in the determination is corrected.

The MPU to which the AU belongs has a timing of giving the MPU time stamp in the A region and the PTS is not corrected (AUs up to AU#4 in MPU#1 in FIG. 98) before leap second insertion (1 The AU is presented at a time matching the NTP time of 8:59:59: before the second time. Further, AUs with corrected PTS or AUs thereafter (AUs after AU#5 in MPU#1 in FIG. 98) are inserted at the NTP time after the leap second insertion (after the second 8:59:59 units). Present the AU at the matching time.

The identification information indicating the timing of adding the MPU time stamp needs to be presented to at least the MPU including the AU to be corrected. That is, it is not always necessary to present the identification information to the MPU that does not include the AU to be corrected.

FIG. 99 is a diagram for explaining a correction method for correcting the time stamp in the receiving device when the leap second is deleted. FIG. 99 shows the same case as FIG. 85(b).

The NTP time of the sending side is synchronized with the NTP server, and the NTP time of the receiving side is reproduced based on the time stamp stored in the NTP packet transmitted from the sending side. A case where the time is synchronized with the NTP time will be described as an example. In this case, both the NTP time on the transmitting side and the NTP time on the receiving side are adjusted by -1 second when the leap second is deleted.

The NTP time in FIG. 99 is common to the NTP time on the transmitting side and the NTP time on the receiving side. In the following description, it is assumed that there is no transmission delay. Also, only the method of correcting the PTS of the AU in the MPU will be described below, but the DTS of all AUs in the MPU can be corrected by using the same method. It should be noted that although FIG. 99 is described assuming that the number of AUs included in one MPU is 5, the number of AUs included in the MPU is not limited to this.

Similar to the transmission method described with reference to FIG. 85, the NTP time immediately before the leap second deletion (up to 8:59:58) is in the C area, and the NTP time after the leap second deletion (after 9:00:00) is D. The area.

The transmitting side (transmitting device) performs the following processing.

As described with reference to FIG. 83 in the sixth embodiment, the MPU time stamp which is the presentation time information (first time information) of the MPU is generated.

-The generated MPU time stamp is not corrected and the MPU time stamp is stored in the MPU time stamp descriptor and transmitted to the receiving device. That is, the first time information is time information that is not corrected on the transmission side of the MPU during leap second adjustment. Also, the identification information indicating whether the timing at which the MPU time stamp is added is the C area or the D area is transmitted to the receiving device. Specifically, the MMTP packet in which a plurality of MPUs are stored is transmitted. The MMTP packet has the same configuration as the MMTP packet described with reference to FIG.

In FIG. 99, the timing at which the time stamp of the MPU is added is indicated by the arrow.

The receiving device performs the following processing.

As described in the sixth embodiment, in the MPU extended time stamp descriptor, the second time information for calculating the PTS or DTS for each AU is stored in the MPU time stamp descriptor of the head AU of the MPU. It is stored as relative information from the PTS. At this time, the relative information stored in the MPU extended time stamp descriptor is not a value considering the insertion of leap seconds.

The receiving device first receives the MMTP packet transmitted by the transmitting side (transmitting device) and including the MPU, the first time information, the second time information, and the identification information, and the “for each AU” in FIG. 99 is received. As shown in PTS, all AUs included in the MPU based on both the first time information stored in the MPU time stamp descriptor and the second time information stored in the MPU extended time stamp descriptor. The PTS (DTS) is calculated for each.

Next, the receiving device corrects the PTS for each AU. In the AU to be corrected, the timing of giving the MPU time stamp of the MPU to which the AU belongs (the timing shown by the arrow in FIG. 99) is the C region, and the PTS for each AU before correction is 8:59: It is AU which shows 59 or later. The AUs to be corrected are the AUs underlined in FIG. 99, which are AU#5 in MPU#2 and AU#1-AU#5 in MPU#3. When the receiving device determines that it corresponds to the PTS correction target AU, it corrects the PTS of the AU by +1 second.

That is, in the correction, for each of the plurality of AUs, the MPU timestamp of the MPU that stores the AU is generated based on the reference time information before leap second adjustment, and the calculated PTS of the AU or It is determined whether the DTS satisfies a correction condition after a predetermined time (8:59:59 in the case of leap second deletion). Then, in the correction, the PTS or DTS of the AU that has been determined to satisfy the above correction conditions in the determination is corrected.

The receiving apparatus can perform synchronous reproduction of all AUs based on the NTP time at which the leap second deletion was performed and the corrected PTS.

The identification information indicating the timing of adding the MPU time stamp needs to be presented to at least the MPU including the AU to be corrected. That is, it is not always necessary to present the identification information to the MPU that does not include the AU to be corrected.

FIG. 100 is a diagram showing an operation flow of the receiving device described in FIGS. 98 and 99.

Note that the operation flow on the transmitting side is the same as the operation flow shown in FIG. 88, so description will be omitted.

The receiving device first calculates the time stamps (PTS and DTS) for all AUs using the MPU time stamp descriptor and the MPU extended time stamp descriptor (S2901).

Next, based on the identification information signaled by the transmitting side, whether the timing of giving the MPU time stamp of the MPU to which the AU belongs is in the A area, the B area, or the C area for each AU. , D area is determined (S2902). Here, since the areas A to D are the same as those defined above, the description thereof will be omitted. Note that, similarly to FIG. 89, the case where the leap second is not inserted or deleted is not shown.

When it is determined in step S2902 that the MPU time stamp of the MPU to which the AU belongs is added based on the NTP time of the area A, the time stamp of the AU calculated in step S2901 is 9:00:0: or later. It is determined whether or not (S2903).

If it is determined that the time stamp of the AU is after 9:00:00 (Yes in step S2903), the time stamp value of the AU is corrected by -1 second (S2904).

When it is determined in step S2902 that the MPU time stamp of the MPU to which the AU belongs is added based on the NTP time of the C area, whether the time stamp of the AU calculated in step S2901 is 8:59:59 or later. Is determined (S2905).

When it is determined that the time stamp of the AU is 8:59:59 or later (Yes in step S2905), the time stamp value of the AU is corrected by +1 second (S2906).

On the other hand, if it is determined in step S2902 that the MPU time stamp of the MPU to which the AU belongs is added based on the NTP time of the B area or the D area, or if the time stamp value of the AU is not 90000 or later in step S2903. If it is determined (No in step S2903), or if it is determined in step S2905 that the time stamp value of AU is not 8:59:59 or later (No in step S2905), the time stamp of AU is not corrected ( In step S2907), the operation of the reception device ends.

As described above, as represented by the MPU time stamp descriptor and the MPU extended time stamp descriptor, the first time information indicating the absolute value of the MPU time stamp and the second time indicating the relative value to the absolute value. When the information and the information are respectively transmitted, the transmission side transmits the absolute value of the time stamp and the identification information indicating the NTP time region on which the absolute value of the time stamp is given.

In the receiving device, after calculating the time stamps (PTS and DTS) of all AUs based on the first time information indicating the absolute value of the MPU time stamp and the second time information indicating the relative value, the respective time stamps, and The time stamp is corrected by determining whether to correct the time stamp based on the identification information indicating the area of the NTP time that is the base of the MPU time stamp that is the base of the time stamp. ..

Information indicating whether or not the time stamp has been corrected for each AU in the receiving device, and whether the timing at which the MPU time stamp of the MPU to which each AU belongs is assigned to the A region, the B region, or the C region , The additional information such as the information indicating whether it is the D area or the like does not have to be indicated by all the MPUs, and may be indicated in the section including the MPU including at least the AU to be corrected.

For example, when the time difference (PTS or DTS) in an arbitrary AU and the time difference between the MPU to which the AU belongs and the NTP time that is the basis for giving the MPU time stamp are N seconds, when a leap second is inserted, or when a leap second is inserted, When the seconds are deleted, the period during which the AU to be corrected exists is at least N seconds before the leap second adjustment. Therefore, the area A at the time of inserting the leap second or the area C at the time of deleting the leap second needs to be at least an area indicating a period of N seconds or more before the leap second adjustment.

On the contrary, the time indicating the area A when the leap second is inserted and the area C when the leap second is deleted may be defined as M seconds, and a constraint may be given so that a time stamp is added so that N<M.

In addition, the B area at the time of inserting a leap second or the D area at the time of deleting a leap second does not necessarily need to be signaled, and information indicating a transition from the area A to the area B is inserted at the time of inserting a leap second. At the time of deletion, it is sufficient to show at least information indicating that the C area has been transferred to the D area.

In short, the receiving device first calculates the time stamps of all AUs based on the first time information (absolute time information) and the second time information (relative time information), and calculates the time stamps of all AUs. With respect to, the time stamp of the AU is corrected based on the identification information indicating whether or not the NTP time that is the basis of the calculation of the MPU time stamp of the MPU to which the AU belongs is before the leap second adjustment. Specifically, when the identification information indicates the time immediately before the leap second insertion, and when the time stamp of the AU indicates 9:00:00 or later, the time stamp is corrected by −1 second. If the identification information indicates the time immediately before the leap second deletion and the time stamp of the AU indicates 8:59:59 or later, the time stamp is corrected by +1 second. Further, in the case of an AU whose time stamp is corrected, the receiving device presents the AU based on the reference time information after the leap second adjustment is performed.

In this way, at the time of adjusting the leap second, it is possible to correct the time stamps of all the AUs by determining the timing of giving the MPU time stamp of the MPU to which the AU belongs, and the MPU time stamp descriptor and MPU extended time stamp description can be described. Appropriate reception processing using the child becomes possible. That is, even when the leap second adjustment is performed on the reference time information serving as the reference of the reference clocks of the transmitting side and the receiving device, the plurality of AUs stored in the MPU can be reproduced at the intended time.

[Supplement: Transmission/reception system and reception device]
As described above, the transmission/reception system including the transmission device that stores the data forming the encoded stream in the MPU and transmits the data and the reception device that receives the transmitted MPU can also be configured as shown in FIG. 101. Is. In addition, the receiving device can also be configured as shown in FIG. FIG. 101 is a diagram showing an example of a specific configuration of the transmission/reception system. FIG. 102 is a diagram illustrating an example of a specific configuration of the receiving device.

The transmission/reception system 1100 includes a transmission device 1200 and a reception device 1300.

The transmission device 1200 includes a generation unit 1201 and a transmission unit 1202. Each of the generation unit 1201 and the transmission unit 1202 is realized by, for example, a microcomputer, a processor, a dedicated circuit, or the like.

The generation unit 1201 generates first time information (MPU time stamp) indicating the presentation time of the first data unit (MPU) based on the reference time information (NTP time) received from the outside.

The transmission unit 1202 includes a first data unit (MPU), first time information (MPU time stamp) generated by the generation unit 1201, and first time information (MPU time stamp). The second time information (relative information) indicating the presentation time (PTS) or the decoding time (DTS) of each unit (AU) and the identification information are transmitted.

The receiving device 1300 includes a receiving unit 1301, a calculating unit 1302, and a correcting unit 1303. Each of the reception unit 1301, the calculation unit 1302, and the calculation unit 1303 is realized by, for example, a microcomputer, a processor, a dedicated circuit, or the like.

A detailed description of each component of the receiving device 1300 will be given in the description of the receiving method.

The receiving method will be described with reference to FIG. FIG. 103 is an operation flow (reception method) by the receiving device.

First, the receiving unit 1301 of the receiving device 1300 receives the first data unit (MPU), the first time information (MPU time stamp), the second time information (relative information), and the identification information (S3001). ).

Next, the calculation unit 1302 of the reception device 1300 uses the first time information (MPU time stamp) and the second time information (relative information) received by the reception unit 1301 to determine a plurality of units received by the reception unit 1302. The presentation time (PTS) or the decoding time (DTS) of each of the second data units (AU) is generated (S3002).

Next, the correction unit 1303 of the reception device 1300, based on the identification information received by the reception unit 1301, the presentation time (PTS) of each of the plurality of second data units (AU) calculated by the calculation unit 1302 or The decoding time (DTS) is corrected (S3003).

Accordingly, the receiving device 1300 can normally reproduce the plurality of second data units stored in the first data unit even when the leap second adjustment is performed.

In the eighth embodiment, reproduction means a process including at least one of decoding and presentation.

(Other embodiments)
Although the transmission device, the reception device, the transmission/reception system, the transmission method, and the reception method according to the embodiment have been described above, the present invention is not limited to this embodiment.

Further, each processing unit included in the transmission device and the reception device according to the above embodiments is typically realized as an LSI that is an integrated circuit. These may be individually made into one chip, or may be made into one chip so as to include a part or all of them.

Further, the integrated circuit is not limited to the LSI, and may be realized by a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after the LSI is manufactured, or a reconfigurable processor that can reconfigure connection and settings of circuit cells inside the LSI may be used.

In each of the above-described embodiments, each component may be configured by dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory.

In other words, the transmitting device and the receiving device include a processing circuit and a storage device electrically connected to the processing circuit (accessible from the control circuit). The processing circuit includes at least one of dedicated hardware and a program execution unit. Further, when the processing circuit includes a program execution unit, the storage device stores the software program executed by the program execution unit. The processing circuit uses the storage device to execute the transmitting method or the receiving method according to the above-described embodiment.

Further, the present invention may be the above software program or a non-transitory computer-readable recording medium in which the above program is recorded. Needless to say, the above program can be distributed via a transmission medium such as the Internet.

Further, all the numbers used above are examples for specifically explaining the present invention, and the present invention is not limited to the exemplified numbers.

Also, the division of functional blocks in the block diagram is an example, and multiple functional blocks are realized as one functional block, one functional block is divided into multiple, and some functions are transferred to other functional blocks. May be. Further, the functions of a plurality of functional blocks having similar functions may be processed in parallel or in time division by a single piece of hardware or software.

Further, the order in which the steps included in the above transmission method or reception method are executed is for the purpose of illustrating the present invention in detail, and may be other than the above order. In addition, some of the above steps may be executed simultaneously (in parallel) with other steps.

Although the transmission device, the reception device, the transmission method, and the reception method according to one or more aspects of the present invention have been described above based on the embodiments, the present invention is not limited to the embodiments. Absent. As long as it does not depart from the gist of the present invention, various modifications made by those skilled in the art may be applied to the present embodiment, or a configuration constructed by combining components in different embodiments may be one or more of the present invention. It may be included in the range of the aspect.

INDUSTRIAL APPLICABILITY The present invention can be applied to an apparatus or device that performs media transport of video data and audio data.

15, 100, 300, 500, 700, 900, 1200 Transmitting device 16, 101, 301 Encoding unit 17, 102 Multiplexing unit 18, 104 Transmitting unit 20, 200, 400, 600, 800, 1000, 1300 Receiving device 21 Packet filtering unit 22 Transmission order type determination unit 23 Random access unit 24, 212 Control information acquisition unit 25 Data acquisition unit 26 Calculation unit 27 Initialization information acquisition unit 28, 206 Decoding instruction unit 29, 204A, 204B, 204C, 204D, 402 Decoding unit 30 Presentation unit 201 Tuner 202 Demodulation unit 203 Demultiplexing unit 205 Display unit 211 Type determination unit 213 Slice information acquisition unit 214 Decoded data generation unit 302 Addition unit 303, 503, 702, 902, 1202 Transmission unit 401, 601, 801, 1001, 1301 Reception unit 501 Division unit 502, 603 Configuration unit 602 Judgment unit 701, 901, 1201 Generation unit 802 First buffer 803 Second buffer 804 Third buffer 805 Fourth buffer 806 Decoding unit 1002 Playback Unit 1302 Calculation unit 1303 Correction unit

Claims (4)

A receiving method for receiving a first data unit in which data forming an encoded stream is stored, comprising:
The first data unit stores a plurality of second data units,
In the receiving method,
Used to calculate the presentation time or decoding time of each of the plurality of second data units together with the first data unit, first time information indicating the presentation time of the first data unit, and the first time information. Received second time information and identification information,
Using the received first time information and the second time information, the presentation time or the decoding time of each of the plurality of received second data units is calculated,
Correcting the presentation time or the decoding time of each of the calculated second data units based on the received identification information, for each of the plurality of second data units,
The identification information is information indicating whether the first time information is generated based on reference time information before leap second adjustment,
The receiving method, wherein the first time information is time information indicating a presentation time of a head second data unit in the presentation order among the plurality of second data units. A transmission method for transmitting a first data unit in which data forming an encoded stream is stored, comprising:
The first data unit stores a plurality of second data units,
In the transmission method,
Generating first time information indicating a presentation time of the first data unit based on reference time information received from the outside,
The first data unit, the generated first time information, and second time information used to calculate the presentation time or decoding time of each of the plurality of second data units together with the first time information, Send identification information and
The identification information is information indicating whether the first time information is generated based on reference time information before leap second adjustment,
The transmission method in which the first time information is time information indicating a presentation time of a first second data unit in the presentation order among the plurality of second data units. A receiving device for receiving a first data unit in which data forming an encoded stream is stored,
The first data unit stores a plurality of second data units,
The receiving device,
Used to calculate the presentation time or decoding time of each of the plurality of second data units together with the first data unit, first time information indicating the presentation time of the first data unit, and the first time information. A receiving unit that receives the second time information and the identification information,
A calculator that calculates the presentation time or the decoding time of each of the plurality of second data units received by the receiver, using the first time information and the second time information received by the receiver. When,
The presentation time or the decoding time of each of the plurality of second data units calculated by the calculator based on the identification information received by the receiver is calculated for each of the plurality of second data units. And a correction unit for correcting
The identification information is information indicating whether the first time information is generated based on reference time information before leap second adjustment,
The receiving device, wherein the first time information is time information indicating a presentation time of a head second data unit in the presentation order among the plurality of second data units. A transmitting device for transmitting a first data unit in which data constituting an encoded stream is stored, comprising:
The first data unit stores a plurality of second data units,
The transmitter is
A generation unit that generates first time information indicating a presentation time of the first data unit based on reference time information received from the outside;
A second data unit, the first time information generated by the generating unit, and a second time used together with the first time information to calculate a presentation time or a decoding time of each of the plurality of second data units. A transmission unit for transmitting time information and identification information,
The identification information is information indicating whether the first time information is generated based on reference time information before leap second adjustment,
The transmission device, wherein the first time information is time information indicating a presentation time of a head second data unit in the presentation order among the plurality of second data units.

Images