JP7306527B2 - decoding device - Google Patents

Description

The present invention relates to a decoding device for decoding encoded data multiplexed in a bitstream in which video signals and audio signals are encoded.

In Japanese digital broadcasting, as described in Non-Patent Document 1 below, video streams and audio streams, which are encoded data of video signals and audio signals, are encoded according to MPEG-2 (Moving Picture Experts Group Phase-2). are multiplexed and transmitted in the transport stream (TS) format, which is the system standard of the . At this time, the encoding device also multiplexes and transmits encoded data of metadata related to the video stream and the audio stream together with the video stream and the audio stream.

In addition to the transport stream (TS) in MPEG-2, there is MMT (MPEG Media Transport) as a new transport system being standardized by MPEG. When transmitting a video component (video stream) and an audio component (audio stream), each component can be transmitted in a different transmission mode (eg, broadcasting, communication, etc.).

Here, HEVC/H. H.265 (hereafter referred to as “HEVC”) is a new video coding system standardized by MPEG and ITU (International Telecommunications Union).
In HEVC, temporal hierarchical coding (coding that is scalable in the temporal direction) has been introduced, and NAL ( (Network Abstraction Layer) A hierarchical level can be specified for each unit.

FIG. 9 is an explanatory diagram showing an example of temporal layered coding in HEVC.
In FIG. 9, TemporalID is identification information indicating the hierarchical level of each access unit (AU).
IRAP is an IRAP (Intra random access point) picture defined by HEVC, and when decoding is started from the middle of a bitstream, pictures following the IRAP picture in display order are decoded normally. is guaranteed.
GOP (Group Of Pictures) is capable of decoding all of the video signals of one or more access units (AU) when the video signals of one or more access units (AU) are encoded by the inter-frame prediction coding method. A set of possible Access Units (AUs). That is, it is a set of an IRAP picture that is the first access unit (AU) in coding order and an access unit (AU) following the IRAP picture (pictures other than the IRAP picture).

Since the content of temporal layered coding is well known, a detailed description is omitted. ) cannot be referenced.
By providing such restrictions, for example, in the example of FIG. 11, access units (AUs) at hierarchy level 2 or lower (TemporalID≦2) refer to access units (AUs) at hierarchy level 3 (TemporalID=3) during decoding. Therefore, it is possible to decode access units (AUs) at hierarchy level 2 or lower (TemporalID≦2) without decoding access units (AUs) at hierarchy level 3. FIG.
In HEVC, temporal hierarchical coding is possible using a reference structure with a maximum of 6 hierarchies.

FIG. 10 is an explanatory diagram showing the encoding order and display order of each picture encoded in the picture structure of FIG.
As shown in FIG. 10, if access units at hierarchy level 3 and access units at hierarchy levels 2 and below are encoded alternately in display order, all access units from hierarchy level 0 to hierarchy level 3 are encoded. When the display frame rate when decoded is 2N (Hz), when only the access units of hierarchical level 2 or lower are decoded, they can be reproduced at the display frame rate of N (Hz). Therefore, if the display frame rate is reproduced by a decoding device that supports N (Hz) or less, only the access units of hierarchical level 2 or less need to be passed to the decoding device.

For example, in MMT, when multiplexing and distributing a video bitstream configured as shown in FIG. Identifiers of different values can be assigned to sets of configured access units for distribution. In MMT, a set of access units assigned the same identifier is called an asset. It is also possible to transmit the identifier of an asset composed of access units at hierarchical level 2 or lower as A ₀ and the identifier of an asset composed of access units at hierarchical level 3 as A ₁ in a different transmission format for each asset. , for example, asset A ₀ can be transmitted over the air and asset A ₁ can be transmitted over the air.
In MMT, in order to synchronize the presentation time between assets, a descriptor is prepared that describes the presentation time of the first access unit in the display order for each GOP in NTP (Network Time Protocol) format. This descriptor can be multiplexed and transmitted. Also, if the assets are different, the presentation time of the first access unit can be transmitted for each asset, and even if multiple assets transmitted in different transmission formats are received by the receiving side, they can be played back in synchronization with the presentation time. You can (present).

STD-B32 (standard for digital broadcasting established by ARIB (Association of Radio Industries and Businesses))

Since the conventional encoding apparatus is configured as described above, a bitstream of time-hierarchically encoded video is configured into different assets according to the hierarchical level of each access unit as shown in FIG. When transmission is performed using a different transmission format, it is necessary for the decoding device to reconstruct a bitstream in the same order as the encoding order as shown in FIG. 10 based on the decoding timing of each access unit. Since the decoding time of each access unit cannot be transmitted, there is a problem that the bitstream cannot be reconstructed.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-described problems. It is an object of the present invention to obtain a decoding device capable of reconstructing and decoding a time-hierarchically-encoded bitstream based on the decoding timing of each access unit in the above.

A decoding device according to the present invention relates to one or more components that constitute one program, and is a decoding device that decodes encoded data of a video signal in an MMT capable of transmitting data in a different transmission format for each component, Indicates the presentation time of the top access unit in presentation order in one or more GOPs, which are a set of multiple access units encoded by the inter-frame predictive encoding method, from the descriptor included in the MMTP payload of the encoded data. presentation time information, time difference information between the decoding time and the presentation time of the first access unit in the encoding order different from the access unit, and a unit representing presentation time information and display time information encoded in units of access units , and the presentation time information, the time difference information, and the unit acquired by the control information decoding means are used to calculate the presentation time and the decoding time of each access unit, and the video included in the encoded data and decoding means for decoding the signal.

According to the present invention, even when a bit stream of time-hierarchy-encoded video is configured and transmitted as different assets according to the hierarchy level of each access unit, the decoding device performs time-hierarchy encoding based on the decoding timing of each access unit. Advantageously, the encoded bitstream can be reconstructed and decoded.

1 is a configuration diagram showing an encoding device according to Embodiment 1 of the present invention; FIG. 4 is a flow chart showing the processing contents (encoding method) of the encoding device according to Embodiment 1 of the present invention; 1 is a configuration diagram showing a decoding device according to Embodiment 1 of the present invention; FIG. 4 is a flow chart showing the processing contents (decoding method) of the decoding device according to Embodiment 1 of the present invention; FIG. 4 is an explanatory diagram showing an outline of encoded data when transmitting a bitstream in MMT; It is an explanatory view showing an example of composition of MPU. FIG. 4 is an explanatory diagram showing an HEVC picture structure descriptor; FIG. 4 is an explanatory diagram showing an example of temporal layered coding in HEVC; FIG. 4 is an explanatory diagram showing an example of a picture structure; FIG. 10 is an explanatory diagram showing the encoding order and the presentation order of each picture encoded in the picture structure of FIG. 9; FIG. 4 is an explanatory diagram showing the structure of a PA message; FIG. 3 is an explanatory diagram showing an example of a bitstream before separation and an example of a bitstream after separation;

Embodiment 1.
FIG. 1 is a configuration diagram showing an encoding apparatus according to Embodiment 1 of the present invention.
In FIG. 1, an audio encoder 1 receives a digital audio signal, encodes the audio signal in units of audio access units (AU), for example, using MPEG-4 audio or the like, and converts the audio signal into It generates an audio stream, which is the encoded data of the audio stream, and encodes metadata related to the audio stream. It also outputs the encoded presentation time (PTS) of the access unit to the voice MMTP packet generator 8 .
The audio MMTP payload generator 2 performs processing to generate an audio MMTP payload consisting of the metadata encoded by the audio encoder 1 and the encoded data of the audio signal in access unit (AU) units.

When a digital video signal is supplied, the HEVC encoding unit 3 encodes the video signal by the HEVC method for each access unit (AU) of the video, and generates a video stream which is the encoded data of the video signal. Along with this, a process of encoding metadata relating to the video stream is performed.
The video MMTP payload generation unit 4 performs processing for generating a video MMTP payload consisting of the metadata encoded by the HEVC encoding unit 3 and the encoded data of the video signal in access unit (AU) units. The HEVC encoding unit 3 and the video MMTP payload generation unit 4 constitute video encoding means.

The control information encoder 5 encodes control information called a PA message defined by MMT as control information relating to the audio stream generated by the audio encoder 1 and the video stream generated by the HEVC encoder 3. process to be performed.
FIG. 11 shows the structure of the PA message. A PA message consists of one or more tables.
One table included in the PA message describes information about one or more video components (video streams) and audio components (audio streams) that make up one program (referred to as a package in MMT). In MMT, video and audio components are called assets.

Specifically, the asset ID that identifies the asset, the asset type that identifies the asset type (such as HEVC format video stream or MPEG-4 audio format audio stream), and the encoded data and metadata of each asset. The packet ID indicating the stored MMTP packet or the IP address in the case of IP distribution, information on the acquisition destination of the asset, and various descriptors for describing the meta information about each asset are as many as the number of assets that make up the package. only included in the table.
In the descriptor, the presentation time (display time) of the leading access unit (AU) in the presentation order (display order) among the access units (AU) that make up the MPU (Media Processing Unit) of each asset. In addition to the descriptors defined by the MMT standard, such as the MPU time stamp descriptor (presentation time information) shown below, users can also define their own new descriptors. child is included.

The MPU is composed of one or more access units (AUs), and is a unit capable of decoding video and audio by itself. Further, when the video signal of one or more access units (AU) is encoded by the inter-frame prediction coding method, the MPU decodes all the video signals of the one or more access units (AU). is the same unit as a GOP, which is a set of multiple access units (AUs) in which .

The MPU time information descriptor includes information (timescale) indicating the unit for describing time information such as decoding time (DTS) and presentation time (PTS), and code Whether information for calculating the decoding time of the first access unit (AU) in the encoding order (initial_presentation_time_delay) and information for calculating the decoding time and presentation time of each access unit constituting the MPU are encoded Flags indicating whether or not (presentation_time_offset_present_flag, decoding_time_offset_present_flag), information indicating the code length when encoding information for calculating the decoding time and presentation time of each access unit (time_offset_length_minus1), and the like are described.

The control MMTP payload generation unit 6 performs processing for generating a control MMTP payload consisting of the encoded data of the control information encoded by the control information encoding unit 5 .
A part of the control information encoding unit 5 and the control MMTP payload generation unit 6 constitute control information encoding means. Another part of the control information encoding unit 5 constitutes time information encoding means.

The video MMTP packet generation unit 9 adds a predetermined MMTP header to the video MMTP payload generated by the video MMTP payload generation unit 4 to generate video MMTP packets forming a bitstream. The MMTP header is composed of a mandatory header containing mandatory encoded information and an extension header containing optionally encoded information. The essential header includes a packet ID and the like assigned according to the type of encoded data included in the MMTP payload.
The extension header contains a flag value indicating whether or not to encode information (presentation time information and decoding time information) for calculating the presentation time and decoding time for each access unit of the encoded data included in the MMTP payload. Accordingly, presentation time information (presentation_time_offset) and decoding time information (decoding_time_offset) are included.

The audio MMTP packet generator 8 adds a predetermined MMTP header to the audio MMTP payload generated by the audio MMTP payload generator 2 to generate audio MMTP packets forming a bitstream. The MMTP header is composed of a mandatory header containing mandatory encoded information and an extension header containing optionally encoded information. The content of the extension header is the same as the extension header encoded by the video MMTP packet generator.

The control MMTP packet generation unit 10 adds a predetermined MMTP header to the control MMTP payload generated by the control MMTP payload generation unit 6 to generate control MMTP packets forming a bitstream.

The MMTP packet multiplexer 7 combines the audio MMTP packets generated by the audio MMTP packet generator, the control MMTP packets generated by the control MMTP packet generator, and the video MMTP packets generated by the video MMTP packet generator. A process of multiplexing and constructing a bitstream is performed.
The MMTP packet multiplexer 7 can also compose a different bitstream for each asset. For example, as shown in FIG. 9, a bitstream 1 composed of MMTP packets of an asset including access units of layer level 2 or lower of a temporally layered encoded video bitstream and an MMTP of an asset including access units of layer level 3 It is also possible to send each bitstream in a different transmission form as a bitstream 2 composed of packets.
The MMTP packet multiplexing unit 7 constitutes multiplexing means.

In the example of FIG. 1, an audio encoder 1, an audio MMTP payload generator 2, an HEVC encoder 3, a video MMTP payload generator 4, a control information encoder 5, and a control MMTP payload, which are components of the encoder. It is assumed that each of the generation unit 6 and the control MMTP packet generation unit 10 is composed of dedicated hardware (for example, a semiconductor integrated circuit mounting a CPU, or a one-chip microcomputer, etc.). , the encoding device may be composed of a computer.
When the encoding device is configured by a computer, an audio encoding unit 1, an audio MMTP payload generation unit 2, an HEVC encoding unit 3, a video MMTP payload generation unit 4, a control information encoding unit 5, a control MMTP payload generation unit 6 and A program describing the processing contents of the control MMTP packet generator 10 and the like may be stored in the memory of the computer, and the CPU of the computer may execute the program stored in the memory.

FIG. 2 is a flow chart showing the processing contents (encoding method) of the encoding device according to Embodiment 1 of the present invention.

FIG. 3 is a configuration diagram showing a decoding device according to Embodiment 1 of the present invention.
In FIG. 3, the MMTP packet analysis unit 12 includes one or more assets including one or more assets output from an encoding device (the encoding device in FIG. 1 or an encoding device corresponding to the encoding device in FIG. 1). bitstream input. The MMTP packet analysis unit 12 analyzes the MMTP header of the MMTP packet that constitutes the bitstream, acquires the packet ID contained in the MMTP header, and converts the packet ID into the code contained in the MMTP payload. If the modified data indicates that it is control information (PA message), it outputs the control MMTP payload, which is the MMTP payload contained in the MMTP packet, to the control MMTP payload processing unit 13 .

The control MMTP payload processing unit 13 decodes the encoded data included in the control MMTP payload output from the MMTP packet analysis unit 12 to decode the PA message, which is control information.
In addition, the control MMTP payload processing unit 13 uses a packet ID indicating an MMTP packet that stores information about assets constituting a package, encoded data and metadata of each asset from a table described in the PA message, or an IP-delivered packet ID. Decrypt information about the source of the asset, such as the IP address, if any. Information about the packet ID and the acquisition destination of the asset is output to the MMTP packet analysis unit.
Also, the control MMTP payload processing unit 13 decodes the MPU time stamp descriptor and MPU time information descriptor relating to the assets forming the package from the table described in the PA message.

The MMTP packet analysis unit 12 acquires the packet ID contained in the MMTP header, compares the acquired packet ID with the packet ID of each asset output from the control MMTP payload processing unit 13, and determines that the packet ID is the MMTP If the coded data included in the payload indicates that it is an audio signal or a video signal, the process of outputting the MMTP packet to the asset separation unit 14 is performed.

The MMTP packet analysis unit 12 also provides information for calculating the decoding time and the presentation time of each access unit constituting the MPU described in the MPU time information descriptor decoded by the control MMTP payload processing unit 13. Presentation_time_offset) and decoding time information (decoding_time_of fset) and describe it in the MPU timestamp descriptor information (initial_presentation_time_delay) for calculating the presentation time of the first access unit (AU) in the presented order and the decoding time of the first access unit (AU) in the encoding order described in the MPU time information descriptor (initial_presentation_time_delay) The presentation time and decoding time of each access unit (AU) are calculated from the decoding time of the first access unit (AU) in the encoding order obtained by decoding the . The calculated presentation time and decoding time are output to audio MMTP payload processing section 15 and video MMTP payload processing section 19 according to the type of encoded data included in the access unit.

The asset separation unit 14 refers to the asset ID, asset type, and packet ID described in the table of the PA message decoded by the control MMTP payload processing unit 13, and includes the MMTP packet output from the MMTP packet analysis unit 12. Identify whether the MMTP payload is an audio MMTP payload or a video MMTP payload, and if it is an audio MMTP payload, extract the audio MMTP payload contained in the MMTP packet, and extract the audio MMTP payload is output to the audio MMTP payload processing unit 15, and if it is a video MMTP payload, the video MMTP payload included in the MMTP packet is extracted, and the video MMTP payload is output to the video MMTP payload processing unit 19. implement.

The audio MMTP payload processing unit 15 reconstructs the MFU (Media Fragment Unit) or MPU of the audio stream from the audio MMTP payload output from the asset separation unit 14, thereby converting the data into a format that can be decoded by the audio stream decoding unit 17 in the subsequent stage. A process of generating an audio elementary stream (audio ES) and storing the audio ES in the audio ES buffer 16 is performed. MFU is a unit smaller than MPU, and one access unit (AU) or one NAL unit can be defined as one MFU.
Also, the audio MMTP payload processing unit 15 extracts metadata about the audio stream included in the audio MMTP payload output from the asset separation unit 14 and stores the metadata in the audio ES buffer 16 . The audio ES buffer 16 is a memory that temporarily stores audio ES and metadata.

When the DTS (decoding time) of each access unit (AU) comes, the audio stream decoding unit 17 extracts the audio ES from the audio ES buffer 16, decodes the audio signal of the access unit (AU), and converts the decoded audio A process of storing the signal and the PTS (presentation time) in the audio data buffer 18 is performed.
The audio data buffer 18 is a memory that temporarily stores the audio signal decoded by the audio stream decoding unit 17 and the PTS (presentation time).

The video MMTP payload processing unit 19 reconstructs the MFU or MPU of the video stream from the video MMTP payload output from the asset separation unit 14, thereby generating an HEVC elementary stream (HEVC ES) and stores the HEVC elementary stream in the HEVCES buffer 20 .
Also, the video MMTP payload processing unit 19 extracts metadata related to the video stream included in the video MMTP payload output from the asset separation unit 14 and stores the metadata in the HEVCES buffer 20 .
The HEVCES buffer 20 is a memory that temporarily stores HEVC elementary streams and metadata.

When the DTS (decoding time) of each access unit (AU) comes, the HEVCES decoding unit 21 extracts the HEVC elementary stream from the HEVCES buffer 20, decodes the video signal of the access unit (AU), and decodes the decoded video signal. and the PTS (presentation time) in the decoded image buffer 22 .
The decoded image buffer 22 is a memory that temporarily stores the decoded image of each access unit (AU) decoded by the HEVCES decoding section 21 and the PTS (presentation time).
The video MMTP payload processing unit 19, HEVCES buffer 20, HEVCES decoding unit 21, and decoded image buffer 22 constitute video decoding means.

In the example of FIG. 3, MMTP packet analysis unit 12, control MMTP payload processing unit 13, asset separation unit 14, audio MMTP payload processing unit 15, audio ES buffer 16, audio stream decoding unit 17, audio Each of the data buffer 18, the video MMTP payload processing unit 19, the HEVCES buffer 20, the HEVCES decoding unit 21, and the decoded image buffer 22 is dedicated hardware (other than the buffers, for example, a semiconductor integrated circuit implementing a CPU, or A one-chip microcomputer, etc.) is assumed, but the decoding device may be configured by a computer.
When the decoding device is configured by a computer, the audio ES buffer 16, the audio data buffer 18, the HEVCES buffer 20, and the decoded image buffer 22 are configured on the internal memory or external memory of the computer, and the MMTP packet analysis unit 12, the control MMTP payload A program describing the processing contents of the processing unit 13, the asset separation unit 14, the audio MMTP payload processing unit 15, the audio stream decoding unit 17, the video MMTP payload processing unit 19, and the HEVCES decoding unit 21 is stored in the memory of the computer, The CPU of the computer may execute the program stored in the memory.

FIG. 4 is a flow chart showing the processing contents (decoding method) of the decoding device according to Embodiment 1 of the present invention.

Next, the operation will be explained.
The processing contents of the first encoding device will be explained.
When a digital audio signal is supplied, the audio encoding unit 1 encodes the audio signal in units of audio access units (AUs) by, for example, MPEG-4 audio, and encodes the audio signal. An audio stream, which is data, is generated, and metadata about the audio stream is encoded (step ST1 in FIG. 2).
When a digital video signal is supplied, the HEVC encoding unit 3 encodes the video signal by the HEVC method in units of video access units (AU), and generates a video stream that is the encoded data of the video signal. At the same time, it encodes metadata about the video stream (step ST2).

Here, FIG. 5 is an explanatory diagram showing an overview of encoded data when transmitting a bitstream in MMT.
In FIG. 5, an access unit (AU) is a unit containing coded data necessary to decode one picture in the case of video, and is composed of one or more samples that are a coding unit in the case of audio. is a frame that is
A NAL unit is an HEVC coding unit, and one access unit (AU) is composed of one or more NAL units.
The MPU is composed of one or more access units, and is a unit capable of decoding video and audio by itself. Further, when the video signal of one or more access units (AU) is encoded by the inter-frame prediction coding method, the MPU decodes all the video signals of the one or more access units (AU). is the same unit as a GOP, which is a set of multiple access units (AUs) in which .
MFU is a unit smaller than MPU, and one access unit (AU) or one NAL unit can be defined as one MFU.

FIG. 6 is an explanatory diagram showing a configuration example of the MPU.
In FIG. 6, the MPU metadata describes metadata related to the MPU. Note that the MPU metadata may not be encoded.
Movie fragment metadata (MF meta) describes metadata attached to encoded data (sample data) of one access unit (AU). For example, when the encoded data of the access unit (AU) is stored in a file format, for each access unit (AU), the address where the encoded data is stored, the data length of the encoded data, the access unit (AU ) is included. Note that movie fragment metadata does not have to be encoded.
MPU metadata, movie fragment metadata, MFU and MMT control information are MMTP packetized and transmitted. An MMTP packet consists of an MMTP header and an MMTP payload.

When audio MMTP payload generation unit 2 receives encoded data of metadata (MPU metadata, MF meta, etc.) and encoded data of an audio signal in access unit (AU) units from audio encoding unit 1, MPU An audio MMTP payload consisting of encoded data of MPU metadata in units, encoded data of MF meta in units of access units (AU), and encoded data (sample data) of audio signals is generated (step ST3).
When the video MMTP payload generation unit 4 receives the encoded data of the metadata (MPU metadata, MF meta, etc.) from the HEVC encoding unit 3 and the encoded data of the video signal in access unit (AU) units, the MPU A video MMTP payload consisting of encoded data of MPU metadata in units, encoded data of MF meta in units of access units (AU), and encoded data (sample data) of video signals is generated (step ST4).

The control information encoding unit 5 encodes control information regarding the audio stream generated by the audio encoding unit 1 and the video stream generated by the HEVC encoding unit 3 (step ST5).
As control information related to the audio stream and the video stream, for example, the PA message and MPU time information descriptor defined by MMT are encoded.
As described above, the PA message describes information about one or more video components (video streams) and audio components (audio streams) that make up one program (referred to as a package in MMT).
That is, the PA message includes an asset ID that identifies the asset (video stream, audio stream) generated by the audio encoder 1 and the HEVC encoder 3, an asset type that identifies the type of asset, and an MPU of each asset. An MPU timestamp descriptor that indicates the presentation time of the leading access unit (AU) in presentation order among the constituent access units (AU), and an MMTP packet that stores the encoded data and metadata of each asset. A packet ID indicating the .

FIG. 7 is an explanatory diagram showing the MPU time information descriptor.
As shown in FIG. 7, the MPU time information descriptor contains a sequence number (mpu_sequence_number) for identifying which MPU related information is included, and the decoding time and presentation of the MPU head access unit in encoding order. In order, the time difference (initial_presentation_time_delay) of the presentation time of the access unit at the head of the MPU, the unit (timescale) representing presentation time information and display time information encoded in access unit units (1/timescale seconds), and in access unit units A flag indicating whether to encode the presentation time information (presentation_time_offset_present_flag), a flag indicating whether to encode the decoding time information in units of access units (decoding_time_offset_present_flag), and presentation time information encoded in units of access units. and the code length of display time information (time_offset_length_minus1). Note that timescale, presentation_time_offset_present_flag, decoding_time_offset_present_flag, and time_offset_length_minus1 do not need to be coded if fixed values are always used.

The control MMTP payload generator 6, upon receiving the coded data of the control information from the control information encoder 5, generates a control MMTP payload consisting of the coded data of the control information (step ST6).
The video MMTP packet generation unit adds a predetermined MMTP header to the video MMTP payload generated by the video MMTP payload generation unit to generate video MMTP packets forming a bitstream. The MMTP header is composed of a mandatory header containing mandatory encoded information and an extension header containing optionally encoded information. The essential header includes a packet ID and the like assigned according to the type of encoded data included in the MMTP payload.

The extension header contains a flag value indicating whether or not to encode information (presentation time information and decoding time information) for calculating the presentation time and decoding time for each access unit of the encoded data included in the MMTP payload. Accordingly, presentation time information (presentation_time_offset) and decoding time information (decoding_time_offset) are included.
The presentation time information (presentation_time_offset) is the difference between the presentation time of the access unit of the encoded data included in the MMTP payload and the presentation time of the MPU head access unit in the order of presentation.
The decoding time information (decoding_time_offset) is the difference between the decoding time of the access unit of the encoded data included in the MMTP payload and the decoding time of the MPU head access unit in the encoding order.
The presentation time information (presentation_time_offset) may encode the difference from the decoding time of the access unit calculated by decoding the decoding time information (decoding_time_offset).

The audio MMTP packet generator 8 adds a predetermined MMTP header to the audio MMTP payload generated by the audio MMTP payload generator 2 to generate audio MMTP packets forming a bitstream. The MMTP header is composed of a mandatory header containing mandatory encoded information and an extension header containing optionally encoded information.

The control MMTP packet generation unit 10 adds a predetermined MMTP header to the control MMTP payload generated by the control MMTP payload generation unit 6 to generate control MMTP packets forming a bitstream.
When generating this MMTP packet, a predetermined MMTP header is attached, and this MMTP header includes a packet ID assigned according to the type of encoded data included in the MMTP payload.

The MMTP packet multiplexer 7 combines the audio MMTP packets generated by the audio MMTP packet generator 8, the control MMTP packets generated by the control MMTP packet generator 10, and the video MMTP packets generated by the video MMTP packet generator 9. Packets are multiplexed to form a bitstream. (Step ST7)

Next, the processing contents of the decoding device will be explained.
The MMTP packet analysis unit 12 receives one or more bitstreams including one or more assets output from an encoding device (the encoding device in FIG. 1 or an encoding device corresponding to the encoding device in FIG. 1). Then, the MMTP headers of the MMTP packets forming the bitstream are analyzed, and the packet IDs contained in the MMTP headers are acquired.
If the packet ID indicates that the encoded data contained in the MMTP payload is control information (PA message, HEVC picture structure descriptor), the MMTP packet analysis unit 12 It outputs the control MNTP payload, which is the MMTP payload contained therein, to the control MNTP payload processing unit 13 .

On the other hand, if the packet ID indicates that the encoded data contained in the MMTP payload is an audio signal or video signal, the MMTP packet is output to the asset separator 14 .
The MMTP packet analysis unit 12 also provides information for calculating the decoding time and the presentation time of each access unit constituting the MPU described in the MPU time information descriptor decoded by the control MMTP payload processing unit 13. Presentation_time_offset) and decoding time information (decoding_time_of fset) and describe it in the MPU timestamp descriptor Each access unit (AU) is obtained from the presentation time of the first access unit (AU) in the presentation order and the decoding time of the first access unit (AU) in the encoding order described in the MPU time information descriptor. A process of calculating the presentation time and the decoding time is performed. The calculated presentation time and decoding time are output to audio MMTP payload processing section 15 and video MMTP payload processing section 19 according to the type of encoded data included in the access unit.

The control MMTP payload processing unit 13 decodes the encoded data included in the control MMTP payload output from the MMTP packet analysis unit 12 to decode the PA message, which is control information.
In addition, the control MMTP payload processing unit 13 uses a packet ID indicating an MMTP packet that stores information about assets constituting a package, encoded data and metadata of each asset from a table described in the PA message, or an IP-delivered packet ID. Decrypt information about the source of the asset, such as the IP address, if any. Information about the packet ID and the acquisition destination of the asset is output to the MMTP packet analysis unit.
Also, the control MMTP payload processing unit 13 decodes the MPU time stamp descriptor and MPU time information descriptor relating to the assets forming the package from the table described in the PA message.

When the control MMTP payload processing unit 13 decodes the PA message, the asset separation unit 14 refers to the asset ID, asset type, and packet ID described in the table of the PA message, and outputs from the MMTP packet analysis unit 12 It specifies whether the MMTP payload contained in the received MMTP packet is an audio MMTP payload or a video MMTP payload.

If the MMTP payload contained in the MMTP packet output from the MMTP packet analysis unit 12 is an audio MMTP payload, the asset separation unit 14 extracts the audio MMTP payload contained in the MMTP packet and extracts the audio The MMTP payload is output to the voice MMTP payload processing unit 15 .
If the MMTP payload included in the MMTP packet output from the MMTP packet analysis unit 12 is a video MMTP payload, the asset separation unit 14 extracts the video MMTP payload included in the MMTP packet and extracts the video MMTP payload. The MMTP payload is output to the video MMTP payload processing unit 19 .

When the audio MMTP payload is received from the asset separation unit 14, the audio MMTP payload processing unit 15 reconstructs the MFU or MPU of the audio stream from the audio MMTP payload, so that the audio stream decoding unit 17 can decode the payload. audio elementary stream (audio ES) is generated, and the audio ES is stored in the audio ES buffer 16 .
Since the processing itself for generating an audio ES from an audio MMTP payload is a known technique, detailed description thereof will be omitted.
Also, the audio MMTP payload processing unit 15 extracts metadata about the audio stream included in the audio MMTP payload output from the asset separation unit 14 and stores the metadata in the audio ES buffer 16 .

The audio stream decoding unit 17 refers to the DTS decoded by the MMTP packet analysis unit to grasp the decoding time of each access unit (AU). , the audio signal of the access unit (AU) is decoded, and the decoded audio signal and the PTS (presentation time) decoded by the MMTP packet analysis unit are stored in the audio data buffer 18 .
As a result, an external playback device (not shown) can reproduce the audio signal at the presentation time by extracting the audio signal and the PTS (presentation time) stored in the audio data buffer 18 .

Upon receiving the video MMTP payload from the asset separation unit 14, the video MMTP payload processing unit 19 reconstructs the MFU or MPU of the video stream from the video MMTP payload, thereby converting the MFU or MPU into a format that can be decoded by the HEVCES decoding unit 21 in the subsequent stage. A HEVC elementary stream (HEVC ES) is generated and the HEVC elementary stream is stored in the HEVCES buffer 20 .
Since the process itself of generating an HEVC elementary stream from a video MMTP payload is a known technology, detailed description thereof will be omitted.
Also, the video MMTP payload processing unit 19 extracts metadata about the video stream included in the video MMTP payload output from the asset separation unit 14 and stores the metadata in the HEVCES buffer 20 .

The HEVCES decryption unit 21 refers to the DTS decrypted by the MMTP packet analysis unit to grasp the decryption time of each access unit (AU). The elementary stream is taken out, the video signal of the access unit (AU) is decoded, and the decoded image, which is the decoded video signal, and the PTS (presentation time) decoded by the MMTP packet analysis unit are stored in the decoded image buffer 22. do.
As a result, an external reproduction device (not shown) can reproduce the decoded image at the presentation time by extracting the decoded image and the PTS (presentation time) stored in the decoded image buffer 22 .
A temporally layered encoded video bitstream composed of access units with TemporalID 0 to M is composed of a bitstream composed of access units with TemporalID 0 to (M−1) and an access unit with TemporalID M. A description will be given of processing when each bitstream is separated into separate bitstreams and transmitted using different transmission paths.
FIG. 12 shows an example of a bitstream before separation and an example of a bitstream after separation.

The MMTP packet analysis unit 12 outputs the control MMTP payloads contained in the MMTP packets forming the input bitstream to the control MMTP payload processing unit 13, and extracts the control MMTP payloads contained in the MMTP packets forming the bitstream. The audio MMTP payload or the video MMTP payload that has been stored is output to the asset separation unit 14 .

Upon receiving the control MMTP payload from the MMTP packet analysis unit 12, the control MMTP payload processing unit 13 decodes the encoded data contained in the control MMTP payload to decode the PA message, which is control information. (Step ST15).
The control MMTP payload processing unit 13 determines that the video bitstream is time-hierarchically encoded from the information about the assets described in the PA message, and that two or more assets (for example, asset 1, asset 2) acquires information such as the fact that the assets are separated from each other and acquired from different transmission paths, and the inter-asset dependency relationship (asset 2 is in a dependency relationship with asset 1).

When receiving the MMTP packet from the MMTP packet analysis unit 12, the asset separation unit 14 refers to the packet ID based on the information about the video asset described in the PA message, and extracts the video MMTP payload contained in the MMTP packet. is output to the video MMTP payload processing unit 19 . For example, if the video asset is composed of asset 1 and asset 2, the video MMTP payload for each asset is output to the video MMTP payload processing unit.

When the video MMTP payload processing unit 19 receives video MMTP payloads related to two or more video assets from the asset separation unit 14 , the video MMTP payload processing unit 19 generates HEVC elementary streams from the respective video MMTP payloads, and transfers the HEVC elementary streams to the HEVCES buffer 20 . , and the decoding time and presentation time of the access unit included in each video MMTP payload are stored in the HEVCES buffer 20 (step ST19).

The HEVCES decoding unit 21 compares the DTS of the access unit of asset 1 and the DTS of the access unit of asset 2, for example, based on the dependency relationship between asset 1 and asset 2, thereby determining the encoding order of the HEVC elementary stream before separation. can be specified, HEVC elementary streams input separately into different bitstreams can be extracted from the HEVCES buffer 20 at the correct decoding time, and the video signal of the access unit (AU) can be decoded.

As is clear from the above, according to the first embodiment, when video signals of one or more access units (AU) are temporally hierarchically encoded, different assets are configured according to the hierarchical levels of each access unit. When doing so, encode the MPU time information descriptor for each MPU that constitutes each asset, and calculate the presentation time and decoding time for each access unit of the encoded data included in the MMTP payload included in the MPU time information descriptor. Presentation time information (presentation_time_offset) and decoding time information (presentation_time_offset) and decoding time information ( decoding_time_offset), even when a bit stream of time-hierarchically-encoded video is configured and transmitted as different assets according to the hierarchical level of each access unit, the decoding device can decode each access unit. There is an effect that an encoding device and a decoding device can be obtained that can reconstruct and decode a time-hierarchically-encoded bitstream based on timing.

Within the scope of the invention, the present invention can freely combine the embodiments, modify any constituent elements of the embodiments, or omit arbitrary constituent elements from the embodiments.

1 audio encoding unit, 2 audio MMTP payload generation unit, 3 HEVC encoding unit (video encoding means), 4 video MMTP payload generation unit (video encoding means), 5 control information encoding unit (control information encoding means , time information encoding means), 6 control MMTP payload generation section (control information encoding means), 7 MMTP packet multiplexing section (multiplexing means), 8 audio MMTP packet generation section, 9 video MMTP packet generation section, 10 control MMTP packet generator, 12 MMTP packet analyzer, 13 control MMTP payload processor (presentation time calculator), 14 asset separator, 15 audio MMTP payload processor, 16 audio ES buffer, 17 audio stream decoder, 18 audio data Buffer, 19 video MMTP payload processing section (video decoding means), 20 HEVCES buffer (video decoding means), 21 HEVCES decoding section (video decoding means), 22 decoded image buffer (video decoding means).

Claims (1)

A decoding device for decoding encoded data of a video signal in an MMT capable of transmitting data in a different transmission format for each of the one or more components constituting one program,
From the descriptor included in the MMTP payload of the encoded data, the presentation time of the first access unit in the order of presentation in one or more GOPs, which is a set of a plurality of access units encoded by the inter-frame predictive encoding method. time difference information between the decoding time and the presentation time of the first access unit different from the access unit in encoding order, and presentation time information and display time information encoded in access unit units. a control information decoding means for decoding a unit;
calculating the presentation time and the decoding time of each access unit using the presentation time information, the time difference information, and the unit acquired by the control information decoding means, and calculating the video signal included in the encoded data; A decoding device comprising decoding means for decoding and .

Images