JP7067653B2 - Decoding device - Google Patents

Description

The present invention relates to a decoding device that decodes coded data in which a video signal or an audio signal is multiplexed into a coded bit stream.

In digital broadcasting in Japan, as described in Non-Patent Document 1 below, the video stream and the audio stream, which are the encoded data of the video signal and the audio signal, are MPEG-2 (Moving Picture Experts Group Phase-2). It is multiplexed and transmitted in the transport stream (TS) format, which is the system standard of. At this time, the coding apparatus also multiplexes and transmits the coded data of the metadata related to the video stream and the audio stream together with the video stream and the audio stream.

In addition to the transport stream (TS) in MPEG-2, there is MMT (MPEG Media Transfer) as a new transport method that is being standardized by MPEG, and MMT is one or more that constitutes one program. When transmitting a video component (video stream) and an audio component (audio stream), it is possible to transmit in different transmission modes (for example, broadcasting, communication, etc.) for each component.

Here, HEVC / H. 265 (hereinafter referred to as "HEVC") is a new video coding method standardized by MPEG and ITU (International Telecommunication Union).
In HEVC, time hierarchical coding (scalable coding in the time direction) is introduced, and NAL (a unit containing coding data necessary for decoding one picture) is a coding unit. Network Operation Layer) Hierarchical level can be specified for each unit.

FIG. 9 is an explanatory diagram showing an example of time-hierarchical coding in HEVC.
In FIG. 9, the Temporal ID is identification information indicating the hierarchy level of each access unit (AU).
IRAP is an IRAP (Intra Random Access Point) picture defined by HEVC, and when decoding is started from the middle of a bitstream, the pictures after the IRAP picture are normally decoded in the display order. Is guaranteed.
GOP (Group Of Pictures) can decode all the video signals of one or more access units when the video signals of one or more access units (AU) are encoded by the interframe prediction coding method. It is a set of a plurality of possible access units (AUs). That is, it is a set of an IRAP picture which is the first access unit (AU) in the coding order and an access unit (AU) (a picture other than the IRAP picture) following the IRAP picture.

Since the content of time-hierarchical coding is known, detailed description thereof will be omitted, but as a limitation of time-hierarchical coding, an access unit (AU) having a hierarchy level higher than that of the access unit (AU) to be encoded has. ) Cannot be referred to.
By providing such a constraint, for example, in the example of FIG. 11, the access unit (AU) of the hierarchy level 2 or lower (TemporalID ≦ 2) refers to the access unit (AU) of the hierarchy level 3 (TemoralID = 3) at the time of decoding. Therefore, it is possible to decode the access unit (AU) of the layer level 2 or lower (TemporalID ≦ 2) without decoding the access unit (AU) of the layer level 3.
In HEVC, time hierarchy coding with a reference structure having a maximum hierarchy of up to 6 is possible.

FIG. 10 is an explanatory diagram showing a coding order and a display order of each picture encoded by the picture structure of FIG.
As shown in FIG. 10, if the access units of the hierarchy level 3 and the access units of the hierarchy level 2 or lower are encoded so as to alternate in the display order, all the access units from the hierarchy level 0 to the hierarchy level 3 are assigned. When the display frame rate in the case of decoding is 2N (Hz) and only the access unit of the layer level 2 or lower is decoded, it can be reproduced at the display frame rate N (Hz). Therefore, when playing back with a decoding device having a display frame rate of N (Hz) or less, only the access unit of layer level 2 or lower needs to be passed to the decoding device.

For example, in MMT, when a bitstream of a video configured as shown in FIG. 10 is multiplexed and distributed, only a set of access units composed of only access units of layer level 2 or lower and an access unit of layer level 3 are used. It is possible to assign different value identifiers to each set of configured access units and distribute them. In MMT, a set of access units to which the same identifier is assigned is called an asset. It is also possible to transmit in a different transmission mode for each asset, with the identifier of the asset composed of access units of layer level ₂ or lower as A0 and the identifier of the asset composed of access units of layer level ₃ as A1. For example, it is also possible to transmit the asset A ₀ by broadcasting and the asset A ₁ by communication.
In MMT, in order to synchronize the presentation time between assets, a descriptor that describes the presentation time of the first access unit in the display order in NTP (Network Time Protocol) format is prepared for each GOP. This descriptor can be multiplexed and transmitted. Also, if the assets are different, the presentation time of the first access unit can be transmitted for each asset, and even if multiple assets transmitted in different transmission modes are received by the receiving side, they will be played back in synchronization with the presentation time. Can be (presented).

STD-B32 (Standard for digital broadcasting established by ARIB (Association of Radio Industries and Businesses))

Since the conventional coding device is configured as described above, the bitstream of the time-layer-encoded video is configured as a different asset according to the layer level of each access unit as shown in FIG. 9, and each asset is configured. When transmission is performed using different transmission modes, it is necessary to reconstruct the bitstreams in the same order as the coding order as shown in FIG. 10 based on the decoding timing of each access unit in the decoding device. Since the decryption time of each access unit cannot be transmitted, there is a problem that the bit stream cannot be reconstructed.

The present invention has been made to solve the above-mentioned problems, and even when a bitstream of time-layer-encoded video is configured and transmitted as different assets according to the layer level of each access unit, a decoding device is used. It is an object of the present invention to obtain a decoding device capable of reconstructing and decoding a bit stream encoded in a time hierarchy based on the decoding timing of each access unit.

The decoding device according to the present invention is a decoding device that decodes coded data of a video signal in an MMT capable of transmitting data in a transmission format different for each component with respect to one or more components constituting one program. From the coded data, the presentation time information indicating the presentation time of the first access unit in the presentation order in one or more GOPs, which is a set of a plurality of access units encoded by the inter-frame predictive coding method, and the access unit. Unlike the control information decoding means for obtaining the time difference information between the decoding time and the presentation time of the first access unit in the coding order, and the unit representing the presentation time information and the display time information encoded in the unit of the access unit. , The presentation time information, the time difference information, and the unit acquired by the control information decoding means are used to calculate the presentation time and the decoding time of each access unit, and the decoding means for decoding the video signal included in the encoded data. It is prepared.

According to the present invention, even when a bitstream of time-layer-encoded video is configured and transmitted as different assets according to the layer level of each access unit, the time layer is based on the decoding timing of each access unit in the decoding device. It has the effect of being able to reconstruct and decode the encoded bitstream.

It is a block diagram which shows the coding apparatus by Embodiment 1 of this invention. It is a flowchart which shows the processing content (coding method) of the coding apparatus according to Embodiment 1 of this invention. It is a block diagram which shows the decoding apparatus by Embodiment 1 of this invention. It is a flowchart which shows the processing content (decoding method) of the decoding apparatus by Embodiment 1 of this invention. It is explanatory drawing which shows the outline of the coded data at the time of transmitting a bit stream by MMT. It is explanatory drawing which shows the structural example of MPU. It is explanatory drawing which shows the HEVC picture structure descriptor. It is explanatory drawing which shows the time layer coding example in HEVC. It is explanatory drawing which shows an example of a picture structure. It is explanatory drawing which shows the coding order and presentation order of each picture which is coded by the picture structure of FIG. It is explanatory drawing which shows the structure of a PA message. It is explanatory drawing which shows an example of the bit stream before separation and the bit stream after separation.

Embodiment 1.
FIG. 1 is a block diagram showing a coding apparatus according to the first embodiment of the present invention.
In FIG. 1, when a digital voice signal is given, the voice coding unit 1 encodes the voice signal for each voice access unit (AU) by a method such as MPEG-4 audio, and the voice signal is encoded. A voice stream which is the coded data of the above is generated, and a process of encoding the metadata related to the voice stream is performed. Further, the presentation time (PTS) of the encoded access unit is output to the voice MMTP packet generation unit 8.
The voice MMTP payload generation unit 2 performs a process of generating a voice MMTP payload composed of metadata encoded by the voice coding unit 1 and voice signal coded data for each access unit (AU).

When a digital video signal is given, the HEVC coding unit 3 encodes the video signal in the video access unit (AU) unit by the HEVC method, and generates a video stream which is encoded data of the video signal. At the same time, a process of encoding the metadata related to the video stream is performed.
The video MMTP payload generation unit 4 performs a process of generating a video MMTP payload composed of metadata encoded by the HEVC coding unit 3 and coded data of a video signal for each access unit (AU). The video coding means is composed of the HEVC coding unit 3 and the video MMTP payload generation unit 4.

The control information coding unit 5 encodes control information called a PA message defined by the MMT as control information regarding the voice stream generated by the voice coding unit 1 and the video stream generated by the HEVC coding unit 3. Perform the processing to be performed.
FIG. 11 shows the structure of the PA message. A PA message consists of one or more tables.
One table included in the PA message describes information about one or more video components (video stream) and audio components (audio stream) constituting one program (referred to as a package in MMT). In MMT, video and audio components are called assets.

Specifically, the asset ID that identifies the asset, the asset type that identifies the asset type (type such as HEVC format video stream or MPEG-4 audio format audio stream), and the coded data and metadata of each asset. Information about the acquisition destination of assets such as the packet ID indicating the stored MMTP packet or the IP address when IP is delivered, and various descriptors for describing meta information about each asset are as many as the number of assets that make up the package. Only included in the table.
In the descriptor, the presentation time (display time) of the first access unit (AU) in the presentation order (display order) among the access units (AU) constituting the MPU (Media Processing Unit) of each asset is described. In addition to the descriptor defined in the MMT standard such as the MPU time stamp descriptor (presentation time information) to be shown, the user can also define a new descriptor independently, and the MPU time information is described as an original descriptor. Includes children.

The MPU is composed of one or more access units (AUs), and is a unit capable of performing video and audio decoding processing by the MPU alone. Further, when the video signal of one or more access units (AU) is encoded by the interframe prediction coding method, the MPU decodes all the video signals of the one or more access units (AU). It is the same unit as GOP, which is a set of multiple access units (AUs) that can be used.

The MPU time information descriptor is a code in the information (timescale) indicating a unit for describing time information such as the decoding time (DTS) and the presentation time (PTS) and the access unit (AU) constituting the MPU. Whether the information for calculating the decoding time of the first access unit (AU) in the order of conversion (initial_presentation_time_delay) and the information for calculating the decoding time and presentation time of each access unit constituting the MPU are encoded. A flag indicating whether or not (presentation_time_offset_present_flag, decoding_time_offset_present_flag), information indicating the code length when encoding information for calculating the decoding time and presentation time of each access unit (time_offset_lenth), etc.

The control MMTP payload generation unit 6 performs a process of generating a control MMTP payload composed of coded data of control information encoded by the control information coding unit 5.
The control information coding unit is composed of a part of the control information coding unit 5 and the control MMTP payload generation unit 6. Further, another part of the control information coding unit 5 constitutes a time information coding means.

The video MMTP packet generation unit 9 adds a predetermined MMTP header to the video MMTP payload generated by the video MMTP payload generation unit 4 to generate a video MMTP packet constituting a bit stream. The MMTP header is composed of a required header containing information to be required and encoded and an extension header containing information to be optionally encoded. The required header includes a packet ID and the like assigned according to the type of coded data included in the MMTP payload.
The extension header is a value of a flag indicating whether or not to encode information for calculating the presentation time and the decoding time (presentation time information and decoding time information) for each access unit of the coded data included in the MMTP payload. Accordingly, presentation time information (presentation_time_offset) and decoding time information (decoding_time_offset) are included.

The voice MMTP packet generation unit 8 adds a predetermined MMTP header to the voice MMTP payload generated by the voice MMTP payload generation unit 2 to generate a voice MMTP packet constituting a bit stream. The MMTP header is composed of a required header containing information to be required and encoded and an extension header containing information to be optionally encoded. The content of the extended header is the same as that of the extended header encoded by the video MMTP packet generation unit.

The control MMTP packet generation unit 10 adds a predetermined MMTP header to the control MMTP payload generated by the control MMTP payload generation unit 6, and generates a control MMTP packet constituting a bit stream.

The MMTP packet multiplexing unit 7 combines a voice MMTP packet generated by the voice MMTP packet generation unit, a control MMTP packet generated by the control MMTP packet generation unit, and a video MMTP packet generated by the video MMTP packet generation unit. Performs the process of multiplexing to form a bitstream.
The MMTP packet multiplexing unit 7 can also configure a different bitstream for each asset. For example, as shown in FIG. 9, a bitstream 1 composed of MMTP packets of an asset containing access units of layer level 2 or lower of a time-layer coded video bitstream, and MMTP of an asset including access units of layer level 3 As the bitstream 2 composed of packets, it is also possible to send each bitstream in a different transmission form.
The MMTP packet multiplexing unit 7 constitutes a multiplexing means.

In the example of FIG. 1, a voice coding unit 1, a voice MMTP payload generation unit 2, a HEVC coding unit 3, a video MMTP payload generation unit 4, a control information coding unit 5, and a control MMTP payload, which are components of the coding device, are used. It is assumed that the generator 6 and the control MMTP packet generator 10 are each composed of dedicated hardware (for example, a semiconductor integrated circuit on which a CPU is mounted, or a one-chip microcomputer). , The coding device may be composed of a computer.
When the coding device is configured by a computer, the audio coding unit 1, the audio MMTP payload generation unit 2, the HEVC coding unit 3, the video MMTP payload generation unit 4, the control information coding unit 5, the control MMTP payload generation unit 6 and A program describing the processing contents of the control MMTP packet generation unit 10 and the like may be stored in the memory of the computer, and the CPU of the computer may execute the program stored in the memory.

FIG. 2 is a flowchart showing the processing content (coding method) of the coding apparatus according to the first embodiment of the present invention.

FIG. 3 is a block diagram showing a decoding device according to the first embodiment of the present invention.
In FIG. 3, the MMTP packet analysis unit 12 has one or more assets including one or more assets output from the coding device (the coding device of FIG. 1 or the coding device corresponding to the coding device of FIG. 1). Bitstream inputs. The MMTP packet analysis unit 12 analyzes the MMTP header of the MMTP packet constituting the bit stream, acquires the packet ID contained in the MMTP header, and the code containing the packet ID in the MMTP payload. If it indicates that the conversion data is control information (PA message), the control MMTP payload, which is the MMTP payload contained in the MMTP packet, is output to the control MMTP payload processing unit 13.

The control MMTP payload processing unit 13 performs decoding processing of the coded data included in the control MMTP payload output from the MMTP packet analysis unit 12 to decode the PA message which is the control information.
Further, the control MMTP payload processing unit 13 distributes information about the assets constituting the package from the table described in the PA message, and a packet ID or IP indicating an MMTP packet storing the coded data and metadata of each asset. Decrypts the information about the acquisition destination of the asset such as the IP address in the case of. Information about the packet ID and the acquisition destination of the asset is output to the MMTP packet analysis unit.
Further, the control MMTP payload processing unit 13 decodes the MPU time stamp descriptor and the MPU time information descriptor relating to the assets constituting the package from the table described in the PA message.

The MMTP packet analysis unit 12 acquires the packet ID included in the MMTP header, collates the acquired packet ID with the packet ID of each asset output from the control MMTP payload processing unit 13, and the packet ID is MMTP. If it indicates that the coded data included in the payload is an audio signal or a video signal, a process of outputting the MMTP packet to the asset separation unit 14 is performed.

Further, the MMTP packet analysis unit 12 is information for calculating the decoding time and the presentation time of each access unit constituting the MPU described in the MPU time information descriptor decoded by the control MMTP payload processing unit 13. According to the value of the flag (presentation_time_offset_pressent_flag, decoding_time_offset_present_flag) indicating whether or not is encoded, the presentation time information (presentation_time_offset) from the MMTP extended header, the presentation time information (presentation_time_offset), and the decoding time descriptor (decoding) Information for calculating the presentation time of the first access unit (AU) in the presented order and the decryption time of the first access unit (AU) in the coding order described in the MPU time information descriptor (initial_presentation_time_delay). Is executed, and the presentation time and the decoding time of each access unit (AU) are calculated from the decoding time of the first access unit (AU) in the coding order obtained by decoding. The calculated presentation time and decoding time are output to the audio MMTP payload processing unit 15 and the video MMTP payload processing unit 19 according to the type of coded data included in the access unit.

The asset separation unit 14 refers to the asset ID, asset type, and packet ID described in the PA message table decoded by the control MMTP payload processing unit 13, and includes them in the MMTP packet output from the MMTP packet analysis unit 12. Identify whether the MMTP payload is a voice MMTP payload or a video MMTP payload, and if it is a voice MMTP payload, extract the voice MMTP payload contained in the MMTP packet and extract the voice MMTP payload. Is output to the audio MMTP payload processing unit 15, and if it is a video MMTP payload, a process of extracting the video MMTP payload contained in the MMTP packet and outputting the video MMTP payload to the video MMTP payload processing unit 19 is performed. implement.

The audio MMTP payload processing unit 15 reconstructs the MFU (Media Fragment Unit) or MPU of the audio stream from the audio MMTP payload output from the asset separation unit 14, so that the audio stream decoding unit 17 in the subsequent stage can decode the audio stream. A process of generating a voice elemental stream (voice ES) and storing the voice ES in the voice ES buffer 16 is performed. An MFU is a unit smaller than an MPU, and one access unit (AU) or one NAL unit can be defined as one MFU.
Further, the audio MMTP payload processing unit 15 extracts metadata related to the audio stream included in the audio MMTP payload output from the asset separation unit 14, and performs a process of storing the metadata in the audio ES buffer 16. The voice ES buffer 16 is a memory for temporarily storing voice ES and metadata.

When the DTS (decoding time) of each access unit (AU) is reached, the audio stream decoding unit 17 takes out the audio ES from the audio ES buffer 16, decodes the audio signal of the access unit (AU), and decodes the decoded audio. A process of storing the signal and the PTS (presentation time) in the voice data buffer 18 is performed.
The audio data buffer 18 is a memory that temporarily stores the audio signal decoded by the audio stream decoding unit 17 and the PTS (presentation time).

The video MMTP payload processing unit 19 reconstructs the MFU or MPU of the video stream from the video MMTP payload output from the asset separation unit 14, so that the HEVC elemental stream (HEVC) in a format that can be decoded by the HEVCES decoding unit 21 in the subsequent stage. ES) is generated, and the process of storing the HEVC elemental stream in the HEVCES buffer 20 is performed.
Further, the video MMTP payload processing unit 19 extracts the metadata related to the video stream included in the video MMTP payload output from the asset separation unit 14, and performs a process of storing the metadata in the HEVCES buffer 20.
The HEVCES buffer 20 is a memory for temporarily storing the HEVC elemental stream and metadata.

When the DTS (decoding time) of each access unit (AU) is reached, the HEVCES decoding unit 21 takes out the HEVC elemental stream from the HEVCES buffer 20, decodes the video signal of the access unit (AU), and decodes the decoded video signal. A process of storing the decoded image and the PTS (presentation time) in the decoded image buffer 22 is performed.
The decoded image buffer 22 is a memory that temporarily stores the decoded image and PTS (presentation time) of each access unit (AU) decoded by the HEVCES decoding unit 21.
The video decoding means is composed of the video MMTP payload processing unit 19, the HEVCES buffer 20, the HEVCES decoding unit 21, and the decoded image buffer 22.

In the example of FIG. 3, the MMTP packet analysis unit 12, the control MMTP payload processing unit 13, the asset separation unit 14, the voice MMTP payload processing unit 15, the voice ES buffer 16, the voice stream decoding unit 17, and the voice, which are the components of the decoding device, are used. Each of the data buffer 18, the video MMTP payload processing unit 19, the HEVCES buffer 20, the HEVCES decoding unit 21, and the decoded image buffer 22 has dedicated hardware (other than the buffer, for example, a semiconductor integrated circuit on which a CPU is mounted, or a semiconductor integrated circuit, or It is assumed that it is composed of a one-chip microcomputer or the like, but the decoding device may be configured by a computer.
When the decoding device is configured by a computer, the audio ES buffer 16, the audio data buffer 18, the HEVCES buffer 20, and the decoded image buffer 22 are configured on the internal memory or the external memory of the computer, and the MMTP packet analysis unit 12 and the control MMTP payload are configured. A program describing the processing contents of the processing unit 13, the asset separation unit 14, the audio MMTP payload processing unit 15, the audio stream decoding unit 17, the video MMTP payload processing unit 19, and the HEVCES decoding unit 21 is stored in the memory of the computer. The CPU of the computer may execute the program stored in the memory.

FIG. 4 is a flowchart showing the processing content (decoding method) of the decoding apparatus according to the first embodiment of the present invention.

Next, the operation will be described.
The processing contents of the first coding apparatus will be described.
When a digital voice signal is given, the voice coding unit 1 encodes the voice signal for each voice access unit (AU) by a method such as MPEG-4 audio, and encodes the voice signal. A voice stream which is data is generated, and metadata about the voice stream is encoded (step ST1 in FIG. 2).
When a digital video signal is given, the HEVC coding unit 3 encodes the video signal in the video access unit (AU) unit by the HEVC method, and generates a video stream which is encoded data of the video signal. At the same time, the metadata related to the video stream is encoded (step ST2).

Here, FIG. 5 is an explanatory diagram showing an outline of coded data when a bit stream is transmitted by MMT.
In FIG. 5, the access unit (AU) is a unit including coding data necessary for decoding one picture in the case of video, and is composed of one or more samples serving as a coding unit in the case of audio. It is a frame to be done.
The NAL unit is a coding unit of HEVC, and one access unit (AU) is composed of one or more NAL units.
The MPU is composed of one or more access units, and is a unit capable of performing video and audio decoding processing by the MPU alone. Further, when the video signal of one or more access units (AU) is encoded by the interframe prediction coding method, the MPU decodes all the video signals of the one or more access units (AU). It is the same unit as GOP, which is a set of multiple access units (AUs) that can be used.
An MFU is a unit smaller than an MPU, and one access unit (AU) or one NAL unit can be defined as one MFU.

FIG. 6 is an explanatory diagram showing a configuration example of the MPU.
In FIG. 6, the MPU metadata describes metadata related to the MPU. The MPU metadata does not have to be encoded.
The movie fragment metadata (MF meta) describes the metadata attached to the coded data (sample data) of one access unit (AU). For example, when the coded data of the access unit (AU) is stored in a file format, the address where the coded data is stored, the data length of the coded data, and the access unit (AU) are stored for each access unit (AU). ) Contains information about the length of time. The movie fragment metadata does not have to be encoded.
MPU metadata, movie fragment metadata, MFU and MMT control information are transmitted in MMTP packets. An MMTP packet consists of an MMTP header and an MMTP payload.

When the voice MMTP payload generation unit 2 receives the coded data of the metadata (MPU metadata, MF meta, etc.) and the coded data of the voice signal of the access unit (AU) unit from the voice coding unit 1, the MPU is received. A voice MMTP payload including the coded data of the MPU metadata of the unit, the coded data of the MF meta of the access unit (AU) unit, and the coded data (sample data) of the voice signal is generated (step ST3).
When the video MMTP payload generation unit 4 receives the coded data of the metadata (MPU metadata, MF meta, etc.) and the coded data of the video signal of the access unit (AU) unit from the HEVC coding unit 3, the MPU A video MMTP payload including the coded data of the MPU metadata of the unit, the coded data of the MF meta of the access unit (AU) unit, and the coded data (sample data) of the video signal is generated (step ST4).

The control information coding unit 5 encodes control information regarding the audio stream generated by the audio coding unit 1 and the video stream generated by the HEVC coding unit 3 (step ST5).
As control information related to the audio stream and the video stream, for example, a PA message specified by the MMT, an MPU time information descriptor, and the like are encoded.
As described above, the PA message describes information about one or more video components (video stream) and audio components (audio stream) constituting one program (referred to as a package in MMT).
That is, in the PA message, the asset ID that identifies the assets (video stream, audio stream) generated by the audio coding unit 1 and the HEVC coding unit 3, the asset type that identifies the asset type, and the MPU of each asset are included. Among the configured access units (AU), the MPU time stamp descriptor indicating the presentation time of the first access unit (AU) in the presentation order, and the MMTP packet storing the coded data and metadata of each asset. A packet ID or the like indicating the above is described.

FIG. 7 is an explanatory diagram showing an MPU time information descriptor.
As shown in FIG. 7, the MPU time information descriptor includes a sequence number (mpu_sequence_number) for identifying which MPU-related information is included, and the decoding time and presentation of the access unit at the head of the MPU in the coding order. In order, the time difference between the presentation times of the access unit at the head of the MPU (initial_presentation_time_delay), the unit representing the presentation time information and the display time information encoded in the unit of the access unit (timescale) (1 / timescale seconds), and the unit of the access unit. A flag indicating whether or not to encode the presentation time information (presentation_time_offset_present_flag), a flag indicating whether or not to encode the decoding time information in the unit of the access unit (decoding_time_offset_present_flag), and the presentation time information encoded in the unit of the access unit. And the code length (time_offset_length_minus1) of the display time information and the like are described. Note that timescale, presentation_time_offset_present_flag, decoding_time_offset_present_flag, and time_offset_length_minus1 do not have to be encoded if fixed values are always used.

When the control MMTP payload generation unit 6 receives the coded data of the control information from the control information coding unit 5, the control MMTP payload generation unit 6 generates a control MMTP payload composed of the coded data of the control information (step ST6).
The video MMTP packet generation unit adds a predetermined MMTP header to the video MMTP payload generated by the video MMTP payload generation unit to generate a video MMTP packet constituting a bit stream. The MMTP header is composed of a required header containing information to be required and encoded and an extension header containing information to be optionally encoded. The required header includes a packet ID and the like assigned according to the type of coded data included in the MMTP payload.

The extension header is a value of a flag indicating whether or not to encode information for calculating the presentation time and the decoding time (presentation time information and decoding time information) for each access unit of the coded data included in the MMTP payload. Accordingly, presentation time information (presentation_time_offset) and decoding time information (decoding_time_offset) are included.
The presentation time information (presentation_time_offset) is the difference between the presentation time of the access unit of the coded data included in the MMTP payload and the presentation time of the MPU head access unit in the presentation order.
The decoding time information (decoding_time_offset) is the difference between the decoding time of the access unit of the coded data included in the MMTP payload and the decoding time of the MPU head access unit in the coding order.
The presentation time information (presentation_time_offset) may encode the difference from the decoding time of the access unit calculated by decoding the decoding time information (decoding_time_offset).

The voice MMTP packet generation unit 8 adds a predetermined MMTP header to the voice MMTP payload generated by the voice MMTP payload generation unit 2 to generate a voice MMTP packet constituting a bit stream. The MMTP header is composed of a required header containing information to be required and encoded and an extension header containing information to be optionally encoded.

The control MMTP packet generation unit 10 adds a predetermined MMTP header to the control MMTP payload generated by the control MMTP payload generation unit 6, and generates a control MMTP packet constituting a bit stream.
When the MMTP packet is generated, a predetermined MMTP header is added, and the MMTP header includes a packet ID assigned according to the type of coded data included in the MMTP payload.

The MMTP packet multiplexing unit 7 includes a voice MMTP packet generated by the voice MMTP packet generation unit 8, a control MMTP packet generated by the control MMTP packet generation unit 10, and a video MMTP generated by the video MMTP packet generation unit 9. Multiplex with packets to form a bitstream. (Step ST7)

Next, the processing contents of the decoding device will be described.
The MMTP packet analysis unit 12 inputs one or more bitstreams including one or more assets output from the coding device (the coding device of FIG. 1 or the coding device corresponding to the coding device of FIG. 1). Then, the MMTP header of the MMTP packet constituting the bit stream is analyzed, and the packet ID included in the MMTP header is acquired.
If the packet ID indicates that the coded data included in the MMTP payload is control information (PA message, HEVC picture structure descriptor), the MMTP packet analysis unit 12 includes the packet ID in the MMTP packet. The control MMTP payload, which is the MMTP payload, is output to the control MMTP payload processing unit 13.

On the other hand, if the packet ID indicates that the coded data included in the MMTP payload is an audio signal or a video signal, the MMTP packet is output to the asset separation unit 14.
Further, the MMTP packet analysis unit 12 is information for calculating the decoding time and the presentation time of each access unit constituting the MPU described in the MPU time information descriptor decoded by the control MMTP payload processing unit 13. According to the value of the flag (presentation_time_offset_pressent_flag, decoding_time_offset_present_flag) indicating whether or not is encoded, the presentation time information (presentation_time_offset) from the MMTP extended header, the presentation time information (presentation_time_offset), and the decoding time descriptor (decoding) The presentation time of the first access unit (AU) in the presented order and the decryption time of the first access unit (AU) in the coding order described in the MPU time information descriptor of each access unit (AU). The process of calculating the presentation time and the decoding time is performed. The calculated presentation time and decoding time are output to the audio MMTP payload processing unit 15 and the video MMTP payload processing unit 19 according to the type of coded data included in the access unit.

The control MMTP payload processing unit 13 performs decoding processing of the coded data included in the control MMTP payload output from the MMTP packet analysis unit 12 to decode the PA message which is the control information.
Further, the control MMTP payload processing unit 13 distributes information about the assets constituting the package from the table described in the PA message, and a packet ID or IP indicating an MMTP packet storing the coded data and metadata of each asset. Decrypts the information about the acquisition destination of the asset such as the IP address in the case of. Information about the packet ID and the acquisition destination of the asset is output to the MMTP packet analysis unit.
Further, the control MMTP payload processing unit 13 decodes the MPU time stamp descriptor and the MPU time information descriptor relating to the assets constituting the package from the table described in the PA message.

When the control MMTP payload processing unit 13 decodes the PA message, the asset separation unit 14 is output from the MMTP packet analysis unit 12 with reference to the asset ID, asset type, and packet ID described in the PA message table. It is specified whether the MMTP payload included in the MMTP packet is the audio MMTP payload or the video MMTP payload.

If the MMTP payload contained in the MMTP packet output from the MMTP packet analysis unit 12 is a voice MMTP payload, the asset separation unit 14 extracts the voice MMTP payload contained in the MMTP packet and the voice MMTP payload. The MMTP payload is output to the audio MMTP payload processing unit 15.
If the MMTP payload contained in the MMTP packet output from the MMTP packet analysis unit 12 is a video MMTP payload, the asset separation unit 14 extracts the video MMTP payload contained in the MMTP packet and captures the video. The MMTP payload is output to the video MMTP payload processing unit 19.

When the audio MMTP payload processing unit 15 receives the audio MMTP payload from the asset separation unit 14, the audio MMTP payload is reconstructed from the audio MMTP payload to a format that can be decoded by the audio stream decoding unit 17 in the subsequent stage. A voice elemental stream (voice ES) is generated, and the voice ES is stored in the voice ES buffer 16.
Since the process itself for generating the voice ES from the voice MMTP payload is a known technique, detailed description thereof will be omitted.
Further, the audio MMTP payload processing unit 15 extracts metadata about the audio stream included in the audio MMTP payload output from the asset separation unit 14, and stores the metadata in the audio ES buffer 16.

The audio stream decoding unit 17 refers to the DTS decoded by the MMTP packet analysis unit, grasps the decoding time of each access unit (AU), and when the decoding time of each access unit (AU) is reached, the audio ES buffer 16 The voice ES is taken out from the voice ES, the voice signal of the access unit (AU) is decoded, and the decoded voice signal and the PTS (presentation time) decoded by the MMTP packet analysis unit are stored in the voice data buffer 18.
As a result, the external playback device (not shown) can reproduce the voice signal at the presentation time by taking out the voice signal and the PTS (presentation time) stored in the voice data buffer 18.

When the video MMTP payload processing unit 19 receives the video MMTP payload from the asset separation unit 14, it reconstructs the MFU or MPU of the video stream from the video MMTP payload, so that it can be decoded by the HEVCES decoding unit 21 in the subsequent stage. A HEVC elemental stream (HEVC ES) is generated, and the HEVC elemental stream is stored in the HEVCES buffer 20.
Since the process itself for generating the HEVC elemental stream from the video MMTP payload is a known technique, detailed description thereof will be omitted.
Further, the video MMTP payload processing unit 19 extracts the metadata related to the video stream included in the video MMTP payload output from the asset separation unit 14, and stores the metadata in the HEVCES buffer 20.

The HEVCES decoding unit 21 refers to the DTS decoded by the MMTP packet analysis unit, grasps the decoding time of each access unit (AU), and when the decoding time of each access unit (AU) is reached, the HEVCES buffer 20 to HEVC. The elemental stream is taken out, the video signal of the access unit (AU) is decoded, and the decoded image which is the decoded video signal and the PTS (presentation time) decoded by the MMTP packet analysis unit are stored in the decoded image buffer 22. do.
As a result, if an external playback device (not shown) takes out the decoded image and the PTS (presentation time) stored in the decoded image buffer 22, the decoded image can be reproduced at the presented time.
A time-hierarchically encoded video bitstream composed of access units having a Temporal ID of 0 to M, a bitstream composed of an access unit having a Temporal ID of 0 to (M-1), and an access unit having a Temporal ID of M. The processing when the bitstreams are separated into the bitstreams and each bitstream is transmitted using a different transmission line will be described.
FIG. 12 shows an example of the bit stream before separation and the bit stream after separation.

The MMTP packet analysis unit 12 outputs the control MMTP payload contained in the MMTP packet constituting the input bitstream to the control MMTP payload processing unit 13, and includes it in the MMTP packet constituting the bitstream. The audio MMTP payload or the video MMTP payload is output to the asset separation unit 14.

When the control MMTP payload processing unit 13 receives the control MMTP payload from the MMTP packet analysis unit 12, it performs decoding processing of the coded data included in the control MMTP payload and decodes the PA message which is the control information. (Step ST15).
In the control MMTP payload processing unit 13, the video bitstream is time-hierarchically encoded from the information about the assets described in the PA message, and two or more assets (for example, asset 1, asset) are used depending on the time-hierarchy level (TemporalID). Information such as the fact that it is separated into 2) and acquired from a different transmission line for each asset, and the dependency relationship between assets (asset 2 has a dependency relationship with asset 1) is acquired.

When the asset separation unit 14 receives the MMTP packet from the MMTP packet analysis unit 12, the asset separation unit 14 refers to the packet ID based on the information about the video asset described in the PA message, and the video MMTP payload included in the MMTP packet. Is output to the video MMTP payload processing unit 19. For example, when the video asset is composed of the asset 1 and the asset 2, the video MMTP payload related to each asset is output to the video MMTP payload processing unit.

When the video MMTP payload processing unit 19 receives the video MMTP payload related to two or more video assets from the asset separation unit 14, it generates a HEVC elemental stream from each video MMTP payload, and the HEVC elemental stream is used as the HEVCES buffer 20. The decoding time and the presentation time of the access unit included in each video MMTP payload are stored in the HEVCES buffer 20 (step ST19).

The HEVCES decoding unit 21 compares the DTS of the access unit of the asset 1 with the DTS of the access unit of the asset 2 based on the dependency relationship between the asset 1 and the asset 2, for example, in order to encode the HEVC elemental stream before separation. The HEVC elemental stream input separately into different bitstreams can be taken out from the HEVCES buffer 20 at the correct decoding time, and the video signal of the access unit (AU) can be decoded.

As is clear from the above, according to the first embodiment, when the video signals of one or more access units (AU) are time-layer coded, different assets are configured according to the layer level of each access unit. In this case, the MPU time information descriptor is encoded in the unit of MPU that constitutes each asset, and the presentation time and decoding time are calculated in the access unit unit of the coded data included in the MMTP payload included in the MPU time information descriptor. The presentation time information (presentation_time_offset) and the decoding time information (presentation_time_offset) and the decoding time information (presentation_time_offset) are used in the unit of the access unit constituting the MPU according to the value of the flag indicating whether to encode the information (presentation time information or decoding time information). Since the decoding_time_offset) is configured to be encoded, even when a bit stream of time-layer-encoded video is configured and transmitted as different assets according to the layer level of each access unit, the decoding device decodes each access unit. There is an effect of obtaining a coding device and a decoding device capable of reconstructing and decoding a bit stream encoded in a time hierarchy based on the timing.

It should be noted that, within the scope of the present invention, any combination of embodiments can be freely combined, any component of the embodiment can be modified, or any component can be omitted in the embodiment.

1 Voice coding unit, 2 Voice MMTP packet generation unit, 3 HEVC coding unit (video coding means), 4 Video MMTP packet generation unit (video coding means), 5 Control information coding unit (control information coding means) , Time information coding means), 6 Control MMTP payload generation unit (control information coding means), 7 MMTP packet multiplexing unit (multiplexing means), 8 Voice MMTP packet generation unit, 9 Video MMTP packet generation unit, 10 Control MMTP packet generation unit, 12 MMTP packet analysis unit, 13 control MMTP payload processing unit (presentation time calculation means), 14 asset separation unit, 15 audio MMTP payload processing unit, 16 audio ES buffer, 17 audio stream decoding unit, 18 audio data Buffer, 19 video MMTP packet processing unit (video decoding means), 20 HEVCES buffer (video decoding means), 21 HEVCES decoding unit (video decoding means), 22 decoded image buffer (video decoding means).

Claims (1)

A decoding device that decodes coded data of a video signal in an MMT capable of transmitting data in a transmission format different for each of the one or more components constituting one program.
From the coded data, presentation time information indicating the presentation time of the first access unit in the presentation order in one or more GOPs, which is a set of a plurality of access units encoded by the inter-frame predictive coding method, and the access. Control information for decoding the time difference information between the decoding time of the first access unit and the presentation time in the coding order unlike the unit, and the unit representing the presentation time information and the display time information encoded in the unit of the access unit. Decoding means and
Using the presentation time information, the time difference information, and the unit acquired by the control information decoding means, the presentation time and the decoding time of each access unit are calculated, and the video signal included in the coded data is calculated. A decryption device equipped with a decoding means for decoding.

Images