JP7003308B2 - Decoding device - Google Patents

Description

The present invention relates to a coding device and a coding method for encoding a video signal or an audio signal to generate a bit stream, and a decoding device and a decoding method for decoding the coded data multiplexed in the bit stream. be.

In digital broadcasting in Japan, as described in Non-Patent Document 1 below, the video stream and the audio stream, which are the encoded data of the video signal and the audio signal, are MPEG-2 (Moving Picture Experts Group Phase-2). It is multiplexed and transmitted in the transport stream (TS) format, which is the system standard of. At this time, the coding apparatus also multiplexes and transmits the coded data of the metadata related to the video stream and the audio stream together with the video stream and the audio stream.

In addition to the transport stream (TS) in MPEG-2, there is MMT (MPEG Media Transmission) as a new transport method that is being standardized by MPEG, and MMT is one or more that constitutes one program. When transmitting a video component (video stream) and an audio component (audio stream), it is possible to transmit in different transmission modes (for example, broadcasting, communication, etc.) for each component.

Here, HEVC / H. 265 (hereinafter referred to as "HEVC") is a new video coding method standardized by MPEG and ITU (International Telecommunication Union).
In HEVC, time-hierarchical coding (time-wise scalable coding) has been introduced, and NAL (a unit containing coding data necessary for decoding one picture) is a coding unit. Network Operation Layer) Hierarchical level can be specified for each unit.

FIG. 8 is an explanatory diagram showing an example of time-hierarchical coding in HEVC.
In FIG. 8, the Temporal ID is identification information indicating the hierarchy level of each access unit (AU).
When TemporalID = L ₀ , the time hierarchy coding is not performed because there is only a hierarchy of 0.
When TemporalID = L1, the maximum layer is ₁ , and time layer coding is performed.
Similarly, when TemporalID = L2 and L3, the maximum layers are ₂ and ₃ layers, respectively, and time layer coding is performed.

Since the content of time-hierarchical coding is known, detailed description thereof will be omitted, but as a limitation of time-hierarchical coding, an access unit (AU) having a hierarchy level higher than that of the access unit (AU) to be encoded has. ) Cannot be referred to.
In HEVC, time-layer coding with a reference structure having a maximum layer of up to 6 is possible.

FIG. 9 is an explanatory diagram showing an example of a picture structure.
In FIG. 9, the IRAP is an IRAP (Intra Random access point) picture defined by HEVC, and when decoding is started from the middle of a bit stream, the pictures after the IRAP picture are normally displayed in the display order. Guaranteed to be decrypted.
GOP (Group Of Pictures) can decode all the video signals of one or more access units when the video signals of one or more access units (AU) are encoded by the interframe prediction coding method. It is a set of a plurality of possible access units (AUs). That is, it is a set of an IRAP picture which is the first access unit (AU) in the coding order and an access unit (AU) (a picture other than the IRAP picture) following the IRAP picture.
Further, the SOP is a set of the first picture in the coding order and the pictures of the hierarchy level 1 or higher following the picture having the hierarchy level 0.
A bitstream of a sequence is composed of one or more GOPs, and one GOP is composed of one or more SOPs. In the example of FIG. 9, the GOP is composed of the SOP of L ₃ and the SOP of L ₂ .

FIG. 10 is an explanatory diagram showing a coding order and a display order of each picture encoded by the picture structure of FIG.
In the first SOP, each picture following the IRAP picture which is the first access unit (AU) in the coding order is encoded with the picture of the previous SOP as a reference image.
Therefore, in order to normally decode each picture following the IRAP picture, it is necessary to refer to the picture of the previous SOP, so that the picture of the previous SOP needs to be already decoded.
Therefore, when decoding is started from the IRAP picture of the first SOP, the picture of the previous SOP cannot be referred to (the picture of the previous SOP is not decoded), so that the IRAP picture is used. Each subsequent picture cannot be decrypted normally.
In HEVC, a picture that follows an IRAP picture in the coding order and has a faster display order than the IRAP picture is called LP (Reading Picture), and when decoding is started from the IRAP picture, LP normally decodes. Can't.
Since the IRAP picture is encoded by the in-frame coding method, it can be normally decoded even if the previous SOP picture is not decoded.

For example, in digital broadcasting, a plurality of bitstreams are delivered at the same time, and the bitstream to be displayed is switched by switching channels by the user.
Since the IRAP picture is periodically inserted into the plurality of bitstreams, the display image can be switched by starting decoding from any of the IRAP pictures when the user channel is switched.

STD-B32 (Standard for digital broadcasting established by ARIB (Association of Radio Industries and Businesses))

Since the conventional coding device is configured as described above, when time-layer coding is performed, the larger the maximum layer, the larger the number of pictures constituting the SOP. Therefore, when the decoding device starts decoding from the IRAP picture, the number of LPs that cannot be normally decoded increases. Therefore, if the bit stream to be displayed is switched by the channel switching by the user, the bit stream to be displayed is switched after the switching. It takes a lot of time to display a picture by a bitstream, and there is a problem that the time when no picture is displayed becomes long.
That is, when the bit stream to be displayed is switched by the channel switching by the user, what is the period from the display time of the first LP (the picture of B25 in the example of FIG. 10) to the display time of the IRAP picture. Since the pictures are not displayed, there is a problem that the time when no pictures are displayed becomes longer as the number of pictures constituting the SOP increases.

The present invention has been made to solve the above-mentioned problems, and it is possible to realize seamless channel switching by eliminating the time when no picture is displayed when the channel is switched by the user. It is an object of the present invention to obtain an apparatus, a decoding apparatus, a coding method and a decoding method.

The decoding device according to the present invention is a decoding device that decodes coded data of a video signal transmitted by a transmission method defined by MMT (MPEG Media Transport), and is a PA message that is control information included in the coded data. Therefore, in the GOP, which is a set of a plurality of access units encoded by the inter-frame predictive coding method, the presentation time information indicating the presentation time of the first access unit in the presentation order and the presentation of the first access unit in the presentation order. The control information decoding means for obtaining the information for calculating the presentation time of the first access unit in the coding order from the presentation time information indicating the time, and the control information decoding means acquired by the control information decoding means for the first access unit in the presentation order. It is provided with a decoding means for decoding a video signal included in the coded data by using the presentation time information and information for calculating the presentation time of the first access unit in the coding order.

According to the present invention, it is a decoding device that decodes coded data of a video signal transmitted by a transmission method defined by MMT (MPEG Media Transport), and from a PA message that is control information included in the coded data. In GOP, which is a set of multiple access units encoded by the inter-frame predictive coding method, the presentation time information indicating the presentation time of the first access unit in the presentation order and the presentation time of the first access unit in the presentation order are displayed. The control information decoding means for obtaining the information for calculating the presentation time of the first access unit in the coding order from the presentation time information shown, and the presentation time of the first access unit in the presentation order acquired by the control information decoding means. Since it is provided with a decoding means for decoding the video signal included in the coded data by using the information and the information for calculating the presentation time of the first access unit in the coding order, the channel can be switched by the user. There is an effect that seamless channel switching can be realized by eliminating the time when no picture is displayed at the time.

It is a block diagram which shows the coding apparatus by Embodiment 1 of this invention. It is a flowchart which shows the processing content (coding method) of the coding apparatus by Embodiment 1 of this invention. It is a block diagram which shows the decoding apparatus by Embodiment 1 of this invention. It is a flowchart which shows the processing content (decoding method) of the decoding apparatus by Embodiment 1 of this invention. It is explanatory drawing which shows the outline of the coded data at the time of transmitting a bit stream by MMT. It is explanatory drawing which shows the structural example of MPU. It is explanatory drawing which shows the HEVC picture structure descriptor. It is explanatory drawing which shows the time layer coding example in HEVC. It is explanatory drawing which shows an example of a picture structure. It is explanatory drawing which shows the coding order and the presentation order of each picture which is coded by the picture structure of FIG.

Embodiment 1.
FIG. 1 is a block diagram showing a coding device according to the first embodiment of the present invention.
In FIG. 1, when a digital voice signal is given, the voice coding unit 1 encodes the voice signal for each voice access unit (AU) by a method such as MPEG-4 audio, and the voice signal is encoded. A voice stream which is the coded data of the above is generated, and a process of encoding the metadata related to the voice stream is performed.
The voice MMTP payload generation unit 2 performs a process of generating a voice MMTP payload composed of metadata encoded by the voice coding unit 1 and voice signal encoded data for each access unit (AU).

When a digital video signal is given, the HEVC coding unit 3 encodes the video signal in the video access unit (AU) unit by the HEVC method, and generates a video stream which is encoded data of the video signal. At the same time, a process of encoding the metadata related to the video stream is performed.
The video MMTP payload generation unit 4 performs a process of generating a video MMTP payload composed of metadata encoded by the HEVC coding unit 3 and coded data of a video signal for each access unit (AU). The video coding means is composed of the HEVC coding unit 3 and the video MMTP payload generation unit 4.
Here, as metadata related to the audio stream and the video stream, for example, time information indicating DTS (decoding time), PTS (presentation time), etc. of each access unit (AU) can be described.

The control information coding unit 5 encodes control information called a PA message defined by the MMT as control information regarding the voice stream generated by the voice coding unit 1 and the video stream generated by the HEVC coding unit 3. Perform the processing to be performed.
The PA message describes information about one or more video components (video streams) and audio components (audio streams) that make up one program (referred to as a package in MMT). In MMT, video and audio components are called assets.
Specifically, the asset ID that identifies the asset, the asset type that identifies the asset type (type such as a video stream in HEVC format or the audio stream in MPEG-4 audio format), and the MPU (Media Processing Unit) of each asset. MPU time stamp descriptor (presentation time information) indicating the presentation time (display time) of the first access unit (AU) in the presentation order (display order) among the configured access units (AU), HEVC picture structure. A descriptor, a packet ID indicating an MMTP packet storing encoded data and metadata of each asset, and the like are included in the PA message for the number of assets constituting the package.
The MPU is composed of one or more access units (AUs), and is a unit capable of performing video and audio decoding processing by the MPU alone. Further, when the video signals of one or more access units (AU) are encoded by the interframe prediction coding method, the MPU decodes all the video signals of the one or more access units (AU). It is the same unit as GOP, which is a set of multiple access units (AU) that can be used.

In the HEVC picture structure descriptor, the number of access units (AU) (number of LPs) whose presentation order is earlier than that of the first access unit (AU) in the coding order among the access units (AU) constituting the MPU. ), Picture type information (rap_type) indicating the coding method of the NAL unit (null unit) constituting the first access unit (AU) in the coding order, and LP. Picture type information (nal_unit_type_of_reading_picture) indicating the coding method of the NAL unit and the like are described.

The control MMTP payload generation unit 6 performs a process of generating a control MMTP payload composed of coded data of control information encoded by the control information coding unit 5.
The control information coding unit 5 and the control MMTP payload generation unit 6 constitute a control information coding unit.

The MMTP packet generation unit 7 includes a voice MMTP payload generated by the voice MMTP payload generation unit 2, a video MMTP payload generated by the video MMTP payload generation unit 4, and a control MMTP payload generated by the control MMTP payload generation unit 6. Is multiplexed to perform a process of generating an MMTP packet constituting a bit stream. When the MMTP packet is generated, a predetermined MMTP header is added, and the MMTP header includes a packet ID assigned according to the type of coded data included in the MMTP payload. The MMTP packet generation unit 7 constitutes a multiplexing means.

In the example of FIG. 1, a voice coding unit 1, a voice MMTP payload generation unit 2, a HEVC coding unit 3, a video MMTP payload generation unit 4, a control information coding unit 5, and a control MMTP payload, which are components of the coding device, are used. It is assumed that each of the generation unit 6 and the MMTP packet generation unit 7 is composed of dedicated hardware (for example, a semiconductor integrated circuit on which a CPU is mounted, a one-chip microcomputer, or the like). The computer may be composed of a computer.
When the coding device is configured by a computer, the voice coding unit 1, the voice MMTP payload generation unit 2, the HEVC coding unit 3, the video MMTP payload generation unit 4, the control information coding unit 5, the control MMTP payload generation unit 6, and so on. A program describing the processing contents of the MMTP packet generation unit 7 may be stored in the memory of the computer, and the CPU of the computer may execute the program stored in the memory.
FIG. 2 is a flowchart showing the processing content (coding method) of the coding apparatus according to the first embodiment of the present invention.

FIG. 3 is a block diagram showing a decoding device according to the first embodiment of the present invention.
In FIG. 3, the stream selection unit 11 is a bit stream (a bit stream composed of MMTP packets) output from a plurality of coding devices (the coding device of FIG. 1 or the coding device corresponding to the coding device of FIG. 1). ), A bitstream to be presented is selected, and a process of outputting the bitstream to the MMTP packet analysis unit 12 is performed.
Further, when a command for switching the bitstream to be presented is given, the stream selection unit 11 selects the bitstream after switching from the bitstreams output from the plurality of coding devices, and selects the bitstream. In addition to outputting to the MMTP packet analysis unit 12, the bit stream before switching is also continuously output to the MMTP packet analysis unit 12 until the presentation time calculated by the control MMTP payload processing unit 13 is reached. The stream selection unit 11 constitutes a bitstream selection means.

The MMTP packet analysis unit 12 analyzes the MMTP header of the MMTP packet constituting the bit stream output from the stream selection unit 11, acquires the packet ID included in the MMTP header, and the packet ID is MMTP. If the coded data contained in the payload indicates that it is control information (PA message, HEVC picture structure descriptor), the control MMTP payload, which is the MMTP packet contained in the MMTP packet, is controlled by the control MMTP payload. Output to the processing unit 13. On the other hand, if the packet ID indicates that the coded data included in the MMTP payload is an audio signal or a video signal, a process of outputting the MMTP packet to the asset separation unit 14 is performed.

The control MMTP payload processing unit 13 performs decoding processing of the coded data contained in the control MMTP payload output from the MMTP packet analysis unit 12, and performs the decryption processing of the PA message which is the control information and the HEVC included in the PA message. Decrypt the picture structure descriptor.
Further, the control MMTP payload processing unit 13 has the presentation time of the first access unit (AU) in the presentation order indicated by the MPU time stamp descriptor described in the PA message, and the number information described in the HEVC picture structure descriptor. From the number of access units (AU) (number of LPs) whose presentation order is earlier than the first access unit (AU) in the coding order indicated by (num_of_leading_picture), the presentation time of the first access unit (AU) in the coding order. Is performed. The first access unit (AU) in the coding order is the IRAP picture of the first SOP. The control MMTP payload processing unit 13 constitutes a presentation time calculation means.

The asset separation unit 14 is included in the MMTP packet output from the MMTP packet analysis unit 12 with reference to the asset ID, asset type and packet ID described in the PA message decoded by the control MMTP payload processing unit 13. Identify whether the MMTP payload is a voice MMTP payload or a video MMTP payload, and if it is a voice MMTP payload, extract the voice MMTP payload contained in the MMTP packet and use the voice MMTP payload as voice. Output to the MMTP payload processing unit 15, and if it is a video MMTP payload, a process of extracting the video MMTP payload contained in the MMTP packet and outputting the video MMTP payload to the video MMTP payload processing unit 19 is performed. ..

The audio MMTP payload processing unit 15 reconstructs the MFU (Media Fragment Unit) or MPU of the audio stream from the audio MMTP payload output from the asset separation unit 14, so that the audio stream decoding unit 17 in the subsequent stage can decode the audio stream. A process of generating a voice elemental stream (voice ES) and storing the voice ES in the voice ES buffer 16 is performed. An MFU is a unit smaller than an MPU, and one access unit (AU) or one NAL unit can be defined as one MFU.
Further, the audio MMTP payload processing unit 15 extracts the metadata related to the audio stream included in the audio MMTP payload output from the asset separation unit 14, and performs a process of storing the metadata in the audio ES buffer 16.
The voice ES buffer 16 is a memory for temporarily storing voice ES and metadata.

The voice stream decoding unit 17 extracts metadata from the voice ES buffer 16 and time information described in the metadata (information indicating DTS (decoding time) and PTS (presentation time) of each access unit (AU)). Is executed.
Further, when the DTS (decoding time) of each access unit (AU) is reached, the audio stream decoding unit 17 takes out the audio ES from the audio ES buffer 16, decodes the audio signal of the access unit (AU), and decodes the audio signal. A process of storing the voice signal and the PTS (presentation time) in the voice data buffer 18 is performed.
The audio data buffer 18 is a memory that temporarily stores the audio signal decoded by the audio stream decoding unit 17 and the PTS (presentation time).

The video MMTP payload processing unit 19 reconstructs the MFU or MPU of the video stream from the video MMTP payload output from the asset separation unit 14, and the HEVC elementary stream (HEVC) in a format that can be decoded by the HEVCES decoding unit 21 in the subsequent stage. ES) is generated, and the process of storing the HEVC elementary stream in the HEVCES buffer 20 is performed.
Further, the video MMTP payload processing unit 19 extracts the metadata related to the video stream included in the video MMTP payload output from the asset separation unit 14, and performs a process of storing the metadata in the HEVCES buffer 20.
The HEVCES buffer 20 is a memory for temporarily storing the HEVC elemental stream and metadata.

The HEVCES decoding unit 21 extracts metadata from the HEVCES buffer 20 and decodes the time information (information indicating the DTS (decoding time) and PTS (presentation time) of each access unit (AU)) described in the metadata. Perform the processing.
Further, when the DTS (decoding time) of each access unit (AU) is reached, the HEVCES decoding unit 21 takes out the HEVC elemental stream from the HEVCES buffer 20, decodes the video signal of the access unit (AU), and decodes the video signal. A process of storing the decoded image and the PTS (presentation time), which are video signals, in the decoded image buffer 22 is performed.
The decoded image buffer 22 is a memory that temporarily stores the decoded image and PTS (presentation time) of each access unit (AU) decoded by the HEVCES decoding unit 21.
The video decoding means is composed of the video MMTP payload processing unit 19, the HEVCES buffer 20, the HEVCES decoding unit 21, and the decoded image buffer 22.

In the example of FIG. 3, a stream selection unit 11, an MMTP packet analysis unit 12, a control MMTP payload processing unit 13, an asset separation unit 14, a voice MMTP payload processing unit 15, a voice ES buffer 16, and a voice stream, which are components of the decoding device, are used. Decoding unit 17, audio data buffer 18, video MMTP payload processing unit 19, HEVCES buffer 20, HEVCES decoding unit 21, and decoded image buffer 22 each have dedicated hardware (other than the buffer, for example, a semiconductor on which a CPU is mounted). It is assumed that it is composed of an integrated circuit or a one-chip microcomputer, but the decoding device may be composed of a computer.
When the decoding device is configured by a computer, the audio ES buffer 16, the audio data buffer 18, the HEVCES buffer 20, and the decoded image buffer 22 are configured on the internal memory or the external memory of the computer, and the stream selection unit 11 and the MMTP packet analysis unit are configured. 12. A computer computer program that describes the processing contents of the control MMTP payload processing unit 13, the asset separation unit 14, the audio MMTP payload processing unit 15, the audio stream decoding unit 17, the video MMTP payload processing unit 19, and the HEVCES decoding unit 21. It may be stored in the memory, and the CPU of the computer may execute the program stored in the memory.
FIG. 4 is a flowchart showing the processing content (decoding method) of the decoding apparatus according to the first embodiment of the present invention.

Next, the operation will be described.
The processing contents of the first coding device will be described.
When a digital voice signal is given, the voice coding unit 1 encodes the voice signal for each voice access unit (AU) by a method such as MPEG-4 audio, and encodes the voice signal. A voice stream which is data is generated, and metadata about the voice stream is encoded (step ST1 in FIG. 2).
When a digital video signal is given, the HEVC coding unit 3 encodes the video signal in the video access unit (AU) unit by the HEVC method, and generates a video stream which is the coded data of the video signal. At the same time, the metadata related to the video stream is encoded (step ST2).

Here, FIG. 5 is an explanatory diagram showing an outline of coded data when a bit stream is transmitted by MMT.
In FIG. 5, the access unit (AU) is a unit including coding data necessary for decoding one picture in the case of video, and is composed of one or more samples serving as a coding unit in the case of audio. It is a frame to be done.
The NAL unit is a coding unit of HEVC, and one access unit (AU) is composed of one or more NAL units.
The MPU is composed of one or more access units, and is a unit capable of performing video and audio decoding processing by the MPU alone. Further, when the video signals of one or more access units (AU) are encoded by the interframe prediction coding method, the MPU decodes all the video signals of the one or more access units (AU). It is the same unit as GOP, which is a set of multiple access units (AU) that can be used.
An MFU is a unit smaller than an MPU, and one access unit (AU) or one NAL unit can be defined as one MFU.

FIG. 6 is an explanatory diagram showing a configuration example of the MPU.
In FIG. 6, the MPU metadata describes metadata related to the MPU, such as time information indicating the DTS (decoding time) and PTS (presentation time) of each access unit (AU) included in the MPU. Can be described.
The movie fragment metadata (MF meta) describes the metadata attached to the coded data (sample data) of one access unit (AU). For example, when the coded data of the access unit (AU) is stored in a file format, the address where the coded data is stored, the data length of the coded data, and the access unit (AU) are stored for each access unit (AU). ) Contains information about the presentation time.
MPU metadata, movie fragment metadata, MFU and MMT control information are transmitted in MMTP packets. An MMTP packet consists of an MMTP header and an MMTP payload.

When the voice MMTP payload generation unit 2 receives the coded data of the metadata (MPU metadata, MF meta, etc.) and the coded data of the voice signal of the access unit (AU) unit from the voice coding unit 1, the MPU is received. A voice MMTP payload including the coded data of the MPU metadata of the unit, the coded data of the MF meta of the access unit (AU) unit, and the coded data (sample data) of the voice signal is generated (step ST3).
When the video MMTP payload generation unit 4 receives the coded data of the metadata (MPU metadata, MF meta, etc.) and the coded data of the video signal of the access unit (AU) unit from the HEVC coding unit 3, the MPU A video MMTP payload including the coded data of the MPU metadata of the unit, the coded data of the MF meta of the access unit (AU) unit, and the coded data (sample data) of the video signal is generated (step ST4).

The control information coding unit 5 encodes control information regarding the audio stream generated by the audio coding unit 1 and the video stream generated by the HEVC coding unit 3 (step ST5).
As control information related to the audio stream and the video stream, for example, a PA message defined by MMT, a HEVC picture structure descriptor, or the like is encoded.
As described above, the PA message describes information about one or more video components (video stream) and audio components (audio stream) constituting one program (referred to as a package in MMT).
That is, in the PA message, the asset ID that identifies the assets (video stream, audio stream) generated by the audio coding unit 1 and the HEVC coding unit 3, the asset type that identifies the asset type, and the MPU of each asset are included. Among the configured access units (AU), the MPU time stamp descriptor indicating the presentation time of the first access unit (AU) in the presentation order, and the MMTP packet storing the coded data and metadata of each asset. A packet ID or the like indicating the above is described.

FIG. 7 is an explanatory diagram showing a HEVC picture structure descriptor.
As shown in FIG. 7, the HEVC picture structure descriptor includes an access unit (AU) having an earlier presentation order than the first access unit (AU) in the coding order among the access units (AU) constituting the MPU. ) Number information (num_of_reading_texture) indicating the number (number of LPs) is described.
Further, the picture type information (rap_type) indicating the coding method of the NAL unit constituting the first access unit (AU) in the coding order and the picture type indicating the coding method of the NAL unit constituting the LP. Information (nal_unit_type_of_reading_code) and the like are described.

When the control MMTP payload generation unit 6 receives the coded data of the control information from the control information coding unit 5, the control MMTP payload generation unit 6 generates a control MMTP payload composed of the coded data of the control information (step ST6).
The MMTP packet generation unit 7 includes an audio MMTP payload generated by the audio MMTP payload generation unit 2, a video MMTP payload generated by the video MMTP payload generation unit 4, and a control MMTP payload generated by the control MMTP payload generation unit 6. And are multiplexed to generate an MMTP packet constituting a bit stream (step ST7).
When the MMTP packet is generated, a predetermined MMTP header is added, and the MMTP header includes a packet ID assigned according to the type of coded data included in the MMTP payload.

Next, the processing contents of the decoding device will be described.
The stream selection unit 11 is provided with a bit stream (a bit stream composed of MMTP packets) output from a plurality of coding devices (the coding device of FIG. 1 or the coding device corresponding to the coding device of FIG. 1). Be done.
The stream selection unit 11 selects a bitstream (a bitstream to be presented) of a channel designated by the user from a plurality of bitstreams, and outputs the bitstream to the MMTP packet analysis unit 12.

When the MMTP packet analysis unit 12 receives a bitstream from the stream selection unit 11, it analyzes the MMTP header of the MMTP packet constituting the bitstream and acquires the packet ID included in the MMTP header.
If the packet ID indicates that the coded data included in the MMTP payload is control information (PA message), the MMTP packet analysis unit 12 controls the MMTP payload included in the MMTP packet. The MMTP payload is output to the control MMTP payload processing unit 13.
On the other hand, if the packet ID indicates that the coded data included in the MMTP payload is an audio signal or a video signal, the MMTP packet is output to the asset separation unit 14.

When the control MMTP payload processing unit 13 receives the control MMTP payload from the MMTP packet analysis unit 12, it executes decoding processing of the coded data included in the control MMTP payload, and performs PA message and PA message which are control information. Decrypt the HEVC picture structure descriptor contained in.

When the control MMTP payload processing unit 13 decodes the PA message, the asset separation unit 14 refers to the asset ID, asset type, and packet ID described in the PA message, and outputs the MMTP output from the MMTP packet analysis unit 12. Identify whether the MMTP payload contained in the packet is an audio MMTP payload or a video MMTP payload.
If the MMTP payload contained in the MMTP packet output from the MMTP packet analysis unit 12 is a voice MMTP payload, the asset separation unit 14 extracts the voice MMTP payload contained in the MMTP packet and the voice MMTP payload. The MMTP payload is output to the audio MMTP payload processing unit 15.
If the MMTP payload contained in the MMTP packet output from the MMTP packet analysis unit 12 is a video MMTP payload, the asset separation unit 14 extracts the video MMTP payload contained in the MMTP packet and captures the video. The MMTP payload is output to the video MMTP payload processing unit 19.

When the audio MMTP payload processing unit 15 receives the audio MMTP payload from the asset separation unit 14, the audio MMTP payload is reconstructed from the audio MMTP payload to a format that can be decoded by the audio stream decoding unit 17 in the subsequent stage. A voice elemental stream (voice ES) is generated, and the voice ES is stored in the voice ES buffer 16.
Since the process itself of generating the voice ES from the voice MMTP payload is a known technique, detailed description thereof will be omitted.
Further, the audio MMTP payload processing unit 15 extracts the metadata related to the audio stream included in the audio MMTP payload output from the asset separation unit 14, and stores the metadata in the audio ES buffer 16.

The voice stream decoding unit 17 extracts metadata from the voice ES buffer 16 and indicates time information (information indicating the DTS (decoding time) and PTS (presentation time) of each access unit (AU)) described in the metadata. ) Is decrypted.
The audio stream decoding unit 17 refers to the decoded DTS, grasps the decoding time of each access unit (AU), and takes out the audio ES from the audio ES buffer 16 at the decoding time of each access unit (AU). , The voice signal of the access unit (AU) is decoded, and the decoded voice signal and the PTS (presentation time) are stored in the voice data buffer 18.
As a result, the external playback device (not shown) can reproduce the voice signal at the presentation time by taking out the voice signal and the PTS (presentation time) stored in the voice data buffer 18.

When the video MMTP payload processing unit 19 receives the video MMTP payload from the asset separation unit 14, it reconstructs the MFU or MPU of the video stream from the video MMTP payload, so that it can be decoded by the HEVCES decoding unit 21 in the subsequent stage. A HEVC elemental stream (HEVC ES) is generated, and the HEVC elemental stream is stored in the HEVCES buffer 20.
Since the process itself for generating the HEVC elemental stream from the video MMTP payload is a known technique, detailed description thereof will be omitted.
Further, the video MMTP payload processing unit 19 extracts the metadata related to the video stream included in the video MMTP payload output from the asset separation unit 14, and stores the metadata in the HEVCES buffer 20.

The HEVCES decoding unit 21 takes out the metadata from the HEVCES buffer 20 and obtains the time information (information indicating the DTS (decoding time) and PTS (presentation time) of each access unit (AU)) described in the metadata. Decrypt.
The HEVCES decoding unit 21 grasps the decoding time of each access unit (AU) with reference to the decoded DTS, and when the decoding time of each access unit (AU) comes, the HEVC elemental stream is taken out from the HEVCES buffer 20. The video signal of the access unit (AU) is decoded, and the decoded image and the PTS (presentation time), which are the decoded video signals, are stored in the decoded image buffer 22.
As a result, if an external playback device (not shown) takes out the decoded image and the PTS (presentation time) stored in the decoded image buffer 22, the decoded image can be reproduced at the presented time.

When an external playback device (not shown) is playing a decoded image and an audio signal and the user performs an operation to switch channels using a remote controller or the like, a command to switch the video stream to be presented (this switching command). Contains information indicating the channel after switching) is given to the stream selection unit 11.
When the stream selection unit 11 receives a channel switching command from the outside (in the case of step ST11: Yes in FIG. 4), the stream selection unit 11 selects the bit stream of the switched channel indicated by the switching command from the plurality of bit streams. Then, the bit stream is output to the MMTP packet analysis unit 12 (step ST12).
Further, the stream selection unit 11 outputs a bitstream from the control MMTP payload processing unit 13 in order to realize seamless channel switching by eliminating the time when no decoded image is displayed when the channel is switched by the user. Until the stop command is received (step ST13: No), the bitstream of the channel before switching is also continuously output to the MMTP packet analysis unit 12 (step ST14). As will be described later, the bitstream output stop command is output when the current time reaches the presentation time of the first access unit (AU) in the coding order.
When the stream selection unit 11 receives a bitstream output stop command from the control MMTP payload processing unit 13, the stream selection unit 11 stops the output of the bitstream of the channel before switching, and MMTP packet analysis unit only the bitstream of the channel after switching. Output to 12.

When the MMTP packet analysis unit 12 receives the bitstream of the channel after switching from the stream selection unit 11, the MMTP packet analysis unit 12 inputs the control MMTP packet included in the MMTP packet constituting the bitstream as before the channel switching. It is output to the control MMTP payload processing unit 13, and the audio MMTP payload or the video MMTP payload included in the MMTP packet constituting the bitstream is output to the asset separation unit 14.
Further, when the MMTP packet analysis unit 12 receives the bitstream of the channel before switching from the stream selection unit 11, the bitstream of the channel before switching is configured by processing and parallel processing for the bitstream of the channel after switching. The control MMTP payload included in the MMTP packet is output to the control MMTP payload processing unit 13, and the audio MMTP payload or video MMTP payload included in the MMTP packet constituting the bitstream is output to the asset separation unit 14. Output to.

When the control MMTP payload processing unit 13 receives the control MMTP payload related to the bit stream of the channel after switching from the MMTP packet analysis unit 12, the coded data included in the control MMTP payload is the same as before the channel switching. Decoding process is performed to decode the PA message which is the control information and the HEVC picture structure descriptor included in the PA message (step ST15).
When the control MMTP payload processing unit 13 decodes the PA message and the HEVC picture structure descriptor included in the PA message, the first access unit (AU) in the presentation order indicated by the MPU time stamp descriptor described in the PA message. And the number of access units (AU) whose presentation order is earlier than the first access unit (AU) in the coding order indicated by the number information (num_of_leading_picture) described in the HEVC picture structure descriptor (number of LPs). From, the presentation time of the first access unit (AU) in the coding order is calculated (step ST16).

In the example of FIG. 10, the first access unit (AU) in the coding order is IRAP32, and the first access unit (AU) in the presentation order is B25. Further, the number of access units (AU) (the number of LPs) whose presentation order is earlier than that of IRAP32 is 7.
Therefore, if the presentation time of B25, which is the first access unit (AU) in the presentation order, is, for example, 18:00:00 and the frame rate is 120 frames / second, the presentation time of IRAP32 is B25. It will be 58 msec (= 7/120) after the presentation time (18:00:00).
When the current time reaches the presentation time of the first access unit (AU) in the coding order (in the case of step ST17: Yes), the control MMTP payload processing unit 13 selects a stream of the bitstream output stop command of the channel before switching. Output to unit 11 (step ST18).

When the asset separation unit 14 receives the MMTP packet related to the channel after switching from the MMTP packet analysis unit 12, the voice MMTP payload contained in the MMTP packet is processed by the voice MMTP payload processing unit 15 as before the channel is switched. And outputs the video MMTP payload contained in the MMTP packet to the video MMTP payload processing unit 19.
Further, when the asset separation unit 14 receives the MMTP packet related to the channel before switching from the MMTP packet analysis unit 12, the MMTP packet related to the channel before switching is processed in parallel with the processing for the bit stream of the channel after switching. The included audio MMTP payload is output to the audio MMTP payload processing unit 15, and the video MMTP payload contained in the MMTP packet is output to the video MMTP payload processing unit 19.

When the video MMTP payload processing unit 19 receives the video MMTP payload related to the channel before switching from the asset separation unit 14 (before the current time becomes the presentation time of the first access unit (AU) in the presentation order), the channel switching is performed. As before, a HEVC elemental stream is generated from the video MMTP payload, the HEVC elemental stream is stored in the HEVCES buffer 20, and metadata about the video stream contained in the video MMTP payload is extracted. , The metadata is stored in the HEVCES buffer 20 (step ST19).
Further, when the video MMTP payload processing unit 19 receives the video MMTP payload related to the channel after switching from the asset separation unit 14, the video related to the channel after switching is processed in parallel with the processing for the bit stream of the channel before switching. A HEVC elemental stream is generated from the MMTP payload, the HEVC elemental stream is stored in the HEVCES buffer 20, metadata about the video stream included in the video MMTP payload is extracted, and the metadata is stored in the HEVCES buffer. Store in 20 (step ST20).

The HEVCES decoding unit 21 takes out the metadata from the HEVCES buffer 20 and describes the time information (DTS (decoding time) and PTS (decoding time) of each access unit (AU)) in the metadata as before the channel is switched. The information indicating the presentation time) is decoded.
As a result, the HEVCES decoding unit 21 grasps the decoding time of each access unit (AU) with reference to the decoded DTS, but the access unit that the IRAP can first decode the channel after switching. (AU) (in the GOP configuration of FIG. 10, the access unit (AU) of IRAP32), and the video signal of the LP access unit (AU) whose presentation order is earlier than that of IRAP cannot be decoded. Therefore, for the channel after switching, the video signal of any access unit (AU) cannot be decoded and displayed until the presentation time of IRAP is reached.

Therefore, the HEVCES decoding unit 21 decodes the video signal of the access unit (AU) related to the channel before switching until the presentation time of the IRAP related to the channel after switching is reached, and is the decoded video signal. The decoded image and PTS (presentation time) are stored in the decoded image buffer 22.
As a result, the external playback device (not shown) can reproduce the decoded image related to the channel before switching until the presentation time of IRAP related to the channel after switching is reached.
Therefore, it is possible to realize seamless channel switching by eliminating the time when no decoded image is displayed when the channel is switched by the user.

When the voice MMTP payload processing unit 15 receives the voice MMTP payload related to the channel before switching from the asset separation unit 14 (before the current time becomes the presentation time of the first access unit (AU) in the presentation order), the channel switching is performed. As before, the audio ES is generated from the audio MMTP payload, the audio ES is stored in the audio ES buffer 16, and the metadata about the audio stream contained in the audio MMTP payload is extracted, and the metadata is obtained. The data is stored in the voice ES buffer 16.
Further, when the voice MMTP payload processing unit 15 receives the voice MMTP payload related to the channel after switching from the asset separation unit 14, the voice related to the channel after switching is processed in parallel with the processing for the bit stream of the channel before switching. A voice ES is generated from the MMTP payload, the voice ES is stored in the voice ES buffer 16, metadata about a voice stream included in the voice MMTP payload is extracted, and the metadata is stored in the voice ES buffer 16. Store.

The voice stream decoding unit 17 takes out the metadata from the voice ES buffer 16 and describes the time information (DTS (decoding time) of each access unit (AU)) and the time information described in the metadata as before the channel is switched. Information indicating PTS (presentation time) is decoded.
The audio stream decoding unit 17 refers to the decoded DTS, grasps the decoding time of each access unit (AU), and takes out the audio ES from the audio ES buffer 16 at the decoding time of each access unit (AU). , The voice signal of the access unit (AU) is decoded, and the decoded voice signal and the PTS (presentation time) are stored in the voice data buffer 18.
As a result, the external playback device (not shown) can reproduce the voice signal at the presentation time by taking out the voice signal and the PTS (presentation time) stored in the voice data buffer 18.

Here, like the HEVCES decoding unit 21, the audio stream decoding unit 17 outputs the audio signal of the access unit (AU) related to the channel before switching until the presentation time of the IRAP related to the channel after switching is reached. It is assumed that the audio signal is decoded and the decoded audio signal and PTS (presentation time) are stored in the audio data buffer 18, but the audio signal is encoded by an inter-frame prediction coding method like a video signal. Therefore, the audio signal of the LP access unit (AU) whose presentation order is earlier than that of IRAP can also be decoded. Therefore, even until the presentation time of the IRAP related to the channel after switching is reached, the access related to the channel after switching is performed without decoding the audio signal of the access unit (AU) related to the channel before switching. The audio signal of the unit (AU) may be decoded.

In the decoding device, the processes of steps ST11 to ST20 are repeatedly executed until the decoding process is completed (step ST21).

As is clear from the above, according to the first embodiment, when the video signals of one or more access units (AU) are encoded by the inter-frame predictive coding method, one or more access units (AU). For each GOP, which is a set of a plurality of access units (AUs) capable of decoding all of the video signals of the above, the presentation time information indicating the presentation time of the first access unit (AU) in the presentation order and the coding order. Since the control information including the number information indicating the number of access units (AU) whose presentation order is earlier than that of the first access unit (AU) is encoded in, the channel is switched by the user on the decoding side. Occasionally, there is an effect of obtaining a coding device capable of realizing seamless channel switching by eliminating the time when no decoded image is displayed.

Further, according to the first embodiment, when a command for switching the bit stream to be presented is given, the stream selection unit 11 selects the bit after switching from the bit streams output from the plurality of coding devices. A stream is selected, the bitstream is output to the MMTP packet analysis unit 12, and the presentation time calculated by the control MMTP payload processing unit 13 is reached (until the bitstream output stop command is received), before switching. The bitstream is also continuously output to the MMTP packet analysis unit 12, and the HEVCES decoding unit 21 decodes the video signal for each access unit from the coded data of the video signal multiplexed in the switched bitstream, and at the same time, controls the MMTP. Until the presentation time calculated by the payload processing unit 13, the user is configured to decode the video signal for each access unit from the coded data of the video signal multiplexed in the bitstream before switching. This has the effect of obtaining a decoding device that can realize seamless channel switching by eliminating the time when no decoded image is displayed when the channel is switched.

Embodiment 2.
In the first embodiment, the HEVCES decoding unit 21 decodes the video signal of the access unit (AU) related to the channel before switching until the presentation time of the IRAP related to the channel after switching is reached, and decodes the video signal. Although the decoded image and the PTS (presentation time) which are the video signals are stored in the decoded image buffer 22, even if it is the LP access unit (AU) related to the channel after switching, the access unit (AU) is shown. ), Depending on the coding method of the NAL unit, it may be possible to decode the IRAP even before decoding.
Whether or not the access unit (AU) of the LP can be decoded even before the decoding of the IRAP can be determined by referring to the picture type information (nal_unit_type_of_leading_picture) described in the HEVC picture structure descriptor to configure the LP. Since the coding method of the NAL unit being used is known, it can be determined. For example, if the coding method of the NAL unit constituting the LP is an in-frame coding method, even if the video signal of the access unit (AU) of the previous SOP is not decoded, it can be decoded. It is possible.

The HEVCES decoding unit 21 decodes the video signal of the access unit (AU) if it is possible to decode the LP access unit (AU) related to the switched channel even before the IRAP decoding. Then, the decoded image and the PTS (presentation time), which are the decoded video signals, are stored in the decoded image buffer 22.
For example, in the GOP configuration of FIG. 10, the LPs B25, B25, B27, and B28 cannot be decoded, but if the B29, B30, and B31 can be decoded, the video signals of the B29, B30, and B31 are decoded. The decoded image and PTS (presentation time), which are the decoded video signals, are stored in the decoded image buffer 22.
As a result, the external playback device (not shown) decodes the channel before switching at the presentation times of B25, B25, B27, and B28 until the presentation time of IRAP related to the channel after switching is reached. The image is reproduced, and at the presentation times of B29, B30, and B31, the decoded image related to the switched channel can be reproduced.

It should be noted that, within the scope of the present invention, any combination of embodiments can be freely combined, any component of each embodiment can be modified, or any component can be omitted in each embodiment. ..

1 Voice coding unit, 2 Voice MMTP payload generation unit, 3 HEVC coding unit (video coding means), 4 Video MMTP payload generation unit (video coding means), 5 Control information coding unit (control information coding means) ), 6 Control MMTP payload generation unit (control information coding means), 7 MMTP packet generation unit (multiplexing means), 11 stream selection unit (bit stream selection means), 12 MMTP packet analysis unit, 13 control MMTP payload processing unit. (Presentation time calculation means), 14 Asset separation unit, 15 Audio MMTP payload processing unit, 16 Audio ES buffer, 17 Audio stream decoding unit, 18 Audio data buffer, 19 Video MMTP payload processing unit (Video decoding means), 20 HEVCES buffer (Video decoding means), 21 HEVCES decoding unit (video decoding means), 22 decoded image buffer (video decoding means).

Claims (1)

It is a decoding device that decodes the coded data of the video signal transmitted by the transmission method specified by MMT (MPEG Media Transport).
From the PA message which is the control information included in the coded data, the presentation indicating the presentation time of the first access unit in the presentation order in the GOP which is a set of a plurality of access units encoded by the inter-frame predictive coding method. A control information decoding means for obtaining time information and information for calculating the presentation time of the first access unit in the coding order from the presentation time information indicating the presentation time of the first access unit in the presentation order.
The coded data is obtained by using the presentation time information of the first access unit in the presentation order and the information for calculating the presentation time of the first access unit in the coding order acquired by the control information decoding means. Decoding means for decoding the video signal included in the
Decoding device equipped with.

Images