JP6257448B2 - Encoding device, decoding device, encoding method, and decoding method - Google Patents

Description

  The present invention relates to an encoding device and an encoding method for encoding a video signal and an audio signal to generate a bit stream, and a decoding device and a decoding method for decoding encoded data multiplexed in the bit stream. is there.

  In Japanese digital broadcasting, as described in Non-Patent Document 1 below, a video stream and an audio stream that are encoded data of a video signal and an audio signal are MPEG-2 (Moving Picture Experts Group Phase-2). The data is multiplexed and transmitted in the transport stream (TS) format, which is a system standard. At this time, the encoding apparatus also multiplexes and transmits the encoded data of the metadata related to the video stream and the audio stream together with the video stream and the audio stream.

  In addition to the transport stream (TS) in MPEG-2, there is MMT (MPEG Media Transport) as a new transport system that is being standardized in MPEG, and MMT is one or more that constitute one program. When transmitting a video component (video stream) and an audio component (audio stream), it is possible to transmit in a different transmission form (for example, broadcast, communication, etc.) for each component.

Here, HEVC / H. H.265 (hereinafter referred to as “HEVC”) is a new video encoding method standardized by MPEG and ITU (International Telecommunication Union).
In HEVC, temporal hierarchical coding (scalable coding in the time direction) has been introduced, and the NAL (the coding unit that constitutes an access unit (a unit including coded data necessary for decoding one picture)) (Network Abstraction Layer) A hierarchical level can be specified for each unit.

FIG. 8 is an explanatory diagram showing an example of time hierarchy coding in HEVC.
In FIG. 8, TemporalID is identification information indicating the hierarchical level of each access unit (AU).
In the case of TemporalID = L ₀ , only the 0th layer is present, and thus no time layer encoding is performed.
When TemporalID = L ₁ , the maximum hierarchy is a hierarchy of 1, and temporal hierarchy encoding is performed.
Similarly, in the case of TemporalID = L ₂ and L ₃ , the maximum hierarchies are two or three hierarchies, respectively, and temporal hierarchy coding is performed.

Since the contents of the time hierarchy coding are known, detailed description thereof will be omitted. However, as a restriction on the time hierarchy coding, an access unit (AU) having a layer level higher than the layer level of the access unit (AU) to be coded is included. ) Cannot be referred to.
In HEVC, temporal hierarchy coding is possible using a reference structure with a maximum hierarchy of up to 6.

FIG. 9 is an explanatory diagram showing an example of a picture structure.
In FIG. 9, IRAP is an IRAP (Intra Random Access Point) picture defined by HEVC. When decoding is started from the middle of a bitstream, the IRAP picture after the IRAP picture is normally displayed in the display order. Guaranteed to be decrypted.
GOP (Group Of Pictures) may decode all of the video signals of the one or more access units when the video signal of one or more access units (AU) is encoded by the inter-frame predictive coding method. It is a set of possible access units (AU). That is, it is a set of an IRAP picture that is the first access unit (AU) in the coding order and an access unit (AU) (picture other than the IRAP picture) that follows the IRAP picture.
The SOP is a set of pictures having a hierarchy level 0, the first picture in the coding order, and pictures of the hierarchy level 1 or higher following the picture.
One sequence of bitstreams is composed of one or more GOPs, and one GOP is composed of one or more SOPs. In the example of FIG. 9, the GOP is composed of the L ₃ SOP and the L ₂ SOP.

FIG. 10 is an explanatory diagram showing the encoding order and display order of each picture encoded with the picture structure of FIG.
In the first SOP, each picture following the IRAP picture that is the first access unit (AU) in the encoding order is encoded using the previous SOP picture as a reference image.
For this reason, in order to normally decode each picture following the IRAP picture, it is necessary to refer to the picture of the previous SOP, and therefore, the picture of the previous SOP needs to be already decoded.
Therefore, when decoding is started from the IRAP picture of the first SOP, the previous SOP picture cannot be referred to (the previous SOP picture has not been decoded). Each subsequent picture cannot be decoded normally.
In HEVC, a picture that follows an IRAP picture in coding order and is earlier in display order than the IRAP picture is called LP (Leading Picture). When decoding starts from an IRAP picture, LP performs decoding normally. I can't.
Note that since the IRAP picture is encoded by the intra-frame encoding method, it can be normally decoded even if the previous SOP picture is not decoded.

For example, in digital broadcasting, a plurality of bit streams are distributed simultaneously, and the bit stream to be displayed is switched by channel switching by the user.
Since IRAP pictures are periodically inserted into the plurality of bit streams, when a user switches channels, display video can be switched by starting decoding from any of the IRAP pictures.

STD-B32 (Standard for digital broadcasting established by ARIB (Radio Industry Association))

Since the conventional coding apparatus is configured as described above, when performing temporal layer coding, the larger the maximum layer, the greater the number of pictures that make up the SOP. For this reason, when the decoding device starts decoding from an IRAP picture, the number of LPs that cannot be decoded normally increases. Therefore, when the bit stream to be displayed is switched by the channel switching by the user, It takes a long time to display a picture by a bitstream, and there is a problem that a time during which no picture is displayed becomes long.
That is, when the bit stream to be displayed is switched by channel switching by the user, what is the interval between the display time of the first LP (B25 picture in the example of FIG. 10) and the display time of the IRAP picture. Since no picture is displayed, there is a problem that the time during which no picture is displayed becomes longer as the number of pictures constituting the SOP increases.

  The present invention has been made in order to solve the above-described problems, and is an encoding that can realize seamless channel switching by eliminating the time during which no picture is displayed when the channel is switched by the user. It is an object to obtain an apparatus, a decoding apparatus, an encoding method, and a decoding method.

  An encoding apparatus according to the present invention encodes a video signal in units of video access units and outputs encoded data of the video signal, and video of one or more access units by the video encoding means. When the signal is encoded by the inter-frame predictive encoding method, the first in the presentation order for each GOP that is a set of a plurality of access units capable of decoding all of the video signals of one or more access units. The control information including the presentation time information indicating the presentation time of the access unit and the number information indicating the number of the access units whose presentation order is earlier than the first access unit in the encoding order is encoded to encode the control information in GOP units. Control information encoding means for outputting encoded data, and the multiplexing means is adapted for the encoded data of the video signal output from the video encoding means and the control information code. Multiplexes the encoded data of the control information output from the means, in which to output the bit stream is coded data after multiplexing.

  According to the present invention, when the video signal of one or more access units is encoded by the inter-frame prediction encoding method by the video encoding means, it is possible to decode all of the video signals of one or more access units. For each GOP that is a set of a plurality of access units, the presentation time information indicating the presentation time of the first access unit in the presentation order and the number indicating the number of access units earlier in the encoding order than the first access unit in the encoding order Control information encoding means for encoding control information including information and outputting encoded data of GOP unit control information is provided, and the multiplexing means encodes control information output from the control information encoding means. Since the data is configured to be included in the bitstream, there is no time when no picture is displayed when the channel is switched by the user. To an effect capable of seamless channel switching.

It is a block diagram which shows the encoding apparatus by Embodiment 1 of this invention. It is a flowchart which shows the processing content (encoding method) of the encoding apparatus by Embodiment 1 of this invention. It is a block diagram which shows the decoding apparatus by Embodiment 1 of this invention. It is a flowchart which shows the processing content (decoding method) of the decoding apparatus by Embodiment 1 of this invention. It is explanatory drawing which shows the outline | summary of the coding data in the case of transmitting a bit stream by MMT. It is explanatory drawing which shows the structural example of MPU. It is explanatory drawing which shows a HEVC picture structure descriptor. It is explanatory drawing which shows the example of time hierarchy encoding in HEVC. It is explanatory drawing which shows an example of a picture structure. It is explanatory drawing which shows the encoding order and presentation order of each picture encoded with the picture structure of FIG.

Embodiment 1 FIG.
FIG. 1 is a block diagram showing an encoding apparatus according to Embodiment 1 of the present invention.
In FIG. 1, when a digital audio signal is given, the audio encoding unit 1 encodes the audio signal in units of audio access units (AU) by a method such as MPEG-4 audio. The audio stream that is the encoded data is generated, and the process for encoding the metadata related to the audio stream is performed.
The audio MMTP payload generation unit 2 performs a process of generating an audio MMTP payload including the metadata encoded by the audio encoding unit 1 and encoded data of an audio signal in units of access units (AU).

When a digital video signal is given, the HEVC encoding unit 3 encodes the video signal by the HEVC method for each video access unit (AU), and generates a video stream that is encoded data of the video signal. At the same time, a process for encoding the metadata related to the video stream is performed.
The video MMTP payload generation unit 4 performs a process of generating a video MMTP payload composed of metadata encoded by the HEVC encoding unit 3 and encoded data of a video signal in units of access units (AU). The HEVC encoding unit 3 and the video MMTP payload generation unit 4 constitute video encoding means.
Here, for example, time information indicating DTS (decoding time) or PTS (presentation time) of each access unit (AU) can be described as metadata related to the audio stream and the video stream.

The control information encoding unit 5 encodes control information called PA message defined by MMT as control information related to the audio stream generated by the audio encoding unit 1 and the video stream generated by the HEVC encoding unit 3. Perform the process.
In the PA message, information on one or more video components (video streams) and audio components (audio streams) constituting one program (referred to as a package in MMT) is described. In MMT, video components and audio components are called assets.
Specifically, an asset ID for identifying an asset, an asset type for identifying an asset type (a type such as a HEVC format video stream or an MPEG-4 audio format audio stream), and an MPU (Media Processing Unit) of each asset are set. MPU time stamp descriptor (presentation time information) indicating the presentation time (display time) of the first access unit (AU) in the presentation order (display order) among the configured access units (AU), HEVC picture structure The descriptor, the packet ID indicating the MMTP packet storing the encoded data and metadata of each asset, and the like are included in the PA message by the number of assets constituting the package.
The MPU is composed of one or more access units (AU), and is a unit that can perform video and audio decoding processing by the MPU alone. Also, the MPU decodes all of the video signals of the one or more access units (AU) when the video signals of the one or more access units (AU) are encoded by the inter-frame predictive coding method. It becomes the same unit as GOP which is a set of a plurality of access units (AU).

  The HEVC picture structure descriptor includes the number of access units (AU) (the number of LPs) among the access units (AU) constituting the MPU that are earlier in presentation order than the first access unit (AU) in the coding order. ) Indicating the number of pieces of information (num_of_leading_picture), picture type information (rap_type) indicating the encoding method of the NAL unit (null unit) constituting the head access unit (AU) in the encoding order, and LP Describes picture type information (nal_unit_type_of_leading_picture) indicating the encoding method of the NAL unit.

The control MMTP payload generation unit 6 performs a process of generating a control MMTP payload composed of encoded data of control information encoded by the control information encoding unit 5.
The control information encoding unit 5 and the control MMTP payload generating unit 6 constitute control information encoding means.

  The MMTP packet generation unit 7 includes an audio MMTP payload generated by the audio MMTP payload generation unit 2, a video MMTP payload generated by the video MMTP payload generation unit 4, a control MMTP payload generated by the control MMTP payload generation unit 6, Is executed to generate MMTP packets constituting the bit stream. When this MMTP packet is generated, a predetermined MMTP header is added, and this MMTP header includes a packet ID assigned according to the type of encoded data included in the MMTP payload. The MMTP packet generation unit 7 constitutes a multiplexing unit.

In the example of FIG. 1, an audio encoding unit 1, an audio MMTP payload generating unit 2, an HEVC encoding unit 3, a video MMTP payload generating unit 4, a control information encoding unit 5, and a control MMTP payload that are components of the encoding device. It is assumed that each of the generation unit 6 and the MMTP packet generation unit 7 is configured with dedicated hardware (for example, a semiconductor integrated circuit on which a CPU is mounted, or a one-chip microcomputer). The digitizing device may be constituted by a computer.
When the encoding apparatus is configured by a computer, an audio encoding unit 1, an audio MMTP payload generating unit 2, an HEVC encoding unit 3, a video MMTP payload generating unit 4, a control information encoding unit 5, a control MMTP payload generating unit 6, and A program describing the processing contents of the MMTP packet generation unit 7 may be stored in a memory of a computer so that the CPU of the computer executes the program stored in the memory.
FIG. 2 is a flowchart showing the processing contents (encoding method) of the encoding apparatus according to Embodiment 1 of the present invention.

3 is a block diagram showing a decoding apparatus according to Embodiment 1 of the present invention.
In FIG. 3, the stream selection unit 11 is a bit stream (a bit stream made up of MMTP packets) output from a plurality of encoding devices (the encoding device in FIG. 1 or an encoding device corresponding to the encoding device in FIG. 1). ) To select a bit stream to be presented and output the bit stream to the MMTP packet analysis unit 12.
Further, when an instruction to switch the bit stream to be presented is given, the stream selection unit 11 selects the bit stream after switching from the bit streams output from the plurality of encoding devices, and selects the bit stream. While outputting to the MMTP packet analysis part 12, until the presentation time calculated by the control MMTP payload processing part 13 is reached, the process of continuously outputting the bit stream before switching to the MMTP packet analysis part 12 is performed. The stream selection unit 11 constitutes a bit stream selection unit.

  The MMTP packet analysis unit 12 analyzes the MMTP header of the MMTP packet constituting the bit stream output from the stream selection unit 11, acquires the packet ID included in the MMTP header, and the packet ID is MMTP If the encoded data included in the payload indicates control information (PA message, HEVC picture structure descriptor), the control MMTP payload, which is the MMTP payload included in the MMTP packet, is set as the control MMTP payload. Output to the processing unit 13. On the other hand, if the packet ID indicates that the encoded data included in the MMTP payload is an audio signal or a video signal, a process of outputting the MMTP packet to the asset separation unit 14 is performed.

The control MMTP payload processing unit 13 performs a decoding process on the encoded data included in the control MMTP payload output from the MMTP packet analysis unit 12, and performs the PA message as control information and the HEVC included in the PA message. Decode the picture structure descriptor.
Also, the control MMTP payload processing unit 13 presents the presentation time of the first access unit (AU) in the presentation order indicated by the MPU time stamp descriptor described in the PA message, and the number information described in the HEVC picture structure descriptor. The presentation time of the first access unit (AU) in the coding order from the number of access units (AU) (the number of LPs) that are presented earlier than the first access unit (AU) in the coding order indicated by (num_of_leading_picture) The process of calculating is performed. The head access unit (AU) in the coding order is the IRAP picture of the head SOP. The control MMTP payload processing unit 13 constitutes a presentation time calculation unit.

  The asset separation unit 14 is included in the MMTP packet output from the MMTP packet analysis unit 12 with reference to the asset ID, asset type, and packet ID described in the PA message decoded by the control MMTP payload processing unit 13. The MMTP payload is an audio MMTP payload or a video MMTP payload, and if it is an audio MMTP payload, the audio MMTP payload included in the MMTP packet is extracted, and the audio MMTP payload is If the video MMTP payload is output to the MMTP payload processing unit 15, the video MMTP payload included in the MMTP packet is extracted, and the video MMTP payload is output to the video MMTP payload processing unit 19. Hodokosuru.

The audio MMTP payload processing unit 15 reconstructs the MFU (Media Fragment Unit) or MPU of the audio stream from the audio MMTP payload output from the asset separation unit 14, so that the audio stream decoding unit 17 can decode the audio stream. A process of generating an audio elementary stream (audio ES) and storing the audio ES in the audio ES buffer 16 is performed. The MFU is a smaller unit than the MPU, and one access unit (AU) or one NAL unit can be defined as one MFU.
In addition, the audio MMTP payload processing unit 15 extracts the metadata regarding the audio stream included in the audio MMTP payload output from the asset separation unit 14 and stores the metadata in the audio ES buffer 16.
The audio ES buffer 16 is a memory that temporarily stores the audio ES and metadata.

The audio stream decoding unit 17 extracts the metadata from the audio ES buffer 16, and time information described in the metadata (information indicating the DTS (decoding time) and PTS (presentation time) of each access unit (AU)) A process of decoding is performed.
Further, when the DTS (decoding time) of each access unit (AU) is reached, the audio stream decoding unit 17 extracts the audio ES from the audio ES buffer 16, decodes the audio signal of the access unit (AU), and decodes the audio signal. A process of storing the audio signal and the PTS (presentation time) in the audio data buffer 18 is performed.
The audio data buffer 18 is a memory that temporarily stores the audio signal decoded by the audio stream decoding unit 17 and the PTS (presentation time).

The video MMTP payload processing unit 19 reconstructs the video stream MFU or MPU from the video MMTP payload output from the asset separation unit 14, thereby enabling the HEVC elementary stream (HEVC) in a format decodable by the HEVCES decoding unit 21 in the subsequent stage. ES) is generated, and the HEVC elementary stream is stored in the HEVCES buffer 20.
In addition, the video MMTP payload processing unit 19 extracts metadata related to the video stream included in the video MMTP payload output from the asset separation unit 14 and stores the metadata in the HEVCES buffer 20.
The HEVCES buffer 20 is a memory that temporarily stores HEVC elementary streams and metadata.

The HEVCES decoding unit 21 extracts metadata from the HEVCES buffer 20 and decodes time information described in the metadata (information indicating DTS (decoding time) and PTS (presentation time) of each access unit (AU)). Perform the process.
When the HEVCES decoding unit 21 reaches the DTS (decoding time) of each access unit (AU), the HEVCES decoding unit 21 extracts the HEVC elementary stream from the HEVCES buffer 20, decodes the video signal of the access unit (AU), and decodes the decoded video signal. A process of storing the decoded image and the PTS (presentation time) which are video signals in the decoded image buffer 22 is performed.
The decoded image buffer 22 is a memory that temporarily stores the decoded image and PTS (presentation time) of each access unit (AU) decoded by the HEVCES decoding unit 21.
The video MMTP payload processing unit 19, the HEVCES buffer 20, the HEVCES decoding unit 21, and the decoded image buffer 22 constitute video decoding means.

In the example of FIG. 3, the stream selection unit 11, the MMTP packet analysis unit 12, the control MMTP payload processing unit 13, the asset separation unit 14, the audio MMTP payload processing unit 15, the audio ES buffer 16, and the audio stream that are constituent elements of the decoding device Each of the decoding unit 17, the audio data buffer 18, the video MMTP payload processing unit 19, the HEVCES buffer 20, the HEVCES decoding unit 21, and the decoded image buffer 22 is dedicated hardware (other than the buffer, for example, a semiconductor on which a CPU is mounted. Although an integrated circuit or a one-chip microcomputer is assumed, the decoding device may be configured by a computer.
When the decoding apparatus is configured by a computer, the audio ES buffer 16, the audio data buffer 18, the HEVCES buffer 20, and the decoded image buffer 22 are configured on the internal memory or the external memory of the computer, and the stream selection unit 11, the MMTP packet analysis unit 12, the control MMTP payload processing unit 13, the asset separation unit 14, the audio MMTP payload processing unit 15, the audio stream decoding unit 17, the video MMTP payload processing unit 19 and the HEVCES decoding unit 21 which describe the processing contents of the computer What is necessary is just to make it memorize | store in memory and to run the program stored in the said memory of CPU of the said computer.
FIG. 4 is a flowchart showing the processing contents (decoding method) of the decoding apparatus according to Embodiment 1 of the present invention.

Next, the operation will be described.
The processing contents of the first encoding device will be described.
When a digital audio signal is given, the audio encoding unit 1 encodes the audio signal in units of audio access units (AU) by a method such as MPEG-4 audio, and encodes the audio signal. An audio stream that is data is generated, and metadata relating to the audio stream is encoded (step ST1 in FIG. 2).
When a digital video signal is given, the HEVC encoding unit 3 encodes the video signal by the HEVC method for each video access unit (AU), and generates a video stream that is encoded data of the video signal. At the same time, the metadata about the video stream is encoded (step ST2).

Here, FIG. 5 is an explanatory diagram showing an outline of encoded data when a bit stream is transmitted by MMT.
In FIG. 5, an access unit (AU) is a unit that includes encoded data necessary for decoding one picture if it is a video, and is composed of one or more samples that are an encoding unit if it is audio. Is the frame to be played.
The NAL unit is a HEVC encoding unit, and one access unit (AU) is composed of one or more NAL units.
The MPU is composed of one or more access units, and is a unit that can perform video and audio decoding processing by the MPU alone. Also, the MPU decodes all of the video signals of the one or more access units (AU) when the video signals of the one or more access units (AU) are encoded by the inter-frame predictive coding method. It becomes the same unit as GOP which is a set of a plurality of access units (AU).
The MFU is a smaller unit than the MPU, and one access unit (AU) or one NAL unit can be defined as one MFU.

FIG. 6 is an explanatory diagram showing a configuration example of the MPU.
In FIG. 6, MPU metadata describes metadata related to the MPU, such as time information indicating DTS (decoding time) and PTS (presentation time) of each access unit (AU) included in the MPU. Can be described.
Movie fragment metadata (MF metadata) describes metadata accompanying encoded data (sample data) of one access unit (AU). For example, when the encoded data of the access unit (AU) is stored in a file format, for each access unit (AU), the address where the encoded data is stored, the data length of the encoded data, the access unit (AU) ) Is included.
MPU metadata, movie fragment metadata, MFU and MMT control information are transmitted as MMTP packets. The MMTP packet is composed of an MMTP header and an MMTP payload.

When the audio MMTP payload generation unit 2 receives the encoded data of the metadata (MPU metadata, MF meta, etc.) and the encoded data of the audio signal in units of access units (AU) from the audio encoding unit 1, An audio MMTP payload including encoded data of MPU metadata in units, encoded data of MF meta in units of access units (AU), and encoded data (sample data) of audio signals is generated (step ST3).
When receiving the encoded data of metadata (MPU metadata, MF meta, etc.) and the encoded data of the video signal in units of access units (AU) from the HEVC encoder 3, the video MMTP payload generator 4 receives the MPU. A video MMTP payload composed of encoded data of MPU metadata in units, encoded data of MF meta in units of access units (AU), and encoded data (sample data) of video signals is generated (step ST4).

The control information encoding unit 5 encodes the control information regarding the audio stream generated by the audio encoding unit 1 and the video stream generated by the HEVC encoding unit 3 (step ST5).
As control information related to an audio stream and a video stream, for example, a PA message or HEVC picture structure descriptor defined by MMT is encoded.
As described above, in the PA message, information on one or more video components (video streams) and audio components (audio streams) constituting one program (referred to as a package in MMT) is described.
That is, the PA message includes an asset ID for identifying an asset (video stream, audio stream) generated by the audio encoding unit 1 and HEVC encoding unit 3, an asset type for identifying the asset type, and an MPU for each asset. MMTP packet storing MPU time stamp descriptor indicating the presentation time of the first access unit (AU) in the order of presentation among the configured access units (AU), encoded data and metadata of each asset A packet ID or the like indicating is described.

FIG. 7 is an explanatory diagram showing the HEVC picture structure descriptor.
As shown in FIG. 7, the HEVC picture structure descriptor includes an access unit (AU) that is earlier in presentation order than the first access unit (AU) in the coding order among the access units (AU) constituting the MPU. ) (Number_of_leading_picture) indicating the number (number of LPs).
Also, picture type information (rap_type) indicating the encoding method of the NAL unit constituting the head access unit (AU) in the encoding order, and picture type indicating the encoding method of the NAL unit constituting the LP. Information (nal_unit_type_of_leading_picture) and the like are described.

When receiving the encoded data of the control information from the control information encoding unit 5, the control MMTP payload generating unit 6 generates a control MMTP payload including the encoded data of the control information (step ST6).
The MMTP packet generator 7 includes an audio MMTP payload generated by the audio MMTP payload generator 2, a video MMTP payload generated by the video MMTP payload generator 4, and a control MMTP payload generated by the control MMTP payload generator 6. And MMTP packets constituting the bit stream are generated (step ST7).
When this MMTP packet is generated, a predetermined MMTP header is added, and this MMTP header includes a packet ID assigned according to the type of encoded data included in the MMTP payload.

Next, processing contents of the decoding device will be described.
The stream selection unit 11 is provided with a bit stream (a bit stream composed of MMTP packets) output from a plurality of encoding devices (the encoding device in FIG. 1 or an encoding device corresponding to the encoding device in FIG. 1). It is done.
The stream selection unit 11 selects a bit stream (bit stream to be presented) of a channel designated by the user from a plurality of bit streams, and outputs the bit stream to the MMTP packet analysis unit 12.

When receiving the bit stream from the stream selection unit 11, the MMTP packet analysis unit 12 analyzes the MMTP header of the MMTP packet constituting the bit stream, and acquires the packet ID included in the MMTP header.
If the packet ID indicates that the encoded data included in the MMTP payload is control information (PA message), the MMTP packet analysis unit 12 controls the MMTP payload included in the MMTP packet. The MMTP payload is output to the control MMTP payload processing unit 13.
On the other hand, if the packet ID indicates that the encoded data included in the MMTP payload is an audio signal or a video signal, the MMTP packet is output to the asset separation unit 14.

  When the control MMTP payload processing unit 13 receives the control MMTP payload from the MMTP packet analysis unit 12, the control MMTP payload processing unit 13 performs a decoding process on the encoded data included in the control MMTP payload, and controls the PA message and the PA message as control information. The HEVC picture structure descriptor included in is decoded.

When the control MMTP payload processing unit 13 decrypts the PA message, the asset separation unit 14 refers to the asset ID, asset type, and packet ID described in the PA message, and outputs the MMTP output from the MMTP packet analysis unit 12. It is specified whether the MMTP payload included in the packet is an audio MMTP payload or a video MMTP payload.
If the MMTP payload included in the MMTP packet output from the MMTP packet analysis unit 12 is an audio MMTP payload, the asset separation unit 14 extracts the audio MMTP payload included in the MMTP packet and extracts the audio The MMTP payload is output to the voice MMTP payload processing unit 15.
If the MMTP payload included in the MMTP packet output from the MMTP packet analysis unit 12 is a video MMTP payload, the asset separation unit 14 extracts the video MMTP payload included in the MMTP packet and extracts the video The MMTP payload is output to the video MMTP payload processing unit 19.

When the audio MMTP payload processing unit 15 receives the audio MMTP payload from the asset separation unit 14, the audio MMTP payload processing unit 15 reconstructs the MFU or MPU of the audio stream from the audio MMTP payload, so that the audio stream decoding unit 17 can decode the audio stream. Audio elementary stream (audio ES) is generated, and the audio ES is stored in the audio ES buffer 16.
Since the process itself for generating the audio ES from the audio MMTP payload is a known technique, detailed description thereof is omitted.
In addition, the audio MMTP payload processing unit 15 extracts metadata about the audio stream included in the audio MMTP payload output from the asset separation unit 14 and stores the metadata in the audio ES buffer 16.

The audio stream decoding unit 17 extracts metadata from the audio ES buffer 16, and information indicating time information (DTS (decoding time) and PTS (presentation time) of each access unit (AU)) described in the metadata. ).
The audio stream decoding unit 17 refers to the decoded DTS, grasps the decoding time of each access unit (AU), and extracts the audio ES from the audio ES buffer 16 when the decoding time of each access unit (AU) comes. The audio signal of the access unit (AU) is decoded, and the decoded audio signal and PTS (presentation time) are stored in the audio data buffer 18.
Thus, if an external playback device (not shown) takes out the audio signal and the PTS (presentation time) stored in the audio data buffer 18, the audio signal can be reproduced at the presentation time.

When the video MMTP payload processing unit 19 receives the video MMTP payload from the asset separation unit 14, the video MMTP payload processing unit 19 reconstructs the MFU or MPU of the video stream from the video MMTP payload, so that the HEVCES decoding unit 21 in the subsequent stage can decode it. A HEVC elementary stream (HEVC ES) is generated, and the HEVC elementary stream is stored in the HEVCES buffer 20.
Since the process itself for generating the HEVC elementary stream from the video MMTP payload is a known technique, detailed description thereof is omitted.
In addition, the video MMTP payload processing unit 19 extracts metadata regarding the video stream included in the video MMTP payload output from the asset separation unit 14, and stores the metadata in the HEVCES buffer 20.

The HEVCES decoding unit 21 extracts the metadata from the HEVCES buffer 20, and obtains time information (information indicating the DTS (decoding time) and PTS (presentation time) of each access unit (AU)) described in the metadata. Decrypt.
The HEVCES decoding unit 21 refers to the decoded DTS, grasps the decoding time of each access unit (AU), and extracts the HEVC elementary stream from the HEVCES buffer 20 at the decoding time of each access unit (AU). Then, the video signal of the access unit (AU) is decoded, and the decoded image and the PTS (presentation time) as the decoded video signal are stored in the decoded image buffer 22.
Thus, if an external playback device (not shown) takes out the decoded image and the PTS (presentation time) stored in the decoded image buffer 22, the decoded image can be reproduced at the presentation time.

When an external playback device (not shown) is playing back a decoded image and an audio signal, if the user performs an operation to switch channels using a remote controller or the like, a command to switch the video stream to be presented (this switching command) Includes information indicating the channel after switching) to the stream selection unit 11.
When receiving a channel switching command from the outside (in the case of step ST11: Yes in FIG. 4), the stream selection unit 11 selects a bit stream of the channel after switching indicated by the switching command from a plurality of bit streams. The bit stream is output to the MMTP packet analysis unit 12 (step ST12).
In addition, the stream selection unit 11 outputs a bit stream from the control MMTP payload processing unit 13 in order to eliminate the time during which no decoded image is displayed when the channel is switched by the user and to realize seamless channel switching. Until a stop command is received (step ST13: No), the bit stream of the channel before switching is continuously output to the MMTP packet analysis unit 12 (step ST14). The bitstream output stop command is output when the current time is the presentation time of the first access unit (AU) in the encoding order, as will be described later.
Upon receiving a bitstream output stop command from the control MMTP payload processing unit 13, the stream selection unit 11 stops outputting the bitstream of the channel before switching, and only the bitstream of the channel after switching is analyzed by the MMTP packet analysis unit. 12 is output.

When the MMTP packet analysis unit 12 receives the bit stream of the channel after switching from the stream selection unit 11, the MMTP payload included in the MMTP packet constituting the bit stream is received as in the case before the channel switching. The audio MMTP payload or video MMTP payload included in the MMTP packet constituting the bit stream is output to the asset separation unit 14.
Further, when the MMTP packet analysis unit 12 receives the bit stream of the channel before switching from the stream selection unit 11, the MMTP packet analysis unit 12 configures the bit stream of the channel before switching by processing parallel to the processing of the bit stream of the channel after switching. The control MMTP payload included in the MMTP packet being output is output to the control MMTP payload processing unit 13, and the audio MMTP payload or the video MMTP payload included in the MMTP packet constituting the bit stream is converted into the asset separation unit 14. Output to.

When receiving the control MMTP payload related to the bit stream of the channel after switching from the MMTP packet analysis unit 12, the control MMTP payload processing unit 13 encodes the encoded data included in the control MMTP payload as before the channel switching. Is decoded to decode the PA message as the control information and the HEVC picture structure descriptor included in the PA message (step ST15).
When the control MMTP payload processing unit 13 decodes the PA message and the HEVC picture structure descriptor included in the PA message, the top access unit (AU) in the presentation order indicated by the MPU time stamp descriptor described in the PA message. Presentation time and the number of access units (AU) that are earlier in presentation order than the top access unit (AU) in the encoding order indicated by the number information (num_of_leading_picture) described in the HEVC picture structure descriptor (number of LPs) From the above, the presentation time of the head access unit (AU) in the encoding order is calculated (step ST16).

In the example of FIG. 10, the head access unit (AU) in the coding order is IRAP32, and the head access unit (AU) in the presentation order is B25. Also, the number of access units (AU) (number of LPs) that are presented earlier than IRAP 32 is seven.
Therefore, if the presentation time of B25, which is the first access unit (AU) in the presentation order, is, for example, 18:00:00 and the frame rate is 120 frames / second, the presentation time of IRAP32 is B25. This is 58 msec (= 7/120) after the presentation time (18:00:00).
When the current time is the presentation time of the first access unit (AU) in the encoding order (step ST17: Yes), the control MMTP payload processing unit 13 selects the output stop instruction of the bit stream of the channel before switching as a stream. It outputs to the part 11 (step ST18).

Upon receiving the MMTP packet related to the channel after switching from the MMTP packet analysis unit 12, the asset separation unit 14 converts the voice MMTP payload included in the MMTP packet into the voice MMTP payload processing unit 15 as before channel switching. And the video MMTP payload included in the MMTP packet is output to the video MMTP payload processing unit 19.
Further, when the asset separation unit 14 receives the MMTP packet related to the channel before switching from the MMTP packet analysis unit 12, the asset separation unit 14 converts the MMTP packet related to the channel before switching into processing in parallel with the processing for the bit stream of the channel after switching. The included audio MMTP payload is output to the audio MMTP payload processing unit 15, and the video MMTP payload included in the MMTP packet is output to the video MMTP payload processing unit 19.

When the video MMTP payload processing unit 19 receives the video MMTP payload related to the channel before switching from the asset separation unit 14 (before the current time is the presentation time of the first access unit (AU) in the presentation order), the video switching is performed. As before, a HEVC elementary stream is generated from the video MMTP payload, the HEVC elementary stream is stored in the HEVCES buffer 20, and metadata about the video stream included in the video MMTP payload is extracted. The metadata is stored in the HEVCES buffer 20 (step ST19).
In addition, when the video MMTP payload processing unit 19 receives the video MMTP payload related to the channel after switching from the asset separation unit 14, the video MMTP payload processing unit 19 performs processing related to the bit stream of the channel before switching and processing related to the channel after switching. A HEVC elementary stream is generated from the MMTP payload, the HEVC elementary stream is stored in the HEVCES buffer 20, metadata about the video stream included in the video MMTP payload is extracted, and the metadata is stored in the HEVCES buffer. 20 (step ST20).

The HEVCES decoding unit 21 takes out the metadata from the HEVCES buffer 20 in the same manner as before channel switching, and extracts time information (DTS (decoding time) of each access unit (AU) or PTS (decoding time)) in the metadata. Information) indicating the presentation time) is decoded.
As a result, the HEVCES decoding unit 21 refers to the decoded DTS to grasp the decoding time of each access unit (AU). However, for the channel after switching, the access unit that can be decoded first by the IRAP. (AU) (the IOP32 access unit (AU) in the GOP configuration of FIG. 10), and the video signal of the LP access unit (AU) whose presentation order is earlier than the IRAP cannot be decoded. For this reason, for the channel after switching, the video signal of any access unit (AU) cannot be decoded and displayed until the IRAP presentation time is reached.

Therefore, the HEVCES decoding unit 21 decodes the video signal of the access unit (AU) related to the channel before switching until the IRAP presentation time related to the channel after switching, and is the decoded video signal. The decoded image and the PTS (presentation time) are stored in the decoded image buffer 22.
Accordingly, an external playback device (not shown) can play back the decoded image related to the channel before switching until the IRAP presentation time related to the channel after switching is reached.
Therefore, seamless channel switching can be realized by eliminating the time during which no decoded image is displayed when the channel is switched by the user.

When the voice MMTP payload processing unit 15 receives the voice MMTP payload related to the channel before switching from the asset separation unit 14 (before the current time is the presentation time of the first access unit (AU) in the presentation order), the channel switching is performed. As before, an audio ES is generated from the audio MMTP payload, the audio ES is stored in the audio ES buffer 16, and metadata about the audio stream included in the audio MMTP payload is extracted, and the meta data is extracted. Data is stored in the audio ES buffer 16.
When the audio MMTP payload processing unit 15 receives the audio MMTP payload related to the channel after switching from the asset separation unit 14, the audio MMTP payload processing unit 15 performs processing related to the bit stream of the channel before switching in parallel with the processing related to the bit stream of the channel before switching. An audio ES is generated from the MMTP payload, the audio ES is stored in the audio ES buffer 16, metadata about an audio stream included in the audio MMTP payload is extracted, and the metadata is stored in the audio ES buffer 16. Store.

The audio stream decoding unit 17 extracts the metadata from the audio ES buffer 16 in the same manner as before the channel switching, and the time information described in the metadata (DTS (decoding time) of each access unit (AU), (Information indicating PTS (presentation time)) is decoded.
The audio stream decoding unit 17 refers to the decoded DTS, grasps the decoding time of each access unit (AU), and extracts the audio ES from the audio ES buffer 16 when the decoding time of each access unit (AU) comes. The audio signal of the access unit (AU) is decoded, and the decoded audio signal and PTS (presentation time) are stored in the audio data buffer 18.
Thus, if an external playback device (not shown) takes out the audio signal and the PTS (presentation time) stored in the audio data buffer 18, the audio signal can be reproduced at the presentation time.

  Here, as in the HEVCES decoding unit 21, the audio stream decoding unit 17 receives the audio signal of the access unit (AU) related to the channel before switching until the IRAP presentation time related to the channel after switching is reached. It is assumed that the decoded audio signal and the PTS (presentation time) are stored in the audio data buffer 18, but the audio signal is encoded by an inter-frame predictive encoding method like a video signal. Therefore, the audio signal of the LP access unit (AU) whose presentation order is earlier than that of the IRAP can also be decoded. Therefore, even before the IRAP presentation time related to the channel after switching is reached, the access related to the channel after switching is not decoded without decoding the audio signal of the access unit (AU) related to the channel before switching. The audio signal of the unit (AU) may be decoded.

  In the decoding apparatus, the processes of steps ST11 to ST20 are repeatedly performed until the decoding process is completed (step ST21).

  As is apparent from the above, according to the first embodiment, when video signals of one or more access units (AU) are encoded by the inter-frame predictive coding method, one or more access units (AU). Presentation time information indicating the presentation time of the first access unit (AU) in the presentation order for each GOP, which is a set of a plurality of access units (AU) capable of decoding all of the video signals, and the encoding order Since the control information including the number information indicating the number of access units (AU) whose presentation order is earlier than that of the top access unit (AU) is encoded, the channel is switched by the user on the decoding side. Sometimes, there is an effect that an encoding device capable of realizing seamless channel switching without a time during which no decoded image is displayed can be obtained. .

  Further, according to the first embodiment, when a command to switch the bit stream to be presented is given, the stream selection unit 11 selects the bit after switching from the bit streams output from the plurality of encoding devices. A stream is selected, and the bit stream is output to the MMTP packet analysis unit 12 and until the presentation time calculated by the control MMTP payload processing unit 13 is reached (until the bit stream output stop command is received). The bit stream is also output to the MMTP packet analysis unit 12, and the HEVCES decoding unit 21 decodes the video signal in units of access units from the encoded data of the video signal multiplexed in the bit stream after switching, and the control MMTP The presentation time calculated by the payload processing unit 13 is reached. Since the video signal for each access unit is decoded from the encoded data of the video signal multiplexed in the bit stream before switching, no decoding is performed when the channel is switched by the user. There is an effect that a decoding device capable of realizing seamless channel switching without a time during which no image is displayed can be obtained.

Embodiment 2. FIG.
In the first embodiment, the HEVCES decoding unit 21 decodes the video signal of the access unit (AU) related to the channel before switching until the IRAP presentation time related to the channel after switching is reached. Although the decoded image and the PTS (presentation time) that are the video signals are stored in the decoded image buffer 22, the access unit (AU) of the LP access unit (AU) related to the channel after switching is shown. Depending on the encoding method of the NAL unit that constitutes), it may be possible to perform decoding even before IRAP decoding.
Whether or not the LP access unit (AU) is decodable even before IRAP decoding is configured by referring to picture type information (nal_unit_type_of_leading_picture) described in the HEVC picture structure descriptor. This can be determined because the encoding method of the NAL unit being used is known. For example, if the encoding method of the NAL unit constituting the LP is an intra-frame encoding method, it can be decoded even if the video signal of the access unit (AU) of the previous SOP is not decoded. Is possible.

The HEVCES decoding unit 21 decodes the video signal of the access unit (AU) if it can decode the LP access unit (AU) related to the channel after switching even before the IRAP decoding. Then, the decoded image and the PTS (presentation time) which are the decoded video signals are stored in the decoded image buffer 22.
For example, in the GOP configuration of FIG. 10, B25, B25, B27, and B28 that are LPs cannot be decoded, but if B29, B30, and B31 can be decoded, the video signals of B29, B30, and B31 are decoded, The decoded image and the PTS (presentation time) that are the decoded video signals are stored in the decoded image buffer 22.
As a result, an external playback device (not shown) decodes the channel before switching at the presentation time of B25, B25, B27, and B28 until the presentation time of IRAP related to the channel after switching is reached. The image is reproduced, and at the presentation time of B29, B30, B31, the decoded image relating to the channel after switching can be reproduced.

  In the present invention, within the scope of the invention, any combination of the embodiments, or any modification of any component in each embodiment, or omission of any component in each embodiment is possible. .

  DESCRIPTION OF SYMBOLS 1 Audio encoding part, 2 Audio MMTP payload production | generation part, 3 HEVC encoding part (video coding means), 4 Video MMTP payload production | generation part (video coding means), 5 Control information coding part (Control information coding means) ), 6 control MMTP payload generation unit (control information encoding unit), 7 MMTP packet generation unit (multiplexing unit), 11 stream selection unit (bit stream selection unit), 12 MMTP packet analysis unit, 13 control MMTP payload processing unit (Presentation time calculation means), 14 asset separation unit, 15 audio MMTP payload processing unit, 16 audio ES buffer, 17 audio stream decoding unit, 18 audio data buffer, 19 video MMTP payload processing unit (video decoding unit), 20 HEVCES buffer (Video decoding means), 21 HEVCES decoding (Video decoding means), 22 the decoded picture buffer (video decoding means).

Claims (6)

Video encoding means for encoding a video signal in units of video access units and outputting encoded data of the video signal;
A plurality of accesses capable of decoding all of the video signals of the one or more access units when the video signal of the one or more access units is encoded by the interframe predictive encoding method by the video encoding means. For each GOP that is a set of units, it includes presentation time information that indicates the presentation time of the first access unit in the presentation order, and number information that indicates the number of access units that are earlier in presentation order than the first access unit in the encoding order. Control information encoding means for encoding control information and outputting encoded data of control information in GOP units;
The encoded data of the video signal output from the video encoding means and the encoded data of the control information output from the control information encoding means are multiplexed, and a bit stream that is the encoded data after multiplexing is output. An encoding device comprising: multiplexing means.   In addition to the presentation time information and the number information, the control information encoding means is one or more units of an encoding unit constituting an access unit whose presentation order is earlier than the first access unit in the encoding order. 2. The encoding apparatus according to claim 1, wherein control information including picture type information indicating an encoding method of a certain null unit is encoded, and encoded data of control information in GOP units is output. Stream selection means for selecting a bit stream to be presented from bit streams output from a plurality of encoding devices;
Video decoding means for decoding a video signal in units of access units from encoded data of the video signal multiplexed in the bit stream selected by the stream selection means;
Decoding processing of encoded data of control information in GOP units, which is a set of a plurality of access units multiplexed in the bit stream selected by the stream selection means, and presenting the first access unit in the presentation order From the presentation time information indicating the time and the number information indicating the number of access units whose order of presentation is earlier than the first access unit in the encoding order, the presentation time indicated by the presentation time information and the number indicated by the number information And a presentation time calculation means for calculating the presentation time of the head access unit in the encoding order,
When the instruction to switch the bit stream to be presented is given, the stream selection unit selects the bit stream after switching from the bit streams output from the plurality of encoding devices, and the bit stream Is output to the video decoding means, and before the presentation time calculated by the presentation time calculating means is reached, the bit stream before switching is also output to the video decoding means,
The video decoding means decodes the video signal in units of access units from the encoded data of the video signal multiplexed in the switched bit stream and until the presentation time calculated by the presentation time calculation means is reached. A decoding apparatus for decoding a video signal for each access unit from encoded data of a video signal multiplexed in the bit stream before switching.   In addition to the presentation time information and the number information, the video decoding means is a null that is one or more units of an encoding unit that constitutes an access unit whose presentation order is earlier than the first access unit in the encoding order. When picture type information indicating a unit encoding method is included in the control information, the picture type information included in the control information multiplexed in the switched bitstream is decoded, and the picture type With reference to the encoding method indicated by the information, the access unit that identifies the decodable access unit among the access units whose presentation order is earlier than the first access unit in the encoding order, and the video signal of the decodable access unit is The decoding device according to claim 3, wherein the decoding device performs decoding. A video encoding means for encoding a video signal in units of video access units and outputting encoded data of the video signal; and
The control information encoding means decodes all of the video signals of the one or more access units when the video signal of one or more access units is encoded by the inter-frame prediction encoding method in the video encoding processing step. For each GOP that is a set of a plurality of access units that can be performed, the presentation time information indicating the presentation time of the first access unit in the presentation order and the access unit whose presentation order is earlier than the first access unit in the coding order A control information encoding processing step that encodes control information including the number information indicating the number and outputs encoded data of the control information in GOP units;
The multiplexing means multiplexes the encoded data of the video signal output in the video encoding processing step and the encoded data of the control information output in the control information encoding processing step, and the encoded data after multiplexing And a multiplexing process step for outputting a bitstream. A stream selection processing step in which the stream selection means selects a bit stream to be presented from among the bit streams output from the plurality of encoding devices;
A video decoding processing step, wherein the video decoding means decodes a video signal in units of access units from encoded data of the video signal multiplexed in the bit stream selected in the stream selection processing step;
The presentation time calculation means performs a decoding process of the encoded data of the control information in GOP units, which is a set of a plurality of access units multiplexed in the bit stream selected in the stream selection processing step, Decoding the presentation time information indicating the presentation time of the first access unit and the number information indicating the number of access units earlier in the encoding order than the first access unit in the encoding order, and the presentation time indicated by the presentation time information A presentation time calculation processing step of calculating the presentation time of the first access unit in the encoding order from the number indicated by the number information,
In the stream selection processing step, when a command to switch the bit stream to be presented is given, the bit stream after switching is selected from the bit streams output from the plurality of encoding devices, and the bit is selected. While outputting the stream to the video decoding means, until the presentation time calculated in the presentation time calculation processing step, also outputs the bit stream before switching to the video decoding means,
In the video decoding processing step, the video signal for each access unit is decoded from the encoded data of the video signal multiplexed in the bit stream after the switching, and at the presentation time calculated in the presentation time calculation processing step. Until then, the decoding method characterized in that the video signal in units of access units is decoded from the encoded data of the video signal multiplexed in the bit stream before switching.

Images