JP2015226305A - Encoding device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an encoding device that can reconstruct and decode a bit stream which is subjected to time hierarchical coding on the basis of a decoding timing of each access unit in a decoding device.SOLUTION: An encoding device has control information encoding means for encoding control information containing instruction time information representing an instruction time of an access unit at the top of an instruction order and decoding time information representing a decoding time of an access unit at the top of an encoding order for each GOP which is an assemble of plural access unit capable of decoding all video signals of one or more access units which are encoded by an inter-frame prediction encoding system, time information encoding means for encoding the difference between the decoding time and the decoding time of the access unit at the top on an access-unit basis, and outputting encoded data of the time information, and multiplexing means for multiplexing the encoded data and outputting a bit stream corresponding to the multiplexed encoded data.

Description

  The present invention relates to an encoding device and an encoding method for encoding a video signal and an audio signal to generate a bit stream, and a decoding device and a decoding method for decoding encoded data multiplexed in the bit stream. is there.

  In Japanese digital broadcasting, as described in Non-Patent Document 1 below, a video stream and an audio stream that are encoded data of a video signal and an audio signal are MPEG-2 (Moving Picture Experts Group Phase-2). The data is multiplexed and transmitted in the transport stream (TS) format, which is a system standard. At this time, the encoding apparatus also multiplexes and transmits the encoded data of the metadata related to the video stream and the audio stream together with the video stream and the audio stream.

  In addition to the transport stream (TS) in MPEG-2, there is MMT (MPEG Media Transport) as a new transport system that is being standardized in MPEG, and MMT is one or more that constitute one program. When transmitting a video component (video stream) and an audio component (audio stream), it is possible to transmit in a different transmission form (for example, broadcast, communication, etc.) for each component.

Here, HEVC / H. H.265 (hereinafter referred to as “HEVC”) is a new video encoding method standardized by MPEG and ITU (International Telecommunication Union).
In HEVC, temporal hierarchical coding (scalable coding in the time direction) has been introduced, and the NAL (the coding unit that constitutes an access unit (a unit including coded data necessary for decoding one picture)) (Network Abstraction Layer) A hierarchical level can be specified for each unit.

FIG. 9 is an explanatory diagram showing an example of time hierarchy coding in HEVC.
In FIG. 9, TemporalID is identification information indicating the hierarchical level of each access unit (AU).
IRAP is an IRAP (Intra Random Access Point) picture defined by HEVC. When decoding is started from the middle of a bitstream, pictures following the IRAP picture in the display order are normally decoded. Is guaranteed.
GOP (Group Of Pictures) may decode all of the video signals of the one or more access units when the video signal of one or more access units (AU) is encoded by the inter-frame predictive coding method. It is a set of possible access units (AU). That is, it is a set of an IRAP picture that is the first access unit (AU) in the coding order and an access unit (AU) (picture other than the IRAP picture) that follows the IRAP picture.

Since the contents of the time hierarchy coding are known, detailed description thereof will be omitted. However, as a restriction on the time hierarchy coding, an access unit (AU) having a higher hierarchy level than that of the access unit (AU) to be encoded ) Cannot be referred to.
By providing such a restriction, for example, in the example of FIG. 11, an access unit (AU) at hierarchical level 2 or lower (Temporal ID ≦ 2) refers to an access unit (AU) at hierarchical level 3 (Temporal ID = 3) at the time of decoding. Therefore, it is possible to decode an access unit (AU) at hierarchical level 2 or lower (Temporal ID ≦ 2) without decoding an access unit (AU) at hierarchical level 3.
In HEVC, temporal hierarchy coding is possible using a reference structure with a maximum hierarchy of up to 6.

FIG. 10 is an explanatory diagram showing the encoding order and display order of each picture encoded with the picture structure of FIG.
As shown in FIG. 10, if the access units at the hierarchy level 3 and the access units at the hierarchy level 2 or lower are encoded so as to alternate in the display order, all access units from the hierarchy level 0 to the hierarchy level 3 are When the display frame rate at the time of decoding is 2N (Hz), if only the access unit of the hierarchical level 2 or lower is decoded, it is possible to reproduce at the display frame rate N (Hz). Therefore, in the case of reproduction by a decoding device that supports a display frame rate of N (Hz) or lower, only an access unit having a hierarchical level of 2 or lower needs to be passed to the decoding device.

For example, in MMT, when a video bit stream configured as shown in FIG. 10 is multiplexed and distributed, only a set of access units composed of access units of hierarchical level 2 and lower and only access units of hierarchical level 3 are used. Different sets of identifiers can be assigned to the set of configured access units and distributed. In MMT, a set of access units assigned the same identifier is called an asset. It is also possible to transmit in different transmission formats for each asset, with the identifier of the asset consisting of access units of hierarchical level 2 or lower being A ₀ and the identifier of the asset consisting of access units of hierarchical level 3 being A ₁ . For example, it is also possible to transmit asset A ₀ by broadcasting and transmit asset A ₁ by communication.
In order to synchronize the presentation time between assets, MMT provides a descriptor that describes the presentation time of the first access unit in the NTP (Network Time Protocol) format in display order in GOP units. This descriptor can be multiplexed and transmitted. Also, if the assets are different, the presentation time of the first access unit can be transmitted for each asset, and even when multiple assets transmitted in different transmission formats are received at the receiving side, playback is performed with the presentation time synchronized (Presentation).

STD-B32 (Standard for digital broadcasting established by ARIB (Radio Industry Association))

  Since the conventional encoding apparatus is configured as described above, different assets are configured according to the hierarchical level of each access unit from the bitstream of the video encoded in time hierarchy as shown in FIG. When transmission is performed using a different transmission form, it is necessary to reconstruct a bitstream in the same order as the encoding order as shown in FIG. 10 based on the decoding timing of each access unit in the decoding device. Since the decoding time of each access unit cannot be transmitted, there is a problem that the bitstream cannot be reconfigured.

  The present invention has been made to solve the above-described problems, and even when a bit stream of video that has been time-hierarchically coded constitutes different assets according to the hierarchy level of each access unit and is transmitted, the decoding apparatus An encoding device, a decoding device, an encoding method, and a decoding method capable of reconstructing and decoding a bit stream encoded in time hierarchy based on the decoding timing of each access unit.

The encoding device according to the present invention is:
Video encoding means for encoding a video signal in units of video access units and outputting encoded data of the video signal;
A plurality of accesses capable of decoding all of the video signals of the one or more access units when the video signal of the one or more access units is encoded by the interframe predictive encoding method by the video encoding means. Control information including presentation time information indicating the presentation time of the first access unit in the presentation order and decoding time information indicating the decoding time of the first access unit in the encoding order is encoded for each GOP that is a set of units. Control information encoding means for outputting encoded data of control information in GOP units;
For each access unit of encoded data of the video signal output from the video encoding means, the difference between the decoding time of this access unit and the decoding time of the first access unit in the encoding order is encoded to obtain time information Time information encoding means for outputting the encoded data of
The encoded data of the video signal output from the video encoding means, the encoded data of time information output from the time information encoding means, and the encoded data of control information output from the control information encoding means. And multiplexing means for outputting a bit stream that is multiplexed encoded data.

  According to the present invention, even when a bit stream of video encoded in time hierarchy is configured and transmitted with different assets according to the hierarchy level of each access unit, the decoding apparatus uses the time hierarchy based on the decoding timing of each access unit. The encoded bit stream can be reconstructed and decoded.

It is a block diagram which shows the encoding apparatus by Embodiment 1 of this invention. It is a flowchart which shows the processing content (encoding method) of the encoding apparatus by Embodiment 1 of this invention. It is a block diagram which shows the decoding apparatus by Embodiment 1 of this invention. It is a flowchart which shows the processing content (decoding method) of the decoding apparatus by Embodiment 1 of this invention. It is explanatory drawing which shows the outline | summary of the coding data in the case of transmitting a bit stream by MMT. It is explanatory drawing which shows the structural example of MPU. It is explanatory drawing which shows a HEVC picture structure descriptor. It is explanatory drawing which shows the example of time hierarchy encoding in HEVC. It is explanatory drawing which shows an example of a picture structure. It is explanatory drawing which shows the encoding order and presentation order of each picture encoded with the picture structure of FIG. It is explanatory drawing which shows the structure of PA message. It is explanatory drawing which shows an example of the bit stream before separation, and the bit stream after separation.

Embodiment 1 FIG.
FIG. 1 is a block diagram showing an encoding apparatus according to Embodiment 1 of the present invention.
In FIG. 1, when a digital audio signal is given, the audio encoding unit 1 encodes the audio signal in units of audio access units (AU) by a method such as MPEG-4 audio. The audio stream that is the encoded data is generated, and the process for encoding the metadata related to the audio stream is performed. The encoded access unit presentation time (PTS) is output to the voice MMTP packet generator 8.
The audio MMTP payload generation unit 2 performs a process of generating an audio MMTP payload including the metadata encoded by the audio encoding unit 1 and encoded data of an audio signal in units of access units (AU).

When a digital video signal is given, the HEVC encoding unit 3 encodes the video signal by the HEVC method for each video access unit (AU), and generates a video stream that is encoded data of the video signal. At the same time, a process for encoding the metadata related to the video stream is performed.
The video MMTP payload generation unit 4 performs a process of generating a video MMTP payload composed of metadata encoded by the HEVC encoding unit 3 and encoded data of a video signal in units of access units (AU). The HEVC encoding unit 3 and the video MMTP payload generation unit 4 constitute video encoding means.

The control information encoding unit 5 encodes control information called PA message defined by MMT as control information related to the audio stream generated by the audio encoding unit 1 and the video stream generated by the HEVC encoding unit 3. Perform the process.
FIG. 11 shows the configuration of the PA message. The PA message is composed of one or more tables.
In one table included in the PA message, information on one or more video components (video streams) and audio components (audio streams) constituting one program (referred to as a package in MMT) is described. In MMT, video components and audio components are called assets.

Specifically, an asset ID for identifying an asset, an asset type for identifying an asset type (a type such as a video stream in HEVC format or an audio stream in MPEG-4 audio format), encoded data and metadata of each asset Various descriptors for describing information on asset acquisition destinations, such as packet IDs indicating stored MMTP packets or IP addresses when IP is distributed, and meta information on each asset are the number of assets constituting the package. Only included in the table.
In the descriptor, the presentation time (display time) of the first access unit (AU) in the presentation order (display order) among the access units (AU) constituting the MPU (Media Processing Unit) of each asset is displayed. In addition to the descriptors defined in the MMT standard, such as the MPU time stamp descriptor (presentation time information) to be shown, the user can also define a new descriptor independently. Includes children.

  The MPU is composed of one or more access units (AU), and is a unit that can perform video and audio decoding processing by the MPU alone. Also, the MPU decodes all of the video signals of the one or more access units (AU) when the video signals of the one or more access units (AU) are encoded by the inter-frame predictive coding method. It becomes the same unit as GOP which is a set of a plurality of access units (AU).

  The MPU time information descriptor includes a code indicating information (timescale) indicating a unit describing time information such as a decoding time (DTS) and a presentation time (PTS), and an access unit (AU) constituting the MPU. Whether the information (initial_presentation_time_delay) for calculating the decoding time of the head access unit (AU) in the order of generation and the information for calculating the decoding time and presentation time of each access unit constituting the MPU are encoded Flags indicating whether or not (presentation_time_offset_present_flag, decoding_time_offset_present_flag), and information indicating the code length when encoding the information for calculating the decoding time and the presentation time of each access unit Information (time_offset_length_minus1) and the like are described.

The control MMTP payload generation unit 6 performs a process of generating a control MMTP payload composed of encoded data of control information encoded by the control information encoding unit 5.
Note that a control information encoding unit includes a part of the control information encoding unit 5 and the control MMTP payload generation unit 6. The other part of the control information encoding unit 5 constitutes time information encoding means.

The video MMTP packet generation unit 9 adds a predetermined MMTP header to the video MMTP payload generated by the video MMTP payload generation unit 4 to generate a video MMTP packet constituting a bit stream. The MMTP header is composed of an essential header including information to be encoded as essential and an extension header including information to be encoded as an option. The essential header includes a packet ID assigned according to the type of encoded data included in the MMTP payload.
The extension header has a flag value indicating whether to encode information (presentation time information and decoding time information) for calculating a presentation time and a decoding time for each access unit of encoded data included in the MMTP payload. Accordingly, presentation time information (presentation_time_offset) and decoding time information (decoding_time_offset) are included.

  The voice MMTP packet generator 8 adds a predetermined MMTP header to the voice MMTP payload generated by the voice MMTP payload generator 2 to generate voice MMTP packets constituting a bit stream. The MMTP header is composed of an essential header including information to be encoded as essential and an extension header including information to be encoded as an option. The contents of the extension header are the same as the extension header encoded by the video MMTP packet generator.

  The control MMTP packet generation unit 10 adds a predetermined MMTP header to the control MMTP payload generated by the control MMTP payload generation unit 6 and generates a control MMTP packet that forms a bit stream.

The MMTP packet multiplexing unit 7 combines the audio MMTP packet generated by the audio MMTP packet generation unit, the control MMTP packet generated by the control MMTP packet generation unit, and the video MMTP packet generated by the video MMTP packet generation unit. A process of multiplexing and forming a bitstream is performed.
The MMTP packet multiplexing unit 7 can also configure a different bit stream for each asset. For example, as shown in FIG. 9, a bitstream 1 composed of an MMTP packet of an asset including an access unit of hierarchical level 2 or lower of a video bitstream encoded in time hierarchy, and an MMTP of an asset including an access unit of hierarchical level 3 It is also possible to send each bit stream in a different transmission form as the bit stream 2 composed of packets.
The MMTP packet multiplexing unit 7 constitutes multiplexing means.

In the example of FIG. 1, an audio encoding unit 1, an audio MMTP payload generating unit 2, an HEVC encoding unit 3, a video MMTP payload generating unit 4, a control information encoding unit 5, and a control MMTP payload that are components of the encoding device. It is assumed that each of the generation unit 6 and the control MMTP packet generation unit 10 is configured by dedicated hardware (for example, a semiconductor integrated circuit on which a CPU is mounted, or a one-chip microcomputer). The encoding device may be configured by a computer.
When the encoding apparatus is configured by a computer, an audio encoding unit 1, an audio MMTP payload generating unit 2, an HEVC encoding unit 3, a video MMTP payload generating unit 4, a control information encoding unit 5, a control MMTP payload generating unit 6, and A program describing the processing contents of the control MMTP packet generation unit 10 and the like may be stored in the memory of a computer, and the CPU of the computer may execute the program stored in the memory.

  FIG. 2 is a flowchart showing the processing contents (encoding method) of the encoding apparatus according to Embodiment 1 of the present invention.

3 is a block diagram showing a decoding apparatus according to Embodiment 1 of the present invention.
3, the MMTP packet analysis unit 12 includes one or more assets including one or more assets output from the encoding device (the encoding device in FIG. 1 or the encoding device corresponding to the encoding device in FIG. 1). The bitstream is input. The MMTP packet analysis unit 12 analyzes the MMTP header of the MMTP packet constituting the bit stream, acquires the packet ID included in the MMTP header, and encodes the packet ID included in the MMTP payload. If the digitized data indicates control information (PA message), a control MMTP payload that is an MMTP payload included in the MMTP packet is output to the control MMTP payload processing unit 13.

The control MMTP payload processing unit 13 performs a decoding process on the encoded data included in the control MMTP payload output from the MMTP packet analysis unit 12 and decodes the PA message as control information.
Further, the control MMTP payload processing unit 13 receives a packet ID indicating an MMTP packet storing information on assets constituting the package, encoded data and metadata of each asset from the table described in the PA message, or IP distribution. Information related to the asset acquisition destination, such as the IP address. Information regarding the packet ID and asset acquisition destination is output to the MMTP packet analysis unit.
In addition, the control MMTP payload processing unit 13 decodes the MPU time stamp descriptor and the MPU time information descriptor related to the assets constituting the package from the table described in the PA message.

  The MMTP packet analysis unit 12 acquires the packet ID included in the MMTP header, collates the acquired packet ID with the packet ID of each asset output from the control MMTP payload processing unit 13, and the packet ID is MMTP If the encoded data included in the payload indicates that it is an audio signal or a video signal, a process of outputting the MMTP packet to the asset separation unit 14 is performed.

  Further, the MMTP packet analysis unit 12 is information for calculating the decoding time and the presentation time of each access unit constituting the MPU described in the MPU time information descriptor decoded by the control MMTP payload processing unit 13. Depending on the value of a flag (presentation_time_offset_present_flag, decoding_time_offset_present_flag) indicating whether or not is encoded, presentation time information (presentation_time_offset) and decoding time information (decoding_time descriptor) are decoded from the MMTP extension header. Described in the presentation time and MPU time information descriptor of the first access unit (AU) in the order of presentation Each access from the decoding time of the head access unit (AU) in the coding order obtained by decoding the information (initial_presentation_time_delay) for calculating the decoding time of the head access unit (AU) in the coded order Processing for calculating the presentation time and decoding time of the unit (AU) is performed. The calculated presentation time and decoding time are output to the audio MMTP payload processing unit 15 and the video MMTP payload processing unit 19 according to the type of encoded data included in the access unit.

  The asset separation unit 14 refers to the asset ID, asset type, and packet ID described in the PA message table decoded by the control MMTP payload processing unit 13 and is included in the MMTP packet output from the MMTP packet analysis unit 12. Whether the MMTP payload is an audio MMTP payload or a video MMTP payload. If the MMTP payload is an audio MMTP payload, the audio MMTP payload included in the MMTP packet is extracted and the audio MMTP payload is extracted. Is output to the audio MMTP payload processing unit 15. If it is a video MMTP payload, the video MMTP payload included in the MMTP packet is extracted and the video MMTP payload is output to the video MMTP payload processing unit 19. Process to implement that.

The audio MMTP payload processing unit 15 reconstructs the MFU (Media Fragment Unit) or MPU of the audio stream from the audio MMTP payload output from the asset separation unit 14, so that the audio stream decoding unit 17 can decode the audio stream. A process of generating an audio elementary stream (audio ES) and storing the audio ES in the audio ES buffer 16 is performed. The MFU is a smaller unit than the MPU, and one access unit (AU) or one NAL unit can be defined as one MFU.
In addition, the audio MMTP payload processing unit 15 extracts the metadata regarding the audio stream included in the audio MMTP payload output from the asset separation unit 14 and stores the metadata in the audio ES buffer 16. The audio ES buffer 16 is a memory that temporarily stores the audio ES and metadata.

When the DTS (decoding time) of each access unit (AU) is reached, the audio stream decoding unit 17 extracts the audio ES from the audio ES buffer 16, decodes the audio signal of the access unit (AU), and decodes the decoded audio. Processing for storing the signal and PTS (presentation time) in the audio data buffer 18 is performed.
The audio data buffer 18 is a memory that temporarily stores the audio signal decoded by the audio stream decoding unit 17 and the PTS (presentation time).

The video MMTP payload processing unit 19 reconstructs the video stream MFU or MPU from the video MMTP payload output from the asset separation unit 14, thereby enabling the HEVC elementary stream (HEVC) in a format decodable by the HEVCES decoding unit 21 in the subsequent stage. ES) is generated, and the HEVC elementary stream is stored in the HEVCES buffer 20.
In addition, the video MMTP payload processing unit 19 extracts metadata related to the video stream included in the video MMTP payload output from the asset separation unit 14 and stores the metadata in the HEVCES buffer 20.
The HEVCES buffer 20 is a memory that temporarily stores HEVC elementary streams and metadata.

When the HEVCES decoding unit 21 reaches the DTS (decoding time) of each access unit (AU), the HEVCES decoding unit 21 extracts the HEVC elementary stream from the HEVCES buffer 20, decodes the video signal of the access unit (AU), and decodes the decoded video signal. The decoded image and the PTS (presentation time) are stored in the decoded image buffer 22.
The decoded image buffer 22 is a memory that temporarily stores the decoded image and PTS (presentation time) of each access unit (AU) decoded by the HEVCES decoding unit 21.
The video MMTP payload processing unit 19, the HEVCES buffer 20, the HEVCES decoding unit 21, and the decoded image buffer 22 constitute video decoding means.

In the example of FIG. 3, the MMTP packet analysis unit 12, the control MMTP payload processing unit 13, the asset separation unit 14, the audio MMTP payload processing unit 15, the audio ES buffer 16, the audio stream decoding unit 17, and the audio, which are components of the decoding device. Each of the data buffer 18, the video MMTP payload processing unit 19, the HEVCES buffer 20, the HEVCES decoding unit 21, and the decoded image buffer 22 has dedicated hardware (other than the buffer, for example, a semiconductor integrated circuit on which a CPU is mounted, or However, the decoding device may be constituted by a computer.
When the decoding apparatus is configured by a computer, the audio ES buffer 16, the audio data buffer 18, the HEVCES buffer 20, and the decoded image buffer 22 are configured on the internal memory or external memory of the computer, and the MMTP packet analysis unit 12, the control MMTP payload A program describing processing contents of the processing unit 13, the asset separation unit 14, the audio MMTP payload processing unit 15, the audio stream decoding unit 17, the video MMTP payload processing unit 19 and the HEVCES decoding unit 21 is stored in a memory of a computer. The CPU of the computer may execute a program stored in the memory.

  FIG. 4 is a flowchart showing the processing contents (decoding method) of the decoding apparatus according to Embodiment 1 of the present invention.

Next, the operation will be described.
The processing contents of the first encoding device will be described.
When a digital audio signal is given, the audio encoding unit 1 encodes the audio signal in units of audio access units (AU) by a method such as MPEG-4 audio, and encodes the audio signal. An audio stream that is data is generated, and metadata relating to the audio stream is encoded (step ST1 in FIG. 2).
When a digital video signal is given, the HEVC encoding unit 3 encodes the video signal by the HEVC method for each video access unit (AU), and generates a video stream that is encoded data of the video signal. At the same time, metadata about the video stream is encoded (step ST2).

Here, FIG. 5 is an explanatory diagram showing an outline of encoded data when a bit stream is transmitted by MMT.
In FIG. 5, an access unit (AU) is a unit that includes encoded data necessary for decoding one picture if it is a video, and is composed of one or more samples that are an encoding unit if it is audio. Is the frame to be played.
The NAL unit is a HEVC encoding unit, and one access unit (AU) is composed of one or more NAL units.
The MPU is composed of one or more access units, and is a unit that can perform video and audio decoding processing by the MPU alone. Also, the MPU decodes all of the video signals of the one or more access units (AU) when the video signals of the one or more access units (AU) are encoded by the inter-frame predictive coding method. It becomes the same unit as GOP which is a set of a plurality of access units (AU).
The MFU is a smaller unit than the MPU, and one access unit (AU) or one NAL unit can be defined as one MFU.

FIG. 6 is an explanatory diagram showing a configuration example of the MPU.
In FIG. 6, MPU metadata describes metadata related to the MPU. The MPU metadata may not be encoded.
Movie fragment metadata (MF metadata) describes metadata accompanying encoded data (sample data) of one access unit (AU). For example, when the encoded data of the access unit (AU) is stored in a file format, for each access unit (AU), the address where the encoded data is stored, the data length of the encoded data, the access unit (AU) ) Is included. Movie fragment metadata may not be encoded.
MPU metadata, movie fragment metadata, MFU and MMT control information are transmitted as MMTP packets. The MMTP packet is composed of an MMTP header and an MMTP payload.

When the audio MMTP payload generation unit 2 receives the encoded data of the metadata (MPU metadata, MF meta, etc.) and the encoded data of the audio signal in units of access units (AU) from the audio encoding unit 1, An audio MMTP payload including encoded data of MPU metadata in units, encoded data of MF meta in units of access units (AU), and encoded data (sample data) of audio signals is generated (step ST3).
When receiving the encoded data of metadata (MPU metadata, MF meta, etc.) and the encoded data of the video signal in units of access units (AU) from the HEVC encoder 3, the video MMTP payload generator 4 receives the MPU. A video MMTP payload composed of encoded data of MPU metadata in units, encoded data of MF meta in units of access units (AU), and encoded data (sample data) of video signals is generated (step ST4).

The control information encoding unit 5 encodes the control information regarding the audio stream generated by the audio encoding unit 1 and the video stream generated by the HEVC encoding unit 3 (step ST5).
As control information related to an audio stream and a video stream, for example, a PA message or an MPU time information descriptor defined by MMT is encoded.
As described above, in the PA message, information on one or more video components (video streams) and audio components (audio streams) constituting one program (referred to as a package in MMT) is described.
That is, the PA message includes an asset ID for identifying an asset (video stream, audio stream) generated by the audio encoding unit 1 and HEVC encoding unit 3, an asset type for identifying the asset type, and an MPU for each asset. MMTP packet storing MPU time stamp descriptor indicating the presentation time of the first access unit (AU) in the order of presentation among the configured access units (AU), encoded data and metadata of each asset A packet ID or the like indicating is described.

FIG. 7 is an explanatory diagram showing the MPU time information descriptor.
In the MPU time information descriptor, as shown in FIG. 7, a sequence number (mpu_sequence_number) for identifying which MPU-related information is included, and the decoding time and presentation of the MPU head access unit in coding order The time difference (initial_presentation_time_delay) of the presentation time of the MPU head access unit in order, the unit of presentation time information encoded in the unit of access unit and the display time information (timescale) (1 / timescale second), the unit of access unit A flag (presentation_time_offset_present_flag) indicating whether or not the presentation time information is encoded, and a flag (encoding or not decoding time information is encoded in units of access units) ecoding_time_offset_present_flag), the code length of the presentation time information and display time information encoding in units of access units (time_offset_length_minus1), etc. are described. Note that timescale, presentation_time_offset_present_flag, decoding_time_offset_present_flag, and time_offset_length_minus1 do not have to be encoded if fixed values are always used.

When receiving the encoded data of the control information from the control information encoding unit 5, the control MMTP payload generating unit 6 generates a control MMTP payload including the encoded data of the control information (step ST6).
The video MMTP packet generation unit adds a predetermined MMTP header to the video MMTP payload generated by the video MMTP payload generation unit to generate a video MMTP packet constituting a bit stream. The MMTP header is composed of an essential header including information to be encoded as essential and an extension header including information to be encoded as an option. The essential header includes a packet ID assigned according to the type of encoded data included in the MMTP payload.

The extension header has a flag value indicating whether to encode information (presentation time information and decoding time information) for calculating a presentation time and a decoding time for each access unit of encoded data included in the MMTP payload. Accordingly, presentation time information (presentation_time_offset) and decoding time information (decoding_time_offset) are included.
The presentation time information (presentation_time_offset) is a difference between the presentation time of the access unit of the encoded data included in the MMTP payload and the presentation time of the MPU head access unit in the presentation order.
The decoding time information (decoding_time_offset) is the difference between the decoding time of the access unit of the encoded data included in the MMTP payload and the decoding time of the MPU head access unit in the encoding order.
The presentation time information (presentation_time_offset) may be encoded as a difference from the decoding time of the access unit calculated by decoding the decoding time information (decoding_time_offset).

  The voice MMTP packet generator 8 adds a predetermined MMTP header to the voice MMTP payload generated by the voice MMTP payload generator 2 to generate voice MMTP packets constituting a bit stream. The MMTP header is composed of an essential header including information to be encoded as essential and an extension header including information to be encoded as an option.

The control MMTP packet generation unit 10 adds a predetermined MMTP header to the control MMTP payload generated by the control MMTP payload generation unit 6 and generates a control MMTP packet that forms a bit stream.
When this MMTP packet is generated, a predetermined MMTP header is added, and this MMTP header includes a packet ID assigned according to the type of encoded data included in the MMTP payload.

  The MMTP packet multiplexing unit 7 includes a voice MMTP packet generated by the voice MMTP packet generation unit 8, a control MMTP packet generated by the control MMTP packet generation unit 10, and a video MMTP generated by the video MMTP packet generation unit 9. A bit stream is multiplexed to form a bit stream. (Step ST7)

Next, processing contents of the decoding device will be described.
The MMTP packet analysis unit 12 inputs one or more bit streams including one or more assets output from the encoding device (the encoding device in FIG. 1 or the encoding device corresponding to the encoding device in FIG. 1). Then, the MMTP header of the MMTP packet constituting the bit stream is analyzed, and the packet ID included in the MMTP header is obtained.
The MMTP packet analysis unit 12 includes the packet ID included in the MMTP packet if the encoded data included in the MMTP payload indicates control information (PA message, HEVC picture structure descriptor). The control MMTP payload, which is the MMTP payload being stored, is output to the control MMTP payload processing unit 13.

On the other hand, if the packet ID indicates that the encoded data included in the MMTP payload is an audio signal or a video signal, the MMTP packet is output to the asset separation unit 14.
Further, the MMTP packet analysis unit 12 is information for calculating the decoding time and the presentation time of each access unit constituting the MPU described in the MPU time information descriptor decoded by the control MMTP payload processing unit 13. Depending on the value of a flag (presentation_time_offset_present_flag, decoding_time_offset_present_flag) indicating whether or not is encoded, presentation time information (presentation_time_offset) and decoding time information (decoding_time descriptor) are decoded from the MMTP extension header. Described in the presentation time and MPU time information descriptor of the first access unit (AU) in the order of presentation It is performed and a decoding time of the beginning of the access unit (AU) a process of calculating a presentation time and decoding time of each access unit (AU) in coding order is. The calculated presentation time and decoding time are output to the audio MMTP payload processing unit 15 and the video MMTP payload processing unit 19 according to the type of encoded data included in the access unit.

The control MMTP payload processing unit 13 performs a decoding process on the encoded data included in the control MMTP payload output from the MMTP packet analysis unit 12 and decodes the PA message as control information.
Further, the control MMTP payload processing unit 13 receives a packet ID indicating an MMTP packet storing information on assets constituting the package, encoded data and metadata of each asset from the table described in the PA message, or IP distribution. Information related to the asset acquisition destination, such as the IP address. Information regarding the packet ID and asset acquisition destination is output to the MMTP packet analysis unit.
In addition, the control MMTP payload processing unit 13 decodes the MPU time stamp descriptor and the MPU time information descriptor related to the assets constituting the package from the table described in the PA message.

  When the control MMTP payload processing unit 13 decrypts the PA message, the asset separation unit 14 refers to the asset ID, asset type, and packet ID described in the PA message table, and is output from the MMTP packet analysis unit 12. Whether the MMTP payload included in the MMTP packet is an audio MMTP payload or a video MMTP payload is specified.

If the MMTP payload included in the MMTP packet output from the MMTP packet analysis unit 12 is an audio MMTP payload, the asset separation unit 14 extracts the audio MMTP payload included in the MMTP packet and extracts the audio The MMTP payload is output to the voice MMTP payload processing unit 15.
If the MMTP payload included in the MMTP packet output from the MMTP packet analysis unit 12 is a video MMTP payload, the asset separation unit 14 extracts the video MMTP payload included in the MMTP packet and extracts the video The MMTP payload is output to the video MMTP payload processing unit 19.

When the audio MMTP payload processing unit 15 receives the audio MMTP payload from the asset separation unit 14, the audio MMTP payload processing unit 15 reconstructs the MFU or MPU of the audio stream from the audio MMTP payload, so that the audio stream decoding unit 17 can decode the audio stream. Audio elementary stream (audio ES) is generated, and the audio ES is stored in the audio ES buffer 16.
Since the process itself for generating the audio ES from the audio MMTP payload is a known technique, detailed description thereof is omitted.
In addition, the audio MMTP payload processing unit 15 extracts metadata about the audio stream included in the audio MMTP payload output from the asset separation unit 14 and stores the metadata in the audio ES buffer 16.

The audio stream decoding unit 17 refers to the DTS decoded by the MMTP packet analysis unit, grasps the decoding time of each access unit (AU), and at the decoding time of each access unit (AU), the audio ES buffer 16 The voice ES is taken out from the voice signal, the voice signal of the access unit (AU) is decoded, and the decoded voice signal and the PTS (presentation time) decoded by the MMTP packet analysis unit are stored in the voice data buffer 18.
Thus, if an external playback device (not shown) takes out the audio signal and the PTS (presentation time) stored in the audio data buffer 18, the audio signal can be reproduced at the presentation time.

When the video MMTP payload processing unit 19 receives the video MMTP payload from the asset separation unit 14, the video MMTP payload processing unit 19 reconstructs the MFU or MPU of the video stream from the video MMTP payload, so that the HEVCES decoding unit 21 in the subsequent stage can decode it. A HEVC elementary stream (HEVC ES) is generated, and the HEVC elementary stream is stored in the HEVCES buffer 20.
Since the process itself for generating the HEVC elementary stream from the video MMTP payload is a known technique, detailed description thereof is omitted.
In addition, the video MMTP payload processing unit 19 extracts metadata regarding the video stream included in the video MMTP payload output from the asset separation unit 14, and stores the metadata in the HEVCES buffer 20.

The HEVCES decoding unit 21 refers to the DTS decoded by the MMTP packet analysis unit to grasp the decoding time of each access unit (AU). When the decoding time of each access unit (AU) comes, the HEVCES buffer 20 reads the HEVC The elementary stream is extracted, the video signal of the access unit (AU) is decoded, and the decoded image that is the decoded video signal and the PTS (presentation time) decoded by the MMTP packet analysis unit are stored in the decoded image buffer 22. To do.
Thus, if an external playback device (not shown) takes out the decoded image and the PTS (presentation time) stored in the decoded image buffer 22, the decoded image can be reproduced at the presentation time.
A temporal bitstream-coded video bitstream composed of access units with TemporalID 0 to M, a bitstream composed of access units with TemporalID 0 to (M-1), and an access unit with TemporalID M The processing when each bit stream is transmitted using different transmission paths will be described.
FIG. 12 shows an example of the bit stream before separation and the bit stream after separation.

  The MMTP packet analysis unit 12 outputs the control MMTP payload included in the MMTP packet constituting the input bit stream to the control MMTP payload processing unit 13, and is included in the MMTP packet constituting the bit stream. The recorded audio MMTP payload or video MMTP payload is output to the asset separation unit 14.

Upon receiving the control MMTP payload from the MMTP packet analysis unit 12, the control MMTP payload processing unit 13 performs a decoding process on the encoded data included in the control MMTP payload and decodes the PA message that is the control information (Step ST15).
The control MMTP payload processing unit 13 encodes the video bitstream based on the information related to the asset described in the PA message, etc., and has two or more assets (for example, asset 1, asset, etc.) according to the temporal hierarchy level (TemporalID). 2) to acquire information such as that the asset is acquired from a different transmission path for each asset, and the dependency relationship between assets (the asset 2 is dependent on the asset 1).

  When the asset separation unit 14 receives the MMTP packet from the MMTP packet analysis unit 12, the asset separation unit 14 refers to the packet ID based on the information about the video asset described in the PA message, and the video MMTP payload included in the MMTP packet. Is output to the video MMTP payload processing unit 19. For example, when the video asset is composed of asset 1 and asset 2, the video MMTP payload related to each asset is output to the video MMTP payload processing unit.

  When the video MMTP payload processing unit 19 receives the video MMTP payloads related to two or more video assets from the asset separation unit 14, the video MMTP payload processing unit 19 generates a HEVC elementary stream from each video MMTP payload, and the HEVC elementary stream is converted into the HEVCES buffer 20. And the decoding time and presentation time of the access unit included in each video MMTP payload are stored in the HEVCES buffer 20 (step ST19).

  The HEVCES decoding unit 21 compares the DTS of the access unit of the asset 1 and the DTS of the access unit of the asset 2 based on, for example, the dependency relationship between the asset 1 and the asset 2, thereby encoding the HEVC elementary stream before separation. The HEVC elementary stream separated and input into different bit streams can be extracted from the HEVCES buffer 20 at the correct decoding time, and the video signal of the access unit (AU) can be decoded.

  As apparent from the above, according to the first embodiment, when video signals of one or more access units (AU) are time-hierarchically encoded, different assets are configured according to the hierarchical level of each access unit. In this case, the MPU time information descriptor is encoded in the unit of MPU constituting each asset, and the presentation time and the decoding time are calculated in the access unit unit of the encoded data included in the MMTP payload included in the MPU time information descriptor. The presentation time information (presentation_time_offset) and the decoding time information (in accordance with the value of a flag indicating whether or not to encode information (presentation time information and decoding time information) for each access unit constituting the MPU decoding_time_offset) is encoded so that the time hierarchy Even when different encoded video bitstreams are configured and transmitted according to the hierarchical level of each access unit, the decoding apparatus re-generates the time-stream encoded bitstream based on the decoding timing of each access unit. There is an effect that an encoding device and a decoding device that can be configured and decoded are obtained.

  In the present invention, within the scope of the invention, free combination of the embodiments, modification of any component of the embodiment, or omission of any component in the embodiment is possible.

  DESCRIPTION OF SYMBOLS 1 Audio encoding part, 2 Audio MMTP payload production | generation part, 3 HEVC encoding part (video coding means), 4 Video MMTP payload production | generation part (video coding means), 5 Control information coding part (Control information coding means) , Time information encoding means), 6 control MMTP payload generation section (control information encoding means), 7 MMTP packet multiplexing section (multiplexing means), 8 audio MMTP packet generation section, 9 video MMTP packet generation section, 10 control MMTP packet generation unit, 12 MMTP packet analysis unit, 13 control MMTP payload processing unit (presentation time calculation means), 14 asset separation unit, 15 audio MMTP payload processing unit, 16 audio ES buffer, 17 audio stream decoding unit, 18 audio data Buffer, 19 Video MMTP payload processing unit (Video decoding means) 20 HEVCES buffer (video decoding means), 21 HEVCES decoder (video decoding means), 22 the decoded picture buffer (video decoding means).

Claims (1)

Video encoding means for encoding a video signal in units of video access units and outputting encoded data of the video signal;
A plurality of accesses capable of decoding all of the video signals of the one or more access units when the video signal of the one or more access units is encoded by the interframe predictive encoding method by the video encoding means. Control information including presentation time information indicating the presentation time of the first access unit in the presentation order and decoding time information indicating the decoding time of the first access unit in the encoding order is encoded for each GOP that is a set of units. Control information encoding means for outputting encoded data of control information in GOP units;
For each access unit of encoded data of the video signal output from the video encoding means, the difference between the decoding time of this access unit and the decoding time of the first access unit in the encoding order is encoded to obtain time information Time information encoding means for outputting the encoded data of
The encoded data of the video signal output from the video encoding means, the encoded data of time information output from the time information encoding means, and the encoded data of control information output from the control information encoding means. An encoding apparatus comprising: multiplexing means for multiplexing and outputting a bit stream that is encoded data after multiplexing.

Images