KR102142735B1 - Transmitting device, transmitting method, receiving device and receiving method - Google Patents

Abstract

When the image data of the ultra high-definition service is transmitted without scalable encoding, it is easy to acquire image data of a resolution suitable for its display capability in a receiver that does not support this ultra-high-definition service. A container of a predetermined format having a video stream containing coded image data is transmitted. In the video stream, auxiliary information for downscaling of spatial and/or temporal resolution of image data is inserted. For example, the auxiliary information is information indicating a precision limit of a motion vector included in coded image data. Further, for example, the auxiliary information is information identifying a picture to be selected when downscaling the temporal resolution at a predetermined rate.

Description

Transmitting device, transmitting method, receiving device and receiving method {TRANSMITTING DEVICE, TRANSMITTING METHOD, RECEIVING DEVICE AND RECEIVING METHOD}

The present invention relates to a transmission device, a transmission method, a reception device and a reception method, and more particularly, to a transmission device or the like for transmitting image data of spatial or temporal ultra-high resolution images.

For example, in addition to HD images having an effective pixel count of 1920x1080, services of spatial ultra-high-resolution images such as 4K and 8K with double and four times the effective pixel count are considered (for example, patents). Reference 1). In addition, for example, in addition to images having a frame frequency of 30 Hz, services of temporal ultra-high resolution images such as frame frequencies of 60 Hz and 120 Hz are considered. In addition, these ultra high resolution image services are appropriately referred to as ultra high definition services.

Japanese Patent Publication No. 2011-057069

When the image data of the ultra-high-definition service described above is scalable-encoded, even in a receiver that does not support this ultra-high-definition service, it is possible to easily acquire image data of a resolution suitable for its display capability. However, when the image data of the ultra-high-definition service is not scalable-encoded, it is difficult for a receiver that does not support this ultra-high-definition service to acquire image data of a resolution suitable for its display capability.

An object of the present invention is to facilitate acquisition of image data of a resolution suitable for its display capability in a receiver that does not support this ultra-high-definition service when the image data of the ultra-high-definition service is transmitted without scalable encoding. Having

The concept of the present invention,

A transmitter for transmitting a container of a predetermined format having a video stream containing encoded image data;

An auxiliary information inserting unit for inserting auxiliary information for downscaling spatial and/or temporal resolution of the image data into the video stream.

It is in a transmission device having a.

In the present invention, a container of a predetermined format having a video stream containing encoded image data is transmitted by the transmitter. The encoded image data is encoded by MPEG4-AVC (MVC), MPEG2video, or HEVC, for example. The container may be, for example, a transport stream (MPEG-2 TS) employed in the digital broadcasting standard. Further, the container may be, for example, an MP4 used for distribution on the Internet or the like, or a container of other format.

By the auxiliary information insertion unit, auxiliary information for downscaling of spatial and/or temporal resolution of image data is inserted into the video stream. For example, the auxiliary information may be information indicating a precision limit of a motion vector included in coded image data. Further, for example, the auxiliary information may be information identifying a picture to be selected when downscaling the temporal resolution at a predetermined rate.

In this way, in the present invention, auxiliary information for downscaling of spatial and/or temporal resolution of image data is inserted into a video stream. Therefore, when the image data of the ultra high-definition service is transmitted without scalable encoding, it is possible to easily acquire image data of a resolution suitable for one's own display capability in a receiver that does not support this ultra-high-definition service.

Further, in the present invention, for example, an identification information inserting unit for inserting identification information indicating that auxiliary information is inserted into a video stream may be further provided in a layer of a container. In this case, the receiver can know that the auxiliary information is inserted into the video stream without decoding the video stream, and can extract the auxiliary information appropriately.

For example, downscaling information indicating a possible ratio in downscaling of spatial and/or temporal resolution may be added to this identification information. Further, spatial and/or temporal resolution information of image data included in the video stream may be added to this identification information. Further, for example, the container is a transport stream, and the identification information inserting unit may insert identification information into a descriptor under the management of a video elementary loop of a program map table included in the transport stream. .

Further, in the present invention, for example, a resolution information inserting unit that inserts spatial and/or temporal resolution information of image data included in a video stream into a container layer may be further provided. In this case, in the case where the image data of the ultra high definition service is transmitted without scalable encoding, in a receiver that does not support this ultra high definition service, it is possible to determine the content of the downscaling process based on this resolution information. Becomes

For example, identification information for identifying whether the video stream is supported for a low-capacity decoder that does not correspond to the spatial and/or temporal resolution of image data may be added to the resolution information. Further, for example, the container is a transport stream, and the resolution information inserting unit may insert the resolution information into a descriptor under the management of the event information table included in the transport stream.

In addition, another concept of the present invention,

A transmitter for transmitting a container of a predetermined format having a video stream containing encoded image data;

Identification information inserting unit for inserting identification information in the layer of the container so that the high-definition service by the video stream can be identified by at least a program unit.

It is in a transmission device having a.

In the present invention, a container of a predetermined format having a video stream containing image data is transmitted by the transmitter. The container may be, for example, a transport stream (MPEG-2 TS) adopted as a digital broadcasting standard. Further, the container may be, for example, an MP4 used for distribution on the Internet or the like, or a container of a format other than that.

By the identification information inserting unit, identification information is inserted into the layer of the container so that the ultra-high-definition service by the video stream can be identified at least in units of programs. For example, the identification information may include spatial and/or temporal resolution information of image data. For example, the container is a transport stream, and the identification information inserting unit may insert identification information into a descriptor under the management of the event information table included in the transport stream.

As described above, in the present invention, identification information is inserted into a layer of a container so that a high-definition service by a video stream can be identified at least in units of programs. Therefore, in the receiver, the ultra-high-definition service can be easily identified, and whether or not a downscaling process of spatial and/or temporal resolution is required and compared with its display ability, and the ratio can be appropriately and immediately determined. have.

Further, in the present invention, for example, the identification information is added to the video stream so that support information indicating whether or not support for a low-capacity decoder that does not correspond to the spatial and/or temporal resolution of the image data is provided. You may work. In this case, the receiver can easily determine whether the video stream has support for a low-capacity decoder, for example, insertion of auxiliary information for downscaling of spatial and/or temporal resolution.

In addition, another concept of the present invention,

A receiver for receiving a video stream including encoded image data;

In the video stream, auxiliary information for downscaling of spatial and/or temporal resolution of the image data is inserted,

A processing unit that performs downscaling of spatial and/or temporal resolution on the encoded image data based on the auxiliary information to obtain display image data of a desired resolution.

It is in a receiving device further comprising.

In the present invention, a video stream including coded image data is received by the receiver. In this video stream, auxiliary information for downscaling of spatial and/or temporal resolution of image data is inserted. Then, the processing unit performs downscaling processing of spatial and/or temporal resolution on the encoded image data based on the auxiliary information to obtain display image data of a desired resolution.

In this way, in the present invention, based on the auxiliary information inserted in the video stream, down-scaling processing of spatial and/or temporal resolution is performed on the encoded image data to obtain display image data of a desired resolution. Therefore, the load of the downscaling process can be reduced.

In addition, in the present invention, for example, the receiving unit receives a container of a predetermined format including a video stream, and the layer of the container indicates a possible ratio in downscaling of spatial and/or temporal resolution. The scaling information is inserted, and the processing unit may control the downscaling process for obtaining display image data based on the downscaling information.

In addition, in the present invention, for example, the receiving unit receives a container of a predetermined format including a video stream, and spatial and/or temporal resolution information of image data included in the video stream is stored in the layer of the container. It is inserted, and the processing unit may control the downscaling process for obtaining display image data based on this resolution information.

ADVANTAGE OF THE INVENTION According to this invention, when image data of ultra high definition service is transmitted without scalable encoding, the receiver which does not support this ultra high definition service can easily acquire the image data of the resolution suitable for one's display ability. have.

1 is a block diagram showing a configuration example of an image transmission/reception system as an embodiment.
2 is a diagram for explaining a downscaling process of spatial resolution.
3 is a block diagram showing a configuration example of a decoder of the receiver.
4 is a diagram for describing a downscaling process of spatial resolution.
5 is a diagram for explaining a case where the precision of the motion vector MV is not limited, for example, when the precision of the motion vector MV1 is 1/4 pixel precision.
FIG. 6 is a diagram for explaining a case where the precision of the motion vector MV is limited, for example, when the precision of the motion vector MV2 is half pixel precision.
7 is a diagram for explaining a downscaling process with temporal resolution.
8 is a block diagram showing an example of a configuration of a transmission data generation unit that generates a transport stream TS.
Fig. 9 is a diagram showing the access unit at the head of the GOP to which the SEI message is inserted as auxiliary information, and access units other than the head.
FIG. 10 is a diagram showing a structural example (Syntax) of an SEI message (downscaling_spatial SEI message) including information indicating precision limitation of a motion vector MV as auxiliary information.
11 is a diagram showing the contents of main information in an example of the structure of an SEI message (downscaling_spatial SEI message).
FIG. 12 is a diagram showing a structural example (Syntax) of an SEI message (picture_temporal_pickup SEI message) including information indicating a picture to be selected when downscaling temporal resolution as auxiliary information at a predetermined rate.
13 is a diagram showing the contents of main information in an example of the structure of an SEI message (picture_temporal_pickup SEI message).
14 is a diagram showing a structural example (Syntax) of the downscaling descriptor (downscaling_descriptor).
15 is a diagram showing an example of a modified structure (Syntax) of the downscaling descriptor (downscaling_descriptor).
Fig. 16 is a diagram showing the contents of main information in the structural example of the downscaling descriptor (downscaling_descriptor).
17 is a diagram showing a structural example (Syntax) of a Super High Resolution Descriptor.
18 is a diagram showing the contents of main information in an example of the structure of a Super High Resolution Descriptor.
19 is a diagram showing a configuration example of a transport stream TS.
20 is a block diagram showing a configuration example of a receiver.

Hereinafter, specific content (hereinafter referred to as "the embodiment") for carrying out the invention will be described. In addition, description is given in the following order.

1. Embodiment

2. Modification

<1. Embodiment>

[Image sending and receiving system]

1 shows a configuration example of the image transmission/reception system 10 as an embodiment. This image transmission/reception system 10 is composed of a broadcasting station 100 and a receiver 200. The broadcasting station 100 carries the transport stream TS as a container on a broadcast wave and transmits it.

The transport stream TS has a video stream containing encoded image data. The transmission image data includes those corresponding to various image services. As an image service, for example, in addition to an HD image service with an effective pixel count of 1920x1080, spatial ultra-high resolution image services (4K, 8K, etc., which are 2x, 4x, respectively, horizontally and vertically, respectively) (ultra-high definition service) Is considered. In addition, as an image service, for example, in addition to an image service having a frame frequency of 30 Hz, temporal ultra-high resolution image services (ultra-high definition service), such as frame frequencies of 60 Hz and 120 Hz, are considered.

With regard to the image data of the ultra high-definition service, there are cases where it is transmitted by scalable encoding, and there are cases where it is transmitted without scalable encoding. By being scalable-encoded, backward compatibility is ensured, and even in a receiver that does not support ultra-high-definition service, it is possible to easily acquire image data of a resolution suitable for its display capability.

When transmitting image data of an ultra-high-definition service, auxiliary information for downscaling spatial and/or temporal resolution of the image data is inserted into the video stream. This auxiliary information is inserted into, for example, a user data area of a picture header or a sequence header of a video stream.

For example, as auxiliary information for downscaling of spatial resolution, it becomes information indicating the precision limitation of motion vectors included in coded image data. For example, when the normal motion vector precision limit is 1/4 pixel precision, the motion vector precision limit is 1/2 pixel precision or to reduce the processing load of spatial resolution downscaling on the receiver side. 1 pixel precision, etc.

In addition, as auxiliary information for downscaling of temporal resolution, it becomes information identifying a picture to be selected when downscaling the temporal resolution at a predetermined ratio. For example, this information indicates that the picture to be selected when downscaling to 1/2 corresponding to one skipped picture (frame). In addition, for example, this information indicates that it is a picture to be selected when downscaling to 1/4 corresponding to three skipped pictures (frames).

By inserting the auxiliary information as described above, when the image data of the ultra-high-definition service is transmitted without scalable encoding, the receiver that does not support this ultra-high-definition service acquires image data of a resolution suitable for its display capability. Can be easily performed. The details of this auxiliary information will be described later.

Further, identification information indicating that auxiliary information is inserted into the video stream is inserted into the layer of the transport stream TS. For example, this identification information is inserted under the management of a video elementary loop (Video ES loop) of a Program Map Table (PMT) included in the transport stream TS. With this identification information, it can be seen that the auxiliary information is inserted into the video stream without decoding the video stream on the receiving side, and it becomes possible to appropriately extract the auxiliary information.

Spatial and/or temporal resolution information of image data included in a video stream may be added to this downscaling information. In this case, the receiving side can grasp the spatial and/or temporal resolution of the image data without decoding the video stream. Details of this downscaling information will be described later.

In addition, identification information is inserted into a layer of the transport stream TS so that ultra-high-definition services by the video stream can be identified at least in units of programs. For example, in the present embodiment, spatial and/or temporal resolution information of image data included in the video stream is inserted into the layer of the transport stream TS. For example, this resolution information is inserted under the management of an event information table (EIT) included in the transport stream TS. With this resolution information (identification information), it is possible to grasp the spatial and/or temporal resolution of the image data without decoding the video stream.

To this resolution information, identification information for identifying whether or not a video stream is supported for a low-capacity decoder that does not correspond to the spatial and/or temporal resolution of image data is added. In this case, the receiving side can easily determine whether the video stream is provided with support for a low-capacity decoder, for example, insertion of auxiliary information for downscaling of spatial and/or temporal resolution. Details of this resolution information will be described later.

The receiver 200 receives the transport stream TS carried on the broadcast wave from the broadcasting station 100. This transport stream TS has a video stream containing encoded image data. The receiver 200 decodes the video stream to obtain display image data.

The receiver 200 is a case where the image data of the ultra-high definition service is transmitted without being scalable encoded, and when it does not correspond to the ultra high-definition service, spatial and/or encoded image data is based on auxiliary information. Alternatively, temporal resolution downscaling is performed, and display image data of a desired resolution is obtained. In this case, the downscaling process is controlled by the resolution of the received image data and the possible ratio of downscaling.

For example, depending on the resolution of the received image data and the possible ratio of downscaling, it may be assumed that the display image data of the desired resolution cannot be obtained, but in that case, the downscaling process is not performed. In addition, when there are a plurality of possible ratios of downscaling, depending on the resolution of the received image data, the ratio of downscaling is selected to obtain display image data of a desired resolution.

[Resolution downscaling]

The downscaling process performed by the receiver 200 will be described. First, a downscaling process of spatial resolution is described. For example, consider a case where the received image data is 8K image data as shown in Fig. 2A. For example, in the receiver 200 having a display capability of 4K, a downscaling process in which the spatial resolution is halved both horizontally and vertically is performed, and as shown in Fig. 2B, 4K Acquisition of image data is performed. Further, for example, in the receiver 200 having a display capability of HD, a downscaling process in which the spatial resolution is set to 1/4 both horizontally and vertically is performed, as shown in Fig. 2C. Acquiring HD image data is performed.

3 shows a configuration example of a decoder of the receiver 200. The received coded image data Ve is subjected to entropy decoding processing by the entropy decoding unit 353a, and inverse quantization processing is performed by the inverse quantization unit 353b. In addition, the data after the inverse quantization process is subjected to an inverse transform process of the spatial frequency in the spatial frequency inverse transform unit 353c to obtain data D(n).

In this case, the inverse transform processing of the spatial frequency is applied only to the frequency components of the region according to the downscaling ratio for each N*N coded block (see the hatched region in Fig. 4A), and as data D(n) , Downscaled image data is obtained. In addition, this example of FIG. 4 shows the case where the ratio of downscaling is 1/2.

From the image data before one frame recorded in the frame buffer 353d (see Fig. 4(b)), pixel data of the region according to the motion vector MV is read for each coded block, supplied to the interpolation filter 353e, and interpolated. It is calculated, and a prediction block after interpolation is generated (see Fig. 4(c)). Then, in the adder 353f, the prediction block after interpolation generated by the interpolation filter 353e is added to the data D(n) (see Fig. 4(d)), and the downscaled image data Vd of the current frame (n) is obtained.

Here, the pixel precision of the motion vector MV added to the coded image data Ve is P. When the spatial frequency inverse transform unit 353c reduces the decoding to 1/2, for example, the pixel precision is 1/2 compared to the original precision P, and the precision becomes rough. In order to perform motion compensation with the pixel precision P of the original motion vector MV, it is necessary to interpolate the image data of the frame buffer 353d to suit the precision of P.

For example, when the original motion vector MV is encoded with 1/4 pixel precision, the precision when motion compensation is performed for the image data stored in the frame buffer 353d by reduction decoding, the pixel precision of the image data is 1 Since it is reduced to /2, in order to compensate motion with the precision of the original motion vector MV, it is necessary to interpolate the image data of the frame buffer 353d by 1/(1/4*1/2).

Therefore, when the precision of the motion vector MV is not limited, the predicted pixel range that is the target of the interpolation filter calculation is large, and the number of taps of the interpolation filter increases, which increases the computational load. On the other hand, when the precision of the motion vector MV is limited, the predicted pixel range that is the target of the interpolation filter operation is small, and the number of taps of the interpolation filter decreases, resulting in a small computational load.

Fig. 5 shows a case where the precision of the motion vector MV is not limited, for example, the precision of the motion vector MV1 is quarter pixel precision. In this case, in calculating interpolation pixels from prediction pixels adjacent to each other, a filter operation corresponding to the number of phases sufficient to cover the precision of MV1 is required. When performing interpolation calculation with a low-pass filter, in order to ensure a pass region of a certain level or more and to make the vicinity of the cutoff frequency steep, the number of filter taps of the interpolation filter increases, and consequently, the number of predicted pixels to be targeted increases.

6 shows a case where the precision of the motion vector MV is limited, for example, when the precision of the motion vector MV2 is half pixel precision. In this case, in obtaining interpolation pixels from prediction pixels adjacent to each other, a filter operation corresponding to the number of phases sufficient to cover the precision of MV2 is required. Since the precision of MV2 is rougher than that of MV1, the number of phases is reduced. In this case, the number of taps of the interpolation filter is at least, and the number of predicted pixels to be targeted is also at least as compared with the case where the above-described limitation is not applied.

In view of this, in the present embodiment, on the transmission side, the motion vector MV is properly encoded with precision limitations as in the motion vector MV2 described above. In that case, in this embodiment, the information of the precision limitation of the motion vector MV is inserted as auxiliary information in the video stream. When downscaling the spatial resolution, the receiver 200 recognizes the precision limitation of the motion vector MV from this auxiliary information and can perform interpolation processing that fits the precision limitation, thereby reducing the processing load. have.

Next, a downscaling process with temporal resolution is described. For example, consider a case where the received image data is 120 fps image data as shown in Fig. 7A. As auxiliary information, a half picture rate flag and a quarter picture rate flag are inserted as auxiliary information in the video stream.

The half picture rate flag is skipped by 1 picture (frame) and becomes "1". That is, the half picture rate flag can identify a picture to be selected when downscaling the time resolution to 1/2. The quarter picture rate flag is skipped 2 pictures (frames) and becomes "1". That is, it is possible to identify a picture to be selected when downscaling the temporal resolution to 1/4 by the quarter picture rate flag.

For example, in the receiver 200 having a display capability of 60 fps, based on the half picture rate flag, as shown in (b) of FIG. 7, only one skipped picture is taken out and decoded, Acquisition of image data is performed. Further, for example, in the receiver 200 having a display capability of 30 fps, based on the quarter picture rate flag, as shown in Fig. 7C, only three skipped pictures are taken out and decoded, It is done to obtain 30 fps image data.

"Structure Example of Transmission Data Generation Unit"

8 shows an example of the configuration of the transmission data generation unit 110 that generates the transport stream TS described above in the broadcasting station 100. The transmission data generation unit 110 includes an image data output unit 111, a video encoder 112, an audio data output unit 115, an audio encoder 116, and a multiplexer 117.

The image data output unit 111 outputs image data corresponding to various image services. Examples of the image service include an HD image service with an effective pixel count of 1920 x 1080, and a spatial ultra-high resolution image service (ultra-high definition service), such as 4K and 8K with double and quadruple effective pixel counts, respectively. In addition, examples of the image service include an image service having a frame frequency of 30 kHz, a temporal ultra-high resolution image service such as a frame frequency of 60 kHz, and 120 kHz (ultra-high definition service). The image data output unit 111 is configured by, for example, a camera that captures an object and outputs image data, or an image data reading unit that reads and outputs image data from a storage medium.

The video encoder 112 encodes the image data output from the image data output unit 111, for example, MPEG4-AVC (MVC), MPEG2video, or HEVC, to obtain encoded image data. In addition, the video encoder 112 generates a video stream (video elementary stream) including this encoded image data by a stream formatter (not shown) provided in the next step.

In this case, for example, with respect to the image data of the ultra high-definition service, in some cases, it is scalable coded to ensure backward compatibility, but in some cases it is not scalable coded. When not scalable, video encoder 112 inserts auxiliary information for downscaling of spatial and/or temporal resolution into a video stream for the convenience of a receiver that does not correspond to this ultra-high definition service.

The audio data output unit 115 outputs audio data corresponding to image data. The audio data output unit 115 is configured by, for example, a microphone or an audio data reading unit that reads and outputs audio data from a storage medium. The audio encoder 116 encodes audio data output from the audio data output unit 115, such as MPEG-2 Audio and AAC, and generates an audio stream (audio elementary stream).

The multiplexer 117 packetizes and multiplies each elementary stream generated by the video encoder 112, the graphics encoder 114, and the audio encoder 116 to generate a transport stream TS. In this case, a PTS (Presentation Time Stamp) is inserted into the header of each PES (Packetized Elementary Stream) for synchronous reproduction on the receiving side.

The multiplexer 117 inserts downscaling information indicating downscaling of a spatially and/or temporally possible resolution of resolution into a layer of the transport stream TS when transmitting image data of an ultra-high definition service without scalable encoding. do. For example, this downscaling information is inserted under the management of a Video Element Loop (PMT) of a Program Map Table (PMT) included in the transport stream TS.

In addition, the multiplexer 117 inserts identification information into a layer of the transport stream TS so that ultra-high-definition service by the video stream can be identified at least in units of programs. For example, in the present embodiment, the multiplexer 117 inserts spatial and/or temporal resolution information of image data included in the video stream into the layer of the transport stream TS. For example, this resolution information is inserted under the management of an event information table (EIT) included in the transport stream TS.

The operation of the transmission data generation unit 110 shown in FIG. 8 will be briefly described. Image data corresponding to various image services output from the image data output unit 111 is supplied to the video encoder 112. In the video encoder 112, for example, MPEG4-AVC (MVC), MPEG2video, or HEVC, etc. is encoded with respect to the image data, and a video stream containing the encoded image data (video elementary stream) This is created. This video stream is supplied to the multiplexer 117.

In this case, for example, with respect to the image data of the ultra high-definition service, in some cases, it is scalable coded to ensure backward compatibility, but in some cases it is not scalable coded. When not scalable encoding, the video encoder 112 inserts auxiliary information for downscaling of spatial and/or temporal resolution into a video stream for the convenience of a receiver that does not support this ultra-high definition service. This is done.

Audio data corresponding to the image data output from the audio data output section 115 is supplied to the audio encoder 116. The audio encoder 116 encodes the audio data, such as MPEG-2 Audio and AAC, and generates an audio stream (audio elementary stream). This audio stream is supplied to the multiplexer 117.

In the multiplexer 117, the elementary stream supplied from each encoder is packetized and multiplexed, and a transport stream TS is generated. In this case, PTS is inserted into each PES header for synchronous reproduction on the receiving side. Further, in the multiplexer 117, downscaling information indicating resolution downscaling at a spatially and/or temporally possible ratio is inserted under the management of the video elementary loop (PM) of the PMT. Further, in the multiplexer 117, spatial and/or temporal resolution information of image data included in the video stream is inserted under the management of the EIT.

[Structure of auxiliary information, identification information, and resolution information and TS configuration]

As described above, in the video stream, auxiliary information for downscaling of spatial and/or temporal resolution of image data is inserted. For example, when the encoding method is MPEG4-AVC, or when the encoding structures such as NAL packets, such as HEVC, are encoding methods similar to each other, this auxiliary information is provided in the part of "SEIs" of the access unit AU. , Inserted as an SEI message.

In this case, information indicating the precision limitation of the motion vector MV as auxiliary information is inserted as an SEI message (downscaling_spatial SEI message). In addition, information indicating a picture to be selected when downscaling the temporal resolution as auxiliary information at a predetermined rate is inserted as an SEI message (picture_temporal_pickup SEI message). Fig. 9(a) shows the access unit at the head of the GOP (Group Of Pictures), and Fig. 9(b) shows the access unit other than the head of the GOP. Since the SEI message is encoded at a position faster than the slices in which the pixel data is encoded, it is possible for the receiver to determine the subsequent decoding processing by identifying the contents of the SEI.

Fig. 10(a) shows a structural example (Syntax) of the "downscaling_spatial SEI message". "Uuid_iso_iec_11578" has a UUID value represented by "ISO/IEC 11578: 1996 AnnexA.". "Userdata_for_downscaling_spatial()" is inserted into the field of "user_data_payload_byte". Fig. 10B shows a structural example (Syntax) of "userdata_for_downscaling_spatial()". Among them, a flag of "constrained_to_half_pixel_MV_flag" and a flag of "constrained_to_integer_pixel_MV_flag" are included. "Userdata_id" is an identifier represented by unsigned 16 bits.

The flag of "constrained_to_half_pixel_MV_flag" indicates that the precision of the motion vector MV is limited to 1/2 pixel precision when "1", as shown in FIG. In addition, the flag of "constrained_to_integer_pixel_MV_flag" shows that when "1", the precision of the motion vector MV is limited to integer pixel precision, as shown in FIG.

Fig. 12A shows a structural example (Syntax) of the "picture_temporal_pickup SEI message". "Uuid_iso_iec_11578" has a UUID value represented by "ISO/IEC 11578: 1996 AnnexA.". "Userdata_for_picture_temporal()" is inserted into the field of "user_data_payload_byte". Fig. 12B shows a structural example (Syntax) of "userdata_for_picture_temporal()". Among them, a flag of the “half picture rate flag” and a flag of the “quarter picture rate flag” are included. "Userdata_id" is an identifier represented by unsigned 16 bits.

The flag of the "half picture rate flag" indicates that it is a picture to be taken out and decoded by a decoder having a display capability with 1/2 of the temporal resolution when "1", as shown in FIG. In addition, the flag of the "quarter picture rate flag", as shown in FIG. 13, indicates that when it is "1", it is a picture to be taken out and decoded by a decoder having a display capability with a temporal resolution of 1/4.

In addition, as described above, for example, under the management of the video elementary loop (Video ES loop) of the program map table (PMT) of the transport stream TS, in the video stream, the spatial and/or spatial data of the aforementioned image data and/or Identification information indicating that auxiliary information for downscaling of temporal resolution is inserted is inserted.

Fig. 14 shows a structural example (Syntax) of the downscaling descriptor (downscaling_descriptor) as this identification information. In addition, Fig. 15 shows an example of a modified structure (Syntax) of this downscaling descriptor (downscaling_descriptor). 16 shows the contents (Semantics) of main information in the structural examples.

The 8-bit field of "downscaling_descriptor_tag" indicates a descriptor type, and here indicates that it is a downscaling descriptor. The 8-bit field of "downscaling_descriptor_length" indicates the length (size) of the descriptor, and indicates the number of bytes that follow as the length of the descriptor.

A 2-bit field of "downscaling_type" indicates a downscaling type. For example, "01" represents downscaling of temporal resolution, "10" represents downscaling of spatial resolution, and "11" represents downscaling of temporal and spatial resolution.

When "downscaling_type" is "01" and "11", the 2-bit field of "temporal_downscaling_factor" becomes valid. This 2-bit field indicates the ratio (downscale) possible in downscaling with temporal resolution. For example, "00" indicates that downscaling is not possible. Also, "01" indicates that downscaling of 1/2 ratio is possible. "10" indicates that downscaling of 1/4 ratio is possible, but also indicates that downscaling of 1/2 ratio is possible. Also, "temporal_downscaling_factor" of "01" and "10" also indicates that auxiliary information for downscaling of temporal resolution is inserted into the video stream.

In addition, when "downscaling_type" is "10" and "11", the 2-bit field of "spatial_downscaling_factor" becomes valid. This 2-bit field indicates the ratio (downscale) possible in downscaling of spatial resolution. For example, "00" indicates that downscaling is not possible. Further, "01" indicates that downscaling of 1/2 ratio is possible horizontally and vertically. "10" indicates that downscaling of 1/4 ratio is possible horizontally and vertically, but also that downscaling of 1/2 ratio is possible. In addition, "spatial_downscaling_factor" of "01" and "10" also indicates that auxiliary information for downscaling of spatial resolution is inserted into the video stream.

The 3-bit field of "spatial resolution class type" indicates the class type of spatial resolution of the transmitted image data. For example, “001” indicates 1920×1080, that is, HD resolution. Further, for example, "010" indicates that it is 3840 x 2160, that is, 4K resolution. In addition, for example, "011" indicates that it is 7680×4320, that is, 8K resolution.

The 3-bit field of "temporal resolution class type" indicates the class type of temporal resolution of the transmitted image data. For example, "001" represents 24 km, 25 km, 29.97 km, 30 km, etc., "010" indicates 50 km, 59.94 km, 60 km, etc., "011" is 100 km, 120 km, etc. And "100" represents 200 kPa, 240 kPa, and the like.

Further, as described above, for example, spatial and/or temporal resolution information of image data included in the video stream is inserted under the management of the event information table (EIT) of the transport stream TS. Fig. 17 shows a structural example (Syntax) of the Super High resolution descriptor as this resolution information. 18 shows the content (Semantics) of the main information in the structural example.

The 3-bit field of "Spatial resolution class type" indicates the class type of spatial resolution of the transmitted image data. For example, “001” indicates 1920×1080, that is, HD resolution. Further, for example, "010" indicates that it is 3840 x 2160, that is, 4K resolution. In addition, for example, "011" indicates that it is 7680×4320, that is, 8K resolution.

The 3-bit field of "Temporal resolution class type" indicates the class type of temporal resolution of transmitted image data. For example, "001" represents 24 km, 25 km, 29.97 km, 30 km, etc., "010" indicates 50 km, 59.94 km, 60 km, etc., "011" is 100 km, 120 km, etc. Represents, "100" represents 200 Pa, 240 Pa, etc.

The 2-bit field of "Backward_compatible_type" indicates whether backward compatibility is guaranteed with respect to the transmitted image data. For example, "00" indicates that backward compatibility is not guaranteed. "01" indicates that backward compatibility with respect to spatial resolution is guaranteed. In this case, the transmitted image data is scalable encoded with respect to spatial resolution, for example. "10" indicates that backward compatibility with respect to temporal resolution is guaranteed. In this case, the transmitted image data is scalable encoded with respect to temporal resolution, for example.

The flag information of "lower_capable_decoder_support_flag" indicates whether there is support for a low-capacity decoder that does not correspond to the spatial and/or temporal resolution of the transmitted image data. For example, "0" indicates that it is not supported. "1" indicates that it is supported. For example, as described above, when the auxiliary information for downscaling of spatial and/or temporal resolution of image data is inserted in the video stream, this flag information becomes "1".

19 shows a configuration example of the transport stream TS. The transport stream TS includes a PES packet "PID1: video PES1" of the video elementary stream and a PES packet "PID2: Audio PES1" of the audio elementary stream. In this video elementary stream, auxiliary information for downscaling of spatial and/or temporal resolution of image data is inserted as an SEI message.

In this case, information indicating the precision limitation of the motion vector MV as auxiliary information is inserted as an SEI message (downscaling_spatial SEI message) (see FIG. 10). Further, information indicating a picture to be selected when downscaling the temporal resolution as auxiliary information at a predetermined rate is inserted as an SEI message (picture_temporal_pickup SEI message) (see FIG. 12).

Further, the transport stream TS includes a Program Map Table (PMT) as Program Specific Information (PSI). This PSI is information describing which program each elementary stream included in the transport stream belongs to. In addition, the transport stream TS includes an event information table (EIT) as service information (SI) for managing events (programs).

In the PMT, there is an elementary loop having information related to each elementary stream. In this configuration example, there is a video elementary loop (Video ES loop). In this video elementary loop, information such as a stream type and a packet identifier (PID) is arranged in correspondence to the above-mentioned one video elementary stream, and a descriptor describing information related to the video elementary stream is also provided. Is placed.

Under the management of the video elementary loop (Video ES loop) of the PMT, a downscaling descriptor (downscaling_descriptor) (see FIG. 14) is inserted. As described above, this descriptor indicates that auxiliary information for downscaling of spatial and/or temporal resolution of image data is inserted into the video stream.

Also, under the supervision of EIT, a Super High resolution descriptor (see FIG. 17). As described above, this descriptor constitutes identification information for identifying an ultra-high-definition service by a video stream in at least a program unit. Specifically, this descriptor contains spatial and/or temporal resolution information of transmitted image data.

"Configuration example of receiver"

20 shows a configuration example of the receiver 200. The receiver 200 includes a CPU 201, a flash ROM 202, a DRAM 203, an internal bus 204, a remote control receiver 205 (RC receiver), and a remote control transmitter 206: RC Transmitter).

In addition, the receiver 200 has an antenna terminal 211, a digital tuner 212, a transport stream buffer (213: TS buffer), and a demultiplexer 214. In addition, the receiver 200 includes a coded buffer 215, a video decoder 216, a decoded buffer 217, a video RAM 218, a coded buffer 241, and an audio decoder 242. ), and a channel mixing unit 243.

The CPU 201 controls the operation of each part of the receiver 200. The flash ROM 202 stores control software and stores data. The DRAM 203 constitutes a work area of the CPU 201. The CPU 201 deploys software or data read from the flash ROM 202 onto the DRAM 203 to activate the software to control each part of the receiver 200. The RC receiver 205 receives the remote control signal (remote control code) transmitted from the RC transmitter 206, and supplies it to the CPU 201. The CPU 201 controls each part of the receiver 200 based on this remote control code. The CPU 201, the flash ROM 202 and the DRAM 203 are connected to each other by an internal bus 204.

The antenna terminal 211 is a terminal for inputting a television broadcast signal received by a receiving antenna (not shown). The digital tuner 212 processes the television broadcast signal input to the antenna terminal 211, and outputs a predetermined transport stream TS corresponding to the user's selected channel. The transport stream buffer 213 (TS buffer) temporarily accumulates the transport stream TS output from the digital tuner 212. The transport stream TS includes a video elementary stream and an audio elementary stream.

The demultiplexer 214 extracts each stream (elementary stream) of video and audio from the transport stream TS temporarily accumulated in the TS buffer 213. Further, the demultiplexer 214 extracts the downscaling descriptor (downscaling_descriptor) and the super high resolution descriptor (Super High resolution descriptor) described above from the transport stream TS, and sends them to the CPU 201.

The CPU 201, from a super high resolution descriptor, provides spatial and temporal resolution information of the received image data, information on whether or not the received image data is backward compatible, and supports for a low-performance decoder in the received image data. It is possible to grasp information, such as whether or not. In addition, the CPU 201, from the downscaling descriptor, information on whether auxiliary information for downscaling processing of spatial and/or temporal resolution is inserted in the video stream, and also of spatial and/or temporal resolution down. In scaling, it is possible to grasp information of a possible ratio.

The CPU 201 controls processing such as decoding in the receiver 200 based on these pieces of information. For example, when image data of an ultra-high-definition service that cannot cope with its display capability is received, when it is not scalable encoded, the CPU 201 is based on the auxiliary information inserted in the video stream. Thus, downscaling processing of spatial and/or temporal resolution is performed, and control is performed so that display image data of a desired resolution is obtained.

The coded buffer 215 temporarily accumulates the video elementary stream extracted from the demultiplexer 214. The video decoder 216 decodes the video stream stored in the coded buffer 215 under the control of the CPU 201 to obtain display image data. Further, depending on the content of the received image data, downscaling of spatial and/or temporal resolution is also impossible, and in some cases, it is not possible to obtain display image data of a resolution that is in its own display capability.

In addition, the video decoder 216 extracts the SEI message inserted in the video stream and sends it to the CPU 201. The SEI message also includes "downscaling_spatial SEI message" and "picture_temporal_pickup SEI message". The CPU 201 causes the video decoder 216 to perform processing based on the auxiliary information included in this SEI message when downscaling processing with spatial and/or temporal resolution is performed.

That is, when downscaling processing with spatial resolution is performed, downscaling processing is performed based on precision limit information of the motion vector MV included in the SEI message of the "downscaling_spatial SEI message", thereby reducing the processing load. On the other hand, when downscaling processing with temporal resolution is performed, downscaling processing is performed based on selected picture information according to a ratio included in the SEI message of the "picture_temporal_pickup SEI message", thereby reducing the processing load.

The decoded buffer 217 temporarily stores the display image data obtained by the video decoder 216. The video RAM 218 retrieves display image data stored in the decoded buffer 217 and outputs it to the display at an appropriate timing.

The coded buffer 241 temporarily accumulates the audio stream extracted from the demultiplexer 214. The audio decoder 242 decodes the audio stream stored in the coded buffer 241 to obtain decoded audio data. The channel mixing unit 243 obtains audio data of each channel for realizing, for example, 5.1ch surround and the like, for the audio data obtained from the audio decoder 242 and supplies it to the speaker.

The operation of the receiver 200 will be described. The television broadcast signal input to the antenna terminal 211 is supplied to the digital tuner 212. In this digital tuner 212, a television broadcast signal is processed and a predetermined transport stream TS corresponding to a user's selection channel is output. The transport stream TS is temporarily stored in the TS buffer 213. The transport stream TS includes a video elementary stream and an audio elementary stream.

In the demultiplexer 214, each stream (elementary stream) of video and audio is extracted from the transport stream TS temporarily accumulated in the TS buffer 213. Further, in the demultiplexer 214, a downscaling descriptor (downscaling_descriptor) and a super high resolution descriptor (Super High resolution descriptor) are extracted from this transport stream TS and sent to the CPU 201. The CPU 201 controls the processing of the decoding and the like in the receiver 200 based on the information included in these descriptors.

The video stream extracted from the demultiplexer 214 is supplied to the coded buffer 215 and temporarily accumulated. In the video decoder 216, decoding processing is performed on the video stream stored in the coded buffer 215 under the control of the CPU 201, and display image data suitable for its display capability is obtained.

In this case, the video decoder 216 extracts an SEI message including the "downscaling_spatial SEI message", "picture_temporal_pickup SEI message", etc. inserted into the basic video stream, and sends it to the CPU 201. The CPU 201 causes the video decoder 216 to perform processing based on the auxiliary information included in this SEI message when downscaling processing with spatial and/or temporal resolution is performed.

The display image data obtained by the video decoder 216 is temporarily accumulated in the decoded buffer 217. Thereafter, in the video RAM 218, the display image data stored in the decoded buffer 217 is loaded at an appropriate timing and output to the display. Thereby, image display is performed on the display.

Further, the audio stream extracted from the demultiplexer 214 is supplied to the coded buffer 241 and is temporarily accumulated. In the audio decoder 242, the audio stream stored in the coded buffer 241 is decoded to obtain decoded audio data. This audio data is supplied to the channel mixing unit 243. In the channel mixing unit 243, audio data of each channel for realizing, for example, 5.1ch surround, etc. is generated for the audio data. This audio data is supplied to, for example, a speaker, and audio output adapted to the image display is made.

As described above, in the image transmission/reception system 10 shown in Fig. 1, auxiliary information for downscaling of spatial and/or temporal resolution of image data is inserted into a video stream and transmitted. Therefore, when the image data of the ultra-high-definition service is transmitted without scalable encoding, the receiver 200 that does not support this ultra-high-definition service can easily acquire the image data of a resolution suitable for its display capability. Can.

<2. Modification>

Moreover, in the above-mentioned embodiment, the example which the container is a transport stream (MPEG-2 TS) was shown. However, the present invention can also be applied to a system having a configuration distributed to a receiving terminal using a network such as the Internet. In the case of Internet distribution, it is often distributed by MP4 or other format containers. That is, as a container, containers of various formats, such as transport stream (MPEG-2 TS) which is adopted as a digital broadcasting standard, and MP4 used in Internet distribution, are applicable.

In addition, the present invention can also take the following structures.

[1] a transmitter for transmitting a container of a predetermined format having a video stream containing encoded image data;

An auxiliary information inserting unit for inserting auxiliary information for downscaling spatial and/or temporal resolution of the image data into the video stream.

Transmission device comprising a.

[2] The transmission device according to [1], wherein the auxiliary information is information indicating a precision limit of a motion vector included in the coded image data.

[3] The transmitting device according to [1] or [2], wherein the auxiliary information is information identifying a picture to be selected when downscaling the temporal resolution at a predetermined rate.

[4] The transmitting apparatus according to any one of [1] to [3], further comprising an identification information inserting unit for inserting identification information indicating that the auxiliary information is inserted into the video stream in a layer of the container. .

[5] The transmitting apparatus according to [4], wherein downscaling information indicating a possible ratio in downscaling of spatial and/or temporal resolution is added to the identification information.

[6] The transmission device according to [4] or [5], wherein spatial and/or temporal resolution information of image data included in the video stream is added to the identification information.

[7] The container is a transport stream,

The identification information inserting unit inserts the identification information into a descriptor under the management of a video elementary loop of a program map table included in the transport stream, and the transmission device according to any one of [4] to [6]. .

[8] The method according to any one of [1] to [7], further comprising a resolution information inserting unit for inserting spatial and/or temporal resolution information of image data included in the video stream into the layer of the container. Transmitting device.

[9] The resolution information includes:

The transmission device according to [8], in which identification information for identifying whether or not support is provided for a low-capacity decoder that does not correspond to the spatial and/or temporal resolution of the image data is added to the video stream.

[10] The container is a transport stream,

The transmission device according to [8] or [9], wherein the resolution information inserting unit inserts the resolution information into a descriptor under the management of an event information table included in the transport stream.

[11] a step of transmitting a container of a predetermined format having a video stream containing encoded image data;

A step of inserting auxiliary information for downscaling the spatial and/or temporal resolution of the image data into the video stream.

Transmission method comprising a.

[12] a transmitter for transmitting a container of a predetermined format having a video stream containing encoded image data;

Identification information inserting unit for inserting identification information in the layer of the container so that the high-definition service by the video stream can be identified by at least a program unit.

Transmission device comprising a.

[13] The transmission device according to [12], wherein the identification information includes spatial and/or temporal resolution information of the image data.

[14] The identification information includes:

The transmission according to [12] or [13], in which support information indicating whether or not support is provided for a low-capacity decoder that does not correspond to the spatial and/or temporal resolution of the image data is added to the video stream. Device.

[15] The container is a transport stream,

The identification information insertion unit is any one of [12] to [14], wherein the identification information is inserted into a descriptor under the management of an event information table included in the transport stream.

[16] a step of transmitting a container of a predetermined format having a video stream containing image data;

Step of inserting identification information in the layer of the container so that the high-definition service by the video stream can be identified at least in program units.

Transmission method comprising a.

[17] A receiving unit for receiving a video stream containing encoded image data,

In the video stream, auxiliary information for downscaling of spatial and/or temporal resolution of the image data is inserted,

A processing unit that performs downscaling of spatial and/or temporal resolution on the encoded image data based on the auxiliary information to obtain display image data of a desired resolution.

A receiving device further comprising.

[18] The receiving unit receives a container of a predetermined format including the video stream,

In the container layer, downscaling information indicating a possible ratio in downscaling of spatial and/or temporal resolution is inserted,

The receiving device according to [17], wherein the processing unit controls the downscaling process for obtaining the display image data based on the downscaling information.

[19] The receiving unit receives a container of a predetermined format including the video stream,

In the layer of the container, spatial and/or temporal resolution information of image data included in the video stream is inserted,

The receiving device according to [17] or [18], wherein the processing unit controls the downscaling process for obtaining the display image data based on the resolution information.

[20] a step of receiving a video stream including encoded image data and inserting auxiliary information for downscaling of spatial and/or temporal resolution of the image data;

A step of downscaling spatial and/or temporal resolution on the encoded image data based on the auxiliary information to obtain display image data of a desired resolution

A receiving method comprising a.

The main feature of the present invention is that by inserting and transmitting auxiliary information (SEI message) for downscaling of spatial and/or temporal resolution of image data into a video stream, it is possible to reduce the load of downscaling processing on the receiving side. This was done (see Fig. 19). In addition, the main feature of the present invention is to decode the video stream at the receiving side by inserting identification information so that the ultra-high-definition service by the video stream can be identified in at least a program unit in a layer of a container (transport stream). Instead, it is possible to identify the ultra-high quality service (see FIG. 19).

10: video transmission and reception system
100: broadcasting station
110: transmission data generation unit
111: image data output unit
112: video encoder
115: audio data output unit
116: audio encoder
117: multiplexer
200: receiver
201: CPU
212: digital tuner
213: transport stream buffer (TS buffer)
214: demultiplexer
215: coded buffer
216 video decoder
217: decoded buffer
218: video RAM
241: coded buffer
242: audio decoder
243: channel mixing unit

Claims (20)

As a transmitting device,
A transmitter for transmitting a container of a predetermined format having a video stream containing encoded image data;
An auxiliary information inserting unit for inserting auxiliary information for downscaling spatial and/or temporal resolution of the image data into the video stream.
Equipped with,
The auxiliary information identifies whether support for a low-capacity decoder that does not correspond to the spatial and/or temporal resolution of the image data is provided to the video stream and includes capability information added to the resolution information. The transmission device. According to claim 1,
The auxiliary information is information indicating a precision limit of a motion vector included in the coded image data. According to claim 1,
The auxiliary information is information for identifying a picture to be selected when downscaling a temporal resolution at a predetermined ratio. According to claim 1,
And an identification information inserting unit for inserting identification information indicating that the auxiliary information is inserted into the video stream in the layer of the container. According to claim 4,
A transmission apparatus in which downscaling information indicating a possible ratio in downscaling of spatial and/or temporal resolution is added to the identification information. According to claim 4,
Spatial and/or temporal resolution information of image data included in the video stream is added to the identification information. According to claim 4,
The container is a transport stream,
The identification information inserting unit inserts the identification information into a descriptor under the management of a video elementary loop of a program map table included in the transport stream. According to claim 1,
And a resolution information inserting unit for inserting spatial and/or temporal resolution information of image data included in the video stream into a layer of the container. delete The method of claim 8,
The container is a transport stream,
The resolution information inserting unit inserts the resolution information into a descriptor under management of an event information table included in the transport stream. As a transmission method,
A step of transmitting a container of a predetermined format having a video stream containing encoded image data;
A step of inserting auxiliary information for downscaling the spatial and/or temporal resolution of the image data into the video stream.
Equipped with,
The auxiliary information identifies whether or not support for a low-capacity decoder that does not correspond to the spatial and/or temporal resolution of the image data is provided to the video stream, and includes capability information added to the resolution information. . As a transmitting device,
A transmitter for transmitting a container of a predetermined format having a video stream containing encoded image data;
Identification information inserting unit for inserting identification information in the layer of the container so that the high-definition service by the video stream can be identified by at least a program unit.
A transmission device comprising a. The method of claim 12,
The identification information includes spatial and/or temporal resolution information of the image data. The method of claim 12,
In the identification information,
A transmission device is added to the video stream, in which support information indicating whether support for a low-capacity decoder that does not correspond to the spatial and/or temporal resolution of the image data is added. The method of claim 12,
The container is a transport stream,
The identification information inserting unit inserts the identification information into a descriptor under management of an event information table included in the transport stream. As a transmission method,
A step of transmitting a container of a predetermined format having a video stream containing image data;
Step of inserting identification information in the layer of the container so that the high-definition service by the video stream can be identified at least in program units.
A transmission method comprising a. As a receiving device,
A receiver for receiving a video stream including encoded image data;
In the video stream, auxiliary information for downscaling of spatial and/or temporal resolution of the image data is inserted,
A processing unit that obtains display image data of a desired resolution by performing downscaling of spatial and/or temporal resolution on the encoded image data based on the auxiliary information.
Equipped with,
The auxiliary information identifies whether or not support for a low-capacity decoder that does not correspond to the spatial and/or temporal resolution of the image data is provided to the video stream, and includes capability information added to the resolution information. . The method of claim 17,
The receiving unit receives a container of a predetermined format including the video stream,
In the container layer, downscaling information indicating a possible ratio in downscaling of spatial and/or temporal resolution is inserted,
The processing unit controls the downscaling process for obtaining the display image data based on the downscaling information. The method of claim 17,
The receiving unit receives a container of a predetermined format including the video stream,
In the layer of the container, spatial and/or temporal resolution information of image data included in the video stream is inserted,
The processing unit controls the downscaling process for obtaining the display image data based on the resolution information. As a receiving method,
A step of receiving a video stream including encoded image data and inserting auxiliary information for downscaling the spatial and/or temporal resolution of the image data;
A step of downscaling spatial and/or temporal resolution on the encoded image data based on the auxiliary information to obtain display image data of a desired resolution
Equipped with,
The auxiliary information includes whether the support for a low-capacity decoder that does not correspond to the spatial and/or temporal resolution of the image data is provided to the video stream and includes capability information added to the resolution information. .

Images