JP2012120143A - Stereoscopic image data transmission device, stereoscopic image data transmission method, stereoscopic image data reception device, and stereoscopic image data reception method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce, when transmitting disparity information to be sequentially updated within a predetermined number of frame periods during which overlapping information is displayed, the data amount of the disparity information.SOLUTION: A segment including disparity information to be sequentially updated during a subtitle display period is transmitted. On a reception side, disparity to be provided between a left eye subtitle and a right eye subtitle can be dynamically changed in conjunction with a change in image content. The disparity information is composed of disparity information of the first frame in the subtitle display period and disparity information of subsequent frames at update frame intervals. Therefore, the amount of transmission data can be reduced, and on the reception side, the capacity of a memory for holding the disparity information can be greatly conserved.

Description

  The present invention relates to a stereoscopic image data transmission device, a stereoscopic image data transmission method, a stereoscopic image data reception device, and a stereoscopic image data reception method, and in particular, a stereoscopic image data transmission device that transmits superimposition information data such as captions together with stereoscopic image data. Etc.

  For example, Patent Document 1 proposes a transmission method that uses a television broadcast radio wave of stereoscopic image data. In this transmission method, stereoscopic image data having left-eye image data and right-eye image data is transmitted, and stereoscopic image display using binocular parallax is performed.

  FIG. 95 shows the relationship between the display position of the left and right images of an object (object) on the screen and the playback position of the stereoscopic image in stereoscopic image display using binocular parallax. For example, with respect to the object A in which the left image La is displayed on the right side and the right image Ra is shifted to the left side as shown in the figure on the screen, the right and left line of sight intersects in front of the screen surface. The position is in front of the screen surface. DPa represents a horizontal disparity vector related to the object A.

  Further, for example, with respect to the object B in which the left image Lb and the right image Rb are displayed at the same position as shown in the figure on the screen, the right and left lines of sight intersect on the screen surface. It becomes on the surface. Further, for example, with respect to the object C displayed on the screen as shown in the figure, the left image Lc is shifted to the left side and the right image Rc is shifted to the right side, the right and left lines of sight intersect at the back of the screen surface. The playback position is behind the screen. DPc represents a horizontal disparity vector related to the object C.

Japanese Patent Laid-Open No. 2005-6114

  As described above, in stereoscopic image display, a viewer usually perceives the perspective of a stereoscopic image using binocular parallax. Superimposition information superimposed on an image, such as subtitles, is expected to be rendered in conjunction with stereoscopic image display not only in a two-dimensional space but also in a three-dimensional sense of depth. For example, when subtitles are superimposed on an image (overlay display), viewers may feel inconsistencies in perspective unless they are displayed in front of the closest object (object) in the perspective. is there.

  Therefore, it is conceivable to transmit parallax information between the left eye image and the right eye image together with the superimposition information data, and to add parallax between the left eye superimposition information and the right eye superimposition information on the receiving side. In this case, in order to dynamically change the parallax to be given between the left eye superimposing information and the right eye superimposing information in accordance with the change of the stereoscopic image, the superimposing information is displayed within a predetermined number of frame periods. It is necessary to send disparity information that is sequentially updated.

  An object of the present invention is to reduce the data amount of disparity information when sending disparity information that is sequentially updated within a predetermined number of frame periods in which superimposition information is displayed.

The concept of this invention is
An image data output unit for outputting stereoscopic image data having left-eye image data and right-eye image data;
A superimposition information data output unit for outputting superimposition information data to be superimposed on an image based on the left eye image data and the right eye image data;
A parallax information output unit for outputting parallax information for shifting the superimposition information to be superimposed on the image based on the left-eye image data and the right-eye image data and providing parallax;
A stereoscopic image data output from the image data output unit, a superimposition information data output from the superimposition information data output unit, and a data transmission unit that transmits parallax information output from the parallax information output unit,
The disparity information is disparity information that is sequentially updated within a predetermined number of frame periods in which the superimposition information is displayed, and the disparity information of the first frame of the predetermined number of frame periods and every subsequent update frame interval. It exists in the stereoscopic image data transmission apparatus which consists of the parallax information of a frame.

  In the present invention, the image data output unit outputs stereoscopic image data of a predetermined transmission format having left eye image data and right eye image data. For example, the transmission format of stereoscopic image data includes a side-by-side (Side By Side) method, a top-and-bottom (Top & Bottom) method, and the like.

  The superimposition information data output unit outputs the superimposition information data to be superimposed on the image based on the left eye image data and the right eye image data. Here, the superimposition information is information such as subtitles, graphics, and text superimposed on the image.

  The disparity information output unit outputs disparity information for giving disparity by shifting the superimposition information to be superimposed on the image based on the left eye image data and the right eye image data. The disparity information is disparity information that is sequentially updated within a predetermined number of frame periods in which superimposition information is displayed. The disparity information of the first frame in the predetermined number of frame periods and the frames for each subsequent update frame interval The parallax information is included. For example, the disparity information is disparity information corresponding to specific superimposition information displayed on the same screen and / or disparity information corresponding to a plurality of superimposition information displayed on the same screen.

  Then, the above-described stereoscopic image data, superimposition information data, and parallax information are transmitted by the data transmission unit. For example, the superimposition information data is DVB subtitle data, and the data transmission unit transmits the disparity information in a subtitle data stream including the subtitle data. For example, the disparity information is disparity information in units of regions or subregions included in this region. Further, for example, the disparity information is disparity information in units of pages including all regions.

  For example, the superimposition information data is ARIB subtitle data, and the data transmission unit transmits the parallax information included in the subtitle data stream including the subtitle data. Further, for example, the superimposition information data is CEA closed caption data, and the data transmission unit transmits the disparity information by including it in the user data area of the video data stream including the closed caption data.

  Thus, in the present invention, the parallax information corresponding to the stereoscopic image data is transmitted together with the stereoscopic image data and the superimposition information data. The disparity information is disparity information that is sequentially updated within a predetermined number of frame periods in which the superimposition information is displayed. The disparity information of the first frame in the predetermined number of frame periods and each subsequent update frame interval And the disparity information of the frame. Therefore, the parallax to be given between the left eye superimposition information and the right eye superimposition information can be dynamically changed on the reception side in accordance with the change of the stereoscopic image. In this case, not all the disparity information of each frame is transmitted, and the data amount of the disparity information can be reduced.

  In the present invention, information on a unit period and information on the number of unit periods may be added to the disparity information as information on each update frame interval. By adding information of each update frame interval to the disparity information, the update frame interval is not fixed, but the update frame interval can be set according to the disparity information curve. Further, by adding unit period information and information on the number of unit periods as update frame interval information, each update frame interval can be easily obtained by calculating “unit period * number”.

  For example, the unit period information is information that represents a unit period measured with a 90 KHz clock in a 24-bit length. The reason why the PTS inserted in the header portion of the PES is 33 bits long is 24 bits long for the following reason. That is, the 33-bit length can express a time exceeding 24 hours, but is an unnecessary length as a display period of superimposition information such as captions. In addition, by using 24 bits, the data size can be reduced and compact transmission can be performed. Further, 24 bits are 8 × 3 bits, and byte alignment is easy.

  In the present invention, for example, flag information indicating whether or not the disparity information is updated may be added to the disparity information for each frame at each update frame interval. In this case, when a period in which the change in the time direction of the disparity information continues is the same, the transmission of the disparity information within the period can be omitted using this flag information, and the data amount of the disparity information can be suppressed. It becomes.

  In the present invention, for example, information for adjusting the update frame interval may be added to the disparity information for each frame for each update frame period. In this case, based on this adjustment information, it is possible to arbitrarily adjust the update frame interval in the direction of shortening or lengthening the update frame interval, and the change in the time direction of the disparity information can be more accurately transmitted to the receiving side. It becomes.

  In the present invention, for example, information specifying a frame period may be inserted into the disparity information. As a result, the disparity information update frame interval intended on the transmission side can be correctly transmitted to the reception side. When this information is not added, for example, the video frame period is referred to on the receiving side.

  In the present invention, for example, information indicating a correspondence level for disparity information, which is essential when displaying superimposition information, may be inserted into the disparity information. In this case, it is possible to control the correspondence to the parallax information on the reception side with this information.

Another concept of the present invention is
Stereoscopic image data including left-eye image data and right-eye image data, superimposition information data to be superimposed on an image based on the left-eye image data, and superimposition information to be superimposed on an image based on the left-eye image data and the right-eye image data A data receiving unit that receives parallax information for shifting the image to give parallax,
The disparity information received by the receiving unit is disparity information that is sequentially updated within a predetermined number of frame periods in which the superimposition information is displayed, and the disparity information of the first frame of the predetermined number of frame periods; It consists of the disparity information of the frame for each subsequent update frame interval,
Using the left eye image data and the right eye image data received by the data receiving unit, the superimposition information data, and the parallax information, the same superimposition information to be superimposed on the left eye image and the right eye image The stereoscopic image data receiving apparatus further includes an image data processing unit that provides parallax and obtains left-eye image data on which the superimposition information is superimposed and right-eye image data on which the superimposition information is superimposed.

  In the present invention, the data reception unit receives the superimposition information data and the parallax information together with the stereoscopic image data including the left eye image data and the right eye image data. The superimposition information data is superimposition information data to be superimposed on the image based on the left eye image data and the right eye image data. Here, the superimposition information is information such as subtitles, graphics, and text superimposed on the image. The parallax information is for giving parallax by shifting the superimposition information to be superimposed on the image based on the left eye image data and the right eye image data. The disparity information is disparity information that is sequentially updated within a predetermined number of frame periods in which superimposition information is displayed. The disparity information of the first frame in the predetermined number of frame periods and the frames for each subsequent update frame interval The parallax information is included.

  Then, the image data processing unit uses the left eye image data and the right eye image data, the superimposition information data, and the parallax information, and gives the parallax to the same superimposition information to be superimposed on the left eye image and the right eye image. Thus, left-eye image data on which superimposition information is superimposed and right-eye image data on which superimposition information is superimposed are obtained.

  Thus, in the present invention, the parallax information corresponding to the stereoscopic image data is received together with the stereoscopic image data and the superimposition information data. The disparity information is disparity information that is sequentially updated within a predetermined number of frame periods in which the superimposition information is displayed. The disparity information of the first frame in the predetermined number of frame periods and each subsequent update frame interval Frame disparity information. Therefore, the parallax to be given between the left eye superimposition information and the right eye superimposition information can be dynamically changed in accordance with the change of the stereoscopic image. Also, not all the disparity information of each frame is transmitted, and it is possible to greatly save the memory capacity for holding the disparity information.

  In the present invention, for example, the image data processing unit performs an interpolation process on disparity information of a plurality of frames constituting disparity information that is sequentially updated within a predetermined number of frame periods, and within the predetermined number of frame periods. You may make it produce | generate and use the parallax information of arbitrary frame intervals. In this case, even when disparity information is transmitted from the transmission side every update frame interval, the disparity imparted to the superimposition information can be controlled at fine intervals, for example, for each frame.

  In this case, the interpolation process may be a linear interpolation process, but may be accompanied by a low-pass filter process in the time direction (frame direction), for example. Thereby, even when disparity information is transmitted from the transmission side every update frame interval, the change in the time direction of the disparity information after the interpolation process can be performed smoothly, and the transition of the disparity given to the superimposition information is A sense of incongruity due to discontinuity at every update frame interval can be suppressed.

  Also, in the present invention, for example, information on the unit period and information on the number of unit periods are added to the disparity information as information on the update frame interval, and the image data processing unit displays the display start time of the superimposition information. As a reference, each update time of the disparity information may be obtained based on unit period information and number information that are information of each update frame interval.

  In this case, the image data processing unit can obtain each update time sequentially from the display start time of the superimposition information. For example, with respect to a certain update time, the next update time is obtained by adding the unit period × the number of times to the certain update time by using the information of the unit period and the number of pieces of information of the next update frame interval. It is easily requested. Note that the display start time of the superimposition information is given by, for example, a PTS inserted in the header part of the PES stream including the disparity information.

  According to the present invention, the parallax information corresponding to the stereoscopic image data is transmitted together with the stereoscopic image data and the superimposition information data, and the parallax information is sequentially updated within a predetermined number of frame periods in which the superimposition information is displayed. Disparity information, and is composed of disparity information of the first frame in the predetermined number of frame periods and disparity information of frames at subsequent update frame intervals. Therefore, the parallax to be given between the left eye superimposition information and the right eye superimposition information can be dynamically changed on the reception side in accordance with the change of the stereoscopic image. Further, the transmission side does not transmit all the disparity information of each frame, and the transmission data amount can be reduced. In addition, on the receiving side, it is possible to greatly save the memory capacity for holding the parallax information.

It is a block diagram which shows the structural example of the image transmission / reception system as embodiment of this invention. It is a block diagram which shows the structural example of the transmission data generation part in a broadcast station. It is a figure which shows the image data of a 1920 * 1080 pixel format. It is a figure for demonstrating the "Top & Bottom" system, the "Side By Side" system, and the "Frame Sequential" system which are the transmission systems of stereo image data (3D image data). It is a figure for demonstrating the example which detects the parallax vector of the right eye image with respect to a left eye image. It is a figure for demonstrating calculating | requiring a parallax vector by a block matching system. It is a figure which shows the example of an image at the time of using the value of the parallax vector for every pixel (pixel) as a luminance value of each pixel (each pixel). It is a figure which shows an example of the parallax vector for every block (Block). It is a figure for demonstrating the downsizing process performed in the parallax information creation part of a transmission data generation part. It is a figure which shows the structural example of the transport stream (bit stream data) containing a video elementary stream, a subtitle elementary stream, and an audio elementary stream. It is a figure which shows the structure of PCS (page_composition_segment) which comprises subtitle data. It is a figure which shows the correspondence of each value of "segment_type" and a segment type. It is a figure for demonstrating the information (Component_type = 0x15, 0x25) which shows the format of the subtitle for 3D newly defined. It is a figure which shows notionally the production method of the subtitle data for stereoscopic images in case the transmission format of stereoscopic image data is a side-by-side system. It is a figure which shows notionally the production method of the subtitle data for stereoscopic images in case the transmission format of stereoscopic image data is a top and bottom system. It is a figure which shows notionally the production method of the subtitle data for stereoscopic images in case the transmission format of stereoscopic image data is a frame sequential system. It is a figure which shows the structural example (syntax) of SCS (Subregion composition segment). It is a figure which shows the structural example (syntax) of "Subregion_payload ()" contained in SCS. It is a figure which shows the main data regulation content (semantics) of SCS. It is a figure which shows the example of an update of the parallax information for every base segment period (BSP). It is a figure which shows the structural example (syntax) of "disparity_temporal_extension ()". The main data regulation content (semantics) in the structural example of “disparity_temporal_extension ()” is shown. It is a figure which shows the example of an update of the parallax information for every base segment period (BSP). It is a figure which shows roughly the flow of the stereo image data and subtitle data (including display control information) from a broadcasting station to a television receiver via a set top box, or from a broadcasting station to a television receiver directly. It is a figure which shows roughly the flow of the stereo image data and subtitle data (including display control information) from a broadcasting station to a television receiver via a set top box, or from a broadcasting station to a television receiver directly. It is a figure which shows roughly the flow of the stereo image data and subtitle data (including display control information) from a broadcasting station to a television receiver via a set top box, or from a broadcasting station to a television receiver directly. It is a figure which shows the example of a subtitle (graphics information) display on an image, and the perspective of a background, a foreground object, and a subtitle. It is a figure which shows the example of a display of a subtitle on an image, and the left eye subtitle LGI and the right eye subtitle RGI for displaying a subtitle. It is a block diagram which shows the structural example of the set top box which comprises a stereo image display system. It is a block diagram which shows the structural example of the bit stream process part which comprises a set top box. An example of generating disparity information (interpolated disparity information) at an arbitrary frame interval by performing interpolation processing with low-pass filter processing on disparity information of a plurality of frames constituting disparity information sequentially updated within a caption display period FIG. It is a block diagram which shows the structural example of the television receiver which comprises a stereo image display system. It is a block diagram which shows the structural example of the transmission data generation part in a broadcast station. It is a figure which shows the structural example of a caption data stream, and the example of a caption unit (caption) display. It is a figure which shows the structural example of the caption data stream produced | generated by a caption encoder, and the example of creation of the disparity vector in that case. It is a figure which shows the other structural example of the caption data stream produced | generated by a caption encoder, and the example of creation of the disparity vector in that case. It is a figure which shows the structural example of the caption data stream produced | generated by a caption encoder, and the example of creation of the disparity vector in that case. It is a figure which shows the other structural example of the caption data stream produced | generated by a caption encoder, and the example of creation of the disparity vector in that case. It is a figure for demonstrating the case where the position of each caption unit superimposed on the 1st, 2nd view is shifted. It is a figure which shows the packet structure of the subtitle code contained in the PES stream of a subtitle sentence data group. It is a figure which shows the packet structure of the control code contained in the PES stream of a caption management data group. It is a figure which shows the structure of the data group in a caption data stream (PES stream). It is a figure which shows roughly the structure of caption management data in case a disparity vector (disparity information) is inserted in the PES stream of a caption management data group. It is a figure which shows roughly the structure of caption data in case a disparity vector (disparity information) is inserted in the PES stream of a caption management data group. It is a figure which shows roughly the structure of caption data in case a disparity vector (disparity information) is inserted in the PES stream of a caption text data group. It is a figure which shows roughly the structure of caption management data in case a disparity vector (disparity information) is inserted in the PES stream of a caption text data group. It is a figure which shows the structure (Syntax) of the data unit (data_unit) contained in a caption data stream. It is a figure which shows the kind of data unit, a data unit parameter, and a function. It is a figure which shows the structure (Syntax) of the data unit (data_unit) of extended display control. It is a figure which shows the structure (Syntax) of "Advanced_Rendering_Control" in the data unit of the extended display control which the PES stream of a caption management data group has. It is a figure which shows the structure (Syntax) of "Advanced_Rendering_Control" in the data unit of the extended display control which the PES stream of a subtitle data group has. It is a figure which shows the main data prescription | regulation content in the structure of "Advanced_Rendering_Control" and the structure of "disparity_information". It is a figure which shows the structure (Syntax) of "disparity_information" in "Advanced_Rendering_Control" in the data unit (data_unit) of the extended display control contained in a caption text data group. It is a figure which shows the structure of "disparity_information". It is a figure which shows the structural example of the general transport stream (multiplexed data stream) containing a video elementary stream, an audio elementary stream, and a caption elementary stream. It is a figure which shows the structural example (Syntax) of a data content descriptor. It is a figure which shows the structural example (Syntax) of "arib_caption_info". It is a figure which shows the structural example of the transport stream (multiplexed data stream) in the case of inserting flag information under PMT. It is a figure which shows the structural example (Syntax) of a data encoding system descriptor. It is a figure which shows the structural example (Syntax) of "additional_arib_caption_info". It is a block diagram which shows the structural example of the bit stream process part of a set top box. It is a block diagram which shows the structural example of the transmission data generation part in a broadcast station. It is a figure which shows that the sequence header part containing the parameter of a sequence unit is arrange | positioned at the head of the elementary stream of video. It is a figure which shows a CEA table schematically. It is a figure which shows the structural example of the 3 byte field of Byte1 "," Byte2 "," Byte3 "which comprises an extended command. It is a figure which shows the example of an update of the parallax information for every base segment period (BSP). It is a figure which shows a CEA table schematically. It is a figure which shows the structural example of the 4-byte field of "Header (Byte1)" "Byte2", "Byte3", "Byte4". It is a figure which shows the structural example (Syntax) of the conventional closed caption data (CC data). It is a figure which shows the structural example (Syntax) of the closed caption data (CC data) correct | amended for disparity information (disparity) correspondence. It is a figure for demonstrating the 2 bit field of "extended_control" which controls 2 fields of "cc_data_1" and "cc_data_2". It is a figure which shows the structural example (syntax) of "caption_disparity_data ()". It is a figure which shows the structural example (syntax) of "disparity_temporal_extension ()". It is a figure which shows the main data prescription | regulation content (semantics) in the structural example of "caption_disparity_data ()". It is a figure which shows the structural example of the general transport stream (multiplexed data stream) containing a video elementary stream, an audio elementary stream, and a caption elementary stream. It is a block diagram which shows the structural example of the bit stream process part of a set top box. It is a figure which shows the other structural example (syntax) of "disparity_temporal_extension ()". The main data specification content (semantics) related to the structural example of “disparity_temporal_extension ()” is shown. It is a figure which shows the example of an update of parallax information at the time of using the other structural example of disparity_temporal_extension (). It is a figure which shows the example of an update of parallax information at the time of using the other structural example of disparity_temporal_extension (). It is a figure which shows the structural example of a subtitle data stream. It is a figure which shows the example of an update of parallax information in the case of transmitting an SCS segment sequentially. It is a figure which shows the example of an update of the disparity information (disparity) represented by the multiple of the interval period (ID: Interval Duration) as an update frame interval as a unit period. It is a figure which shows the structural example of the subtitle data stream in which each segment of DDS, PCS, RCS, CDS, ODS, DSS, and EOS is included as PES payload data. It is a figure which shows the example of a display of the subtitle in which the page area | region (Area for Page_default) contains two regions (Region) as a caption display area. When the DSS segment includes both disparity information in units of regions and disparity information in units of pages including all regions as disparity information (Disparity) sequentially updated in the caption display period, each region and page It is a figure which shows an example of this parallax information curve. It is a figure which shows what kind of structure the parallax information of a page and each region is sent. It is a figure (1/4) which shows the structural example (syntax) of DSS. It is a figure (2/4) which shows the structural example of DSS. It is a figure (3/4) which shows the structural example of DSS. It is a figure (4/4) which shows the structural example of DSS. FIG. 2 is a diagram (1/2) showing the main data definition contents (semantics) of the DSS. FIG. 2 is a diagram (2/2) showing the main data definition contents of the DSS. It is a block diagram which shows the other structural example of an image transmission / reception system. In stereoscopic image display using binocular parallax, it is a figure for demonstrating the relationship between the display position of the left-right image of the object on a screen, and the reproduction position of the stereoscopic image.

Hereinafter, modes for carrying out the invention (hereinafter referred to as “embodiments”) will be described. The description will be given in the following order.
1. Embodiment 2. FIG. Modified example

<1. Embodiment>
[Image transmission / reception system configuration example]
FIG. 1 shows a configuration example of an image transmission / reception system 10 as an embodiment. The image transmission / reception system 10 includes a broadcasting station 100, a set top box (STB) 200, and a television receiver (TV) 300.

  The set top box 200 and the television receiver 300 are connected by a digital interface of HDMI (High Definition Multimedia Interface). The set top box 200 and the television receiver 300 are connected using an HDMI cable 400. The set top box 200 is provided with an HDMI terminal 202. The television receiver 300 is provided with an HDMI terminal 302. One end of the HDMI cable 400 is connected to the HDMI terminal 202 of the set top box 200, and the other end of the HDMI cable 400 is connected to the HDMI terminal 302 of the television receiver 300.

[Description of broadcasting station]
The broadcasting station 100 transmits the bit stream data BSD on a broadcast wave. The broadcast station 100 includes a transmission data generation unit 110 that generates bit stream data BSD. The bit stream data BSD includes stereoscopic image data, audio data, superimposition information data, disparity information, and the like. The stereoscopic image data has a predetermined transmission format, and has left-eye image data and right-eye image data for displaying a stereoscopic image. The superimposition information is generally subtitles, graphics information, text information, etc., but in this embodiment, it is a subtitle.

"Configuration example of transmission data generator"
FIG. 2 shows a configuration example of the transmission data generation unit 110 in the broadcast station 100. The transmission data generation unit 110 transmits disparity information (disparity vector) with a data structure that can be easily linked to the DVB (Digital Video Broadcasting) method, which is one of existing broadcasting standards. The transmission data generation unit 110 includes a data extraction unit (archive unit) 111, a video encoder 112, and an audio encoder 113. Further, the transmission data generation unit 110 includes a subtitle generation unit 114, a disparity information creation unit 115, a subtitle processing unit 116, a subtitle encoder 118, and a multiplexer 119.

  A data recording medium 111a is detachably attached to the data extraction unit 111, for example. In this data recording medium 111a, audio data and parallax information are recorded in association with stereoscopic image data including left eye image data and right eye image data. The data extraction unit 111 extracts and outputs stereoscopic image data, audio data, parallax information, and the like from the data recording medium 111a. The data recording medium 111a is a disk-shaped recording medium, a semiconductor memory, or the like.

  The stereoscopic image data recorded on the data recording medium 111a is stereoscopic image data of a predetermined transmission method. An example of a transmission method of stereoscopic image data (3D image data) will be described. Here, although the following 1st-3rd transmission systems are mentioned, transmission systems other than these may be used. Also, here, as shown in FIG. 3, the case where the image data of the left eye (L) and the right eye (R) is image data of a predetermined resolution, for example, a 1920 × 1080 pixel format. Let's take an example.

  The first transmission method is a top-and-bottom method. As shown in FIG. 4A, in the first half of the vertical direction, the data of each line of the left eye image data is transmitted, and the vertical direction In the latter half of the method, the data of each line of the left eye image data is transmitted. In this case, since the lines of the left eye image data and the right eye image data are thinned out to ½, the vertical resolution is halved with respect to the original signal.

  The second transmission method is a side-by-side method, and as shown in FIG. 4B, the pixel data of the left eye image data is transmitted in the first half in the horizontal direction, and the second half in the horizontal direction. Then, the pixel data of the right eye image data is transmitted. In this case, in the left eye image data and the right eye image data, the pixel data in the horizontal direction is thinned out to 1/2. The horizontal resolution is halved with respect to the original signal.

  The third transmission method is a frame sequential method, in which left-eye image data and right-eye image data are sequentially switched and transmitted for each frame, as shown in FIG. 4C. This frame sequential method may be referred to as a full frame method or a backward compatible method.

  Moreover, the parallax information recorded on the data recording medium 111a is, for example, a parallax vector for each pixel (pixel) constituting the image. A detection example of a disparity vector will be described. Here, an example in which the parallax vector of the right eye image with respect to the left eye image is detected will be described. As shown in FIG. 5, the left eye image is a detected image, and the right eye image is a reference image. In this example, the disparity vectors at the positions (xi, yi) and (xj, yj) are detected.

  A case where a disparity vector at the position (xi, yi) is detected will be described as an example. In this case, for example, a 4 × 4, 8 × 8, or 16 × 16 pixel block (parallax detection block) Bi is set in the left eye image with the pixel at the position (xi, yi) at the upper left. Then, a pixel block matching the pixel block Bi is searched in the right eye image.

  In this case, a search range centered on the position of (xi, yi) is set in the right eye image, and each pixel in the search range is sequentially set as a pixel of interest, for example, 4 × 4 similar to the above-described pixel block Bi. 8 × 8 or 16 × 16 comparison blocks are sequentially set.

  Between the pixel block Bi and the sequentially set comparison blocks, the sum of absolute difference values for each corresponding pixel is obtained. Here, as shown in FIG. 6, when the pixel value of the pixel block Bi is L (x, y) and the pixel value of the comparison block is R (x, y), the pixel block Bi, a certain comparison block, The sum of absolute differences between the two is represented by Σ | L (x, y) −R (x, y) |.

  When n pixels are included in the search range set in the right eye image, n total sums S1 to Sn are finally obtained, and the minimum sum Smin is selected. Then, the position of the upper left pixel (xi ′, yi ′) is obtained from the comparison block from which the sum Smin is obtained. Thereby, the disparity vector at the position (xi, yi) is detected as (xi′−xi, yi′−yi). Although detailed description is omitted, for the disparity vector at the position (xj, yj), the left eye image has the pixel at the position (xj, yj) at the upper left, for example, 4 × 4, 8 × 8, or 16 × Sixteen pixel blocks Bj are set and detected in the same process.

  The video encoder 112 performs encoding such as MPEG4-AVC, MPEG2, or VC-1 on the stereoscopic image data extracted from the data extraction unit 111, and generates a video data stream (video elementary stream). The audio encoder 113 performs encoding such as AC3 or AAC on the audio data extracted from the data extraction unit 111 to generate an audio data stream (audio elementary stream).

  The subtitle generating unit 114 generates subtitle data that is DVB (Digital Video Broadcasting) subtitle data. This subtitle data is subtitle data for a two-dimensional image. The subtitle generation unit 114 constitutes a superimposition information data output unit.

  The disparity information creating unit 115 performs a downsizing process on the disparity vector (horizontal disparity vector) for each pixel (pixel) extracted from the data extracting unit 111, and disparity information (horizontal disparity to be applied to the subtitle) Vector). The parallax information creation unit 115 constitutes a parallax information output unit. Note that the disparity information applied to the subtitle can be attached in units of pages, regions, or objects. The disparity information does not necessarily have to be generated by the disparity information creating unit 115, and a configuration in which the disparity information is separately supplied from the outside is also possible.

  FIG. 7 shows an example of data in the relative depth direction given as the luminance value of each pixel (pixel). Here, the data in the relative depth direction can be handled as a disparity vector for each pixel by a predetermined conversion. In this example, the luminance value of the person portion is high. This means that the value of the parallax vector of the person portion is large, and therefore, in stereoscopic image display, this means that the person portion is perceived as being raised. In this example, the luminance value of the background portion is low. This means that the value of the parallax vector in the background portion is small, and therefore, in stereoscopic image display, this means that the background portion is perceived as a sunken state.

  FIG. 8 shows an example of a disparity vector for each block. The block corresponds to an upper layer of pixels (picture elements) located at the lowermost layer. This block is configured by dividing an image (picture) region into a predetermined size in the horizontal direction and the vertical direction. The disparity vector of each block is obtained, for example, by selecting the disparity vector having the largest value from the disparity vectors of all pixels (pixels) existing in the block. In this example, the disparity vector of each block is indicated by an arrow, and the length of the arrow corresponds to the magnitude of the disparity vector.

  FIG. 9 shows an example of the downsizing process performed by the disparity information creating unit 115. First, as shown in FIG. 9A, the disparity information creating unit 115 obtains a disparity vector for each block using a disparity vector for each pixel (pixel). As described above, a block corresponds to an upper layer of pixels located at the lowest layer, and is configured by dividing an image (picture) region into a predetermined size in the horizontal direction and the vertical direction. Then, the disparity vector of each block is obtained, for example, by selecting the disparity vector having the largest value from the disparity vectors of all the pixels (pixels) existing in the block.

  Next, the disparity information creating unit 115 obtains a disparity vector for each group (Group Of Block) using the disparity vector for each block, as illustrated in FIG. A group is an upper layer of a block, and is obtained by grouping a plurality of adjacent blocks together. In the example of FIG. 9B, each group is composed of four blocks bounded by a broken line frame. The disparity vector of each group is obtained, for example, by selecting the disparity vector having the largest value from the disparity vectors of all blocks in the group.

  Next, the disparity information creating unit 115 obtains a disparity vector for each partition (Partition) using the disparity vector for each group, as shown in FIG. The partition is an upper layer of the group and is obtained by grouping a plurality of adjacent groups together. In the example of FIG. 9C, each partition is configured by two groups bounded by a broken line frame. The disparity vector of each partition is obtained, for example, by selecting the disparity vector having the largest value from the disparity vectors of all groups in the partition.

  Next, the disparity information creating unit 115 obtains the disparity vector of the entire picture (entire image) located in the highest layer using the disparity vector for each partition, as shown in FIG. In the example of FIG. 9D, the entire picture includes four partitions that are bounded by a broken line frame. Then, the disparity vector for the entire picture is obtained, for example, by selecting the disparity vector having the largest value from the disparity vectors for all partitions included in the entire picture.

  In this way, the disparity information creating unit 115 performs the downsizing process on the disparity vector for each pixel (pixel) located in the lowest layer, and the disparity vectors of the respective regions in each layer of the block, group, partition, and entire picture Can be requested. In the example of the downsizing process shown in FIG. 9, finally, in addition to the pixel (pixel) layer, disparity vectors of four layers of blocks, groups, partitions, and entire pictures are obtained. However, the number of hierarchies, how to cut areas in each hierarchy, and the number of areas are not limited to this.

  Returning to FIG. 2, the subtitle processing unit 116 converts the subtitle data generated by the subtitle generation unit 114 for a stereoscopic image (for a three-dimensional image) corresponding to the transmission format of the stereoscopic image data extracted from the data extraction unit 111. Convert to subtitle data. The subtitle processing unit 116 constitutes a superimposition information data processing unit, and the converted subtitle data for stereoscopic image data constitutes transmission superimposition information data.

  The stereoscopic image subtitle data includes left-eye subtitle data and right-eye subtitle data. Here, the left-eye subtitle data is data corresponding to the left-eye image data included in the above-described stereoscopic image data, and the display of the left-eye subtitle superimposed on the left-eye image data included in the stereoscopic image data on the receiving side. Data for generating data. The right-eye subtitle data is data corresponding to the right-eye image data included in the above-described stereoscopic image data, and display data of the right-eye subtitle to be superimposed on the right-eye image data included in the stereoscopic image data on the receiving side. It is data for generating.

  In this case, the subtitle processing unit 116 shifts at least the left eye subtitle or the right eye subtitle based on the disparity information (horizontal disparity vector) to be applied to the subtitle from the disparity information creating unit 115, and Parallax can be given to the right eye subtitle. In this way, by giving parallax between the left eye subtitle and the right eye subtitle, each object in the image can be displayed in the display of the subtitle (caption) without performing parallax processing on the receiving side. The consistency of perspective between the two can be maintained in an optimum state.

  The subtitle processing unit 116 includes a display control information generation unit 117. The display control information generation unit 117 generates display control information related to the subregion. Here, the sub-region is an area defined only within the region. This subregion includes a left eye subregion (left eye SR) and a right eye subregion (right eye SR). Hereinafter, as appropriate, the subregion is referred to as the left eye SR, and the right eye subregion is referred to as the right eye SR.

  The left-eye subregion is an area set corresponding to the display position of the left-eye subtitle in the region that is the display area of the transmission superimposition information data. The right-eye subregion is an area set corresponding to the display position of the right-eye subtitle in the region that is the display area of the transmission superimposition information data. For example, the left eye subregion constitutes a first display area, and the right eye subregion constitutes a second display area. These left eye SR and right eye SR regions are set for each subtitle data generated by the subtitle generating unit 116, for example, based on a user operation or automatically. In this case, the regions of the left eye SR and the right eye SR are set so that the left eye subtitle in the left eye SR and the right eye subtitle in the right eye SR correspond to each other.

  The display control information includes area information for the left eye SR and area information for the right eye SR. The display control information includes information on a target frame that displays a left-eye subtitle included in the left eye SR and information on a target frame that displays a right-eye subtitle included in the right eye SR. Here, the information of the target frame displaying the left eye subtitle included in the left eye SR indicates the frame of the left eye image, and the information of the target frame displaying the right eye subtitle included in the right eye SR is the frame of the right eye image. Indicates.

  The display control information includes disparity information (disparity) for shifting the display position of the left eye subtitle included in the left eye SR and disparity information for shifting the display position of the right eye subtitle included in the right eye SR. And are included. These pieces of parallax information are for giving parallax between the left eye subtitle included in the left eye SR and the right eye subtitle included in the right eye SR.

  In this case, the display control information generation unit 117 performs shift adjustment to be included in the above-described display control information based on the disparity information (horizontal disparity vector) to be applied to, for example, the subtitle created by the disparity information creation unit 115. Is obtained. Here, the disparity information “Disparity1” of the left eye SR and the disparity information “Disparity2” of the right eye SP have the same absolute value, and the difference between them corresponds to the disparity information (Disparity) to be applied to the subtitle. It is determined to be a value. For example, when the transmission format of the stereoscopic image data is the side-by-side format, the value corresponding to the disparity information (Disparity) is “Disparity / 2”. Further, for example, when the transmission format of the stereoscopic image data is a top-and-bottom method, the value corresponding to the disparity information (Disparity) is “Disparity”.

  The subtitle data has segments such as DDS, PCS, RSC, CDS, and ODS. A display definition segment (DDS) designates a display size for HDTV. PCS (page composition segment) designates a region position within a page. The RCS (region composition segment) specifies the size of the region (Region), the encoding mode of the object (object), and specifies the start position of the object (object). CDS (CLUT definition segment) designates CLUT contents. The ODS (objectdata segment) includes encoded pixel data (Pixeldata).

  In this embodiment, an SCS (Subregion composition segment) segment is newly defined. The display control information generated by the display control information generation unit 117 as described above is inserted into this SCS segment. Details of the processing of the subtitle processing unit 116 will be described later.

  Returning to FIG. 2, the subtitle encoder 118 generates a subtitle data stream (subtitle elementary stream) including the subtitle data for stereoscopic images output from the subtitle processing unit 116 and display control information. The multiplexer 119 multiplexes each data stream from the video encoder 119, the audio encoder 120, and the subtitle encoder 125 to obtain a multiplexed data stream as bit stream data (transport stream) BSD.

  In this embodiment, the multiplexer 119 inserts identification information for identifying that subtitle data for stereoscopic images is included in the subtitle data stream. Specifically, Stream_content ('0x03' = DVB subtitles) & Component_type (for 3D target) is described in a component descriptor (Component_Descriptor) inserted under an EIT (Event Information Table). Component_type (for3D target) is newly defined to indicate subtitle data for stereoscopic images.

  The operation of the transmission data generation unit 110 shown in FIG. 2 will be briefly described. The stereoscopic image data extracted from the data extraction unit 111 is supplied to the video encoder 112. In the video encoder 112, the stereoscopic image data is encoded by MPEG4-AVC, MPEG2, VC-1, or the like, and a video data stream including the encoded video data is generated. This video data stream is supplied to the multiplexer 119.

  The audio data extracted by the data extraction unit 111 is supplied to the audio encoder 113. The audio encoder 113 performs encoding such as MPEG-2Audio AAC or MPEG-4 AAC on the audio data, and generates an audio data stream including the encoded audio data. This audio data stream is supplied to the multiplexer 119.

  The subtitle generation unit 114 generates subtitle data (for two-dimensional images) that is DVB subtitle data. This subtitle data is supplied to the disparity information creating unit 115 and the subtitle processing unit 116.

  The disparity vector for each pixel (pixel) extracted from the data extracting unit 111 is supplied to the disparity information creating unit 115. In the disparity information creating unit 115, downsizing processing is performed on the disparity vector for each pixel, and disparity information (horizontal disparity vector = Disparity) to be applied to the subtitle is created. This disparity information is supplied to the subtitle processing unit 116.

  The subtitle processing unit 116 converts the 2D image subtitle data generated by the subtitle generation unit 114 into 3D image subtitle data corresponding to the transmission format of the 3D image data extracted from the data extraction unit 111 described above. Is done. The stereoscopic image subtitle data includes left-eye subtitle data and right-eye subtitle data. In this case, the subtitle processing unit 116 shifts at least the left-eye subtitle or the right-eye subtitle based on the disparity information to be applied to the subtitle from the disparity information creation unit 115, and In some cases, parallax may be provided between them.

  The display control information generation unit 117 of the subtitle processing unit 116 generates display control information (region information, target frame information, and disparity information) related to the subregion (Subregion). As described above, the subregion includes the left eye subregion (left eye SR) and the right eye subregion (right eye SR). Therefore, area information, target frame information, and parallax information of the left eye SR and right eye SR are generated as display control information.

  As described above, the left eye SR is set corresponding to the display position of the left eye subtitle, for example, in a region that is a display area of superimposing information data for transmission based on a user operation. . Similarly, the right eye SR is set corresponding to the display position of the right eye subtitle, for example, based on a user operation or automatically in a region that is a display area of the superimposed information data for transmission.

  The stereoscopic image subtitle data and display control information obtained by the subtitle processing unit 117 are supplied to the subtitle encoder 118. This subtitle encoder 118 generates a subtitle data stream including stereoscopic image subtitle data and display control information. The subtitle data stream includes a newly defined SCS segment including display control information, as well as segments such as DDS, PCS, RCS, CDS, and ODS into which stereoscopic image subtitle data is inserted.

  As described above, each data stream from the video encoder 112, the audio encoder 113, and the subtitle encoder 118 is supplied to the multiplexer 119. In the multiplexer 119, each data stream is packetized and multiplexed to obtain a multiplexed data stream as bit stream data (transport stream) BSD.

  FIG. 10 shows a configuration example of a transport stream (bit stream data). This transport stream includes PES packets obtained by packetizing each elementary stream. In this configuration example, the PES packet “Video PES” of the video elementary stream, the PES packet “AudioPES” of the audio elementary stream, and the PES packet ““ Subtitle PES ”of the subtitle elementary stream are included.

  In this embodiment, the subtitle elementary stream (subtitle data stream) includes subtitle data for stereoscopic images and display control information. This stream includes SCS segments including newly defined display control information, as well as conventionally known segments such as DDS, PCS, RCS, CDS, and ODS.

  FIG. 11 shows the structure of PCS (page_composition_segment). The segment type of this PCS is “0x10” as shown in FIG. “Region_horizontal_address” and “region_vertical_address” indicate the start position of the region. Note that the structure of other segments such as DDS, RSC, and ODS is not shown. As shown in FIG. 12, the DDS segment type is “0x14”, the RCS segment type is “0x11”, the CDS segment type is “0x12”, and the ODS segment type is “0x13”. . For example, as shown in FIG. 12, the SCS segment type is “0x40”. The detailed structure of the SCS segment will be described later.

  Returning to FIG. 10, the transport stream includes a PMT (Program Map Table) as PSI (Program Specific Information). This PSI is information describing to which program each elementary stream included in the transport stream belongs. Further, the transport stream includes an EIT (Event Information Table) as SI (Serviced Information) for managing each event. In the EIT, metadata for each program is described.

  The PMT includes a program descriptor (Program Descriptor) that describes information related to the entire program. The PMT includes an elementary loop having information related to each elementary stream. In this configuration example, there are a video elementary loop, an audio elementary loop, and a subtitle elementary loop. In each elementary loop, information such as a packet identifier (PID) is arranged for each stream, and a descriptor (descriptor) describing information related to the elementary stream is also arranged, although not shown. The

  A component descriptor (Component_Descriptor) is inserted under the EIT. In this embodiment, Stream_content ('0x03' = DVB subtitles) & Component_type (for 3D target) is described in this component descriptor. Thereby, it is possible to identify that the subtitle data stream includes the subtitle data for stereoscopic images. In this embodiment, as shown in FIG. 13, when “stream_content” of “component_descriptor” indicating distribution content indicates a subtitle (subtitle), information indicating a 3D subtitle format (Component_type = 0x15, 0x25) Is newly defined.

[Process of subtitle processing section]
Details of the processing of the subtitle processing unit 116 of the transmission data generating unit 110 shown in FIG. 2 will be described. As described above, the subtitle processing unit 116 converts the subtitle data for two-dimensional images into subtitle data for stereoscopic images. Further, as described above, the subtitle processing unit 116 generates display control information (including region information, target frame information, and disparity information of the left eye SR and right eye SR) in the display control information generation unit 117.

  FIG. 14 conceptually shows a method for creating stereoscopic image subtitle data when the transmission format of stereoscopic image data is the side-by-side format. FIG. 14A shows a region based on subtitle data for a two-dimensional image. In this example, the region includes three objects.

  First, the subtitle processing unit 116 converts the size of the region based on the above-described subtitle data for a two-dimensional image into a size suitable for the side-by-side method, as shown in FIG. Generate bitmap data of that size.

  Next, as shown in FIG. 14C, the subtitle processing unit 116 uses the bitmap data after the size conversion as a constituent element of the region in the stereoscopic image subtitle data. That is, the bitmap data after the size conversion is an object corresponding to the left eye subtitle in the region and an object corresponding to the right eye subtitle in the region.

  As described above, the subtitle processing unit 116 converts the subtitle data for two-dimensional images into subtitle data for stereoscopic images, and DDS, PCS, RCS, CDS, ODS corresponding to the subtitle data for stereoscopic images. Create a segment such as

  Next, as shown in FIG. 14C, the subtitle processing unit 116, based on a user operation or automatically, on the region of the region in the stereoscopic image subtitle data, as shown in FIG. Set the right eye SR. The left eye SR is set in an area including an object corresponding to the left eye subtitle. The right eye SR is set in an area including an object corresponding to the right eye subtitle.

  The subtitle processing unit 116 creates an SCS segment including the region information, target frame information, and disparity information of the left eye SR and right eye SR set as described above. For example, the subtitle processing unit 116 creates an SCS including the left eye SR and right eye SR region information, target frame information, and disparity information in common, or the left eye SR and right eye SR region information, target frame information, SCS segments each including disparity information are created.

  FIG. 15 conceptually shows a method for creating stereoscopic image subtitle data when the transmission format of stereoscopic image data is the top-and-bottom method. FIG. 15A shows a region (region) based on subtitle data for a two-dimensional image. In this example, the region includes three objects.

  First, the subtitle processing unit 116 converts the size of the region based on the above-described subtitle data for a two-dimensional image into a size suitable for the top-and-bottom method, as shown in FIG. Generate bitmap data of that size.

  Next, as shown in FIG. 15C, the subtitle processing unit 116 uses the bitmap data after the size conversion as a constituent element of the region of the stereoscopic image subtitle data. That is, the bitmap data after the size conversion is used as a region object on the left eye image (leftview) side and a region object on the right eye image (Right view) side.

  As described above, the subtitle processing unit 116 converts the subtitle data for the two-dimensional image into the subtitle data for the stereoscopic image, and PCS, RCS, CDS, ODS, etc. corresponding to the stereoscopic image subtitle data. Create a segment.

  Next, as shown in FIG. 15C, the subtitle processing unit 116, on the region of the region in the stereoscopic image subtitle data, automatically or based on a user operation or automatically. Set the right eye SR. The left eye SR is set to an area including an object in the region on the left eye image side. The right eye SR is set to an area including an object in the region on the left eye image side.

  The subtitle processing unit 116 creates an SCS segment including the region information, target frame information, and disparity information of the left eye SR and right eye SR set as described above. For example, the subtitle processing unit 116 creates an SCS including the left eye SR and right eye SR region information, target frame information, and disparity information in common, or the left eye SR and right eye SR region information, target frame information, SCS segments each including disparity information are created.

  FIG. 16 conceptually shows a method for creating stereoscopic image subtitle data when the transmission format of stereoscopic image data is the frame sequential method. FIG. 16A shows a region based on subtitle data for a two-dimensional image. In this example, the region includes one object. When the transmission format of the stereoscopic image data is the frame sequential method, the 2D image subtitle data is used as it is as the stereoscopic image subtitle data. In this case, segments such as DDS, PCS, RCS, and ODS corresponding to subtitle data for 2D images become segments such as DDS, PCS, RCS, and ODS corresponding to subtitle data for stereoscopic images as they are.

  Next, as shown in FIG. 16B, the subtitle processing unit 116, based on the user operation or automatically, on the region of the region in the stereoscopic image subtitle data, as shown in FIG. Set the right eye SR. The left eye SR is set in an area including an object corresponding to the left eye subtitle. The right eye SR is set in an area including an object corresponding to the right eye subtitle.

  The subtitle processing unit 116 creates an SCS segment including the region information, target frame information, and disparity information of the left eye SR and right eye SR set as described above. For example, the subtitle processing unit 116 creates an SCS including the left eye SR and right eye SR region information, target frame information, and disparity information in common, or the left eye SR and right eye SR region information, target frame information, SCS segments each including disparity information are created.

  FIG. 17 and FIG. 18 show a structural example (syntax) of an SCS (Subregion Composition segment). FIG. 19 shows the main data definition contents (semantics) of the SCS. This structure includes information of “Sync_byte”, “segment_type”, “page_id”, and “segment_length”. “Segment_type” is 8-bit data indicating the segment type, and is “0x40” indicating SCS here (see FIG. 12). “Segment_length” is 8-bit data indicating the length (size) of the segment.

  FIG. 18 shows a portion including substantial information of the SCS. In this structure example, display control information for the left eye SR and right eye SR, that is, area information for the left eye SR and right eye SR, target frame information, parallax information, and display on / off command information can be transmitted. In this structure example, display control information for an arbitrary number of subregions can be held.

  “Region_id” is 8-bit information indicating an identifier of a region. “Subregion_id” is 8-bit information indicating an identifier of a subregion (Subregion). “Subregion_visible_flag” is 1-bit flag information (command information) for controlling on / off of display (superimposition) of a corresponding subregion. “Subregion_visible_flag = 1” indicates that the display of the corresponding subregion is on, and also indicates that the display of the corresponding subregion displayed before that is off.

  “Subregion_extent_flag” is 1-bit flag information indicating whether or not the subregion and the region are the same in terms of size and position. “Subregion_extent_flag = 1” indicates that the subregion and the region are the same in terms of size and position. On the other hand, “subregion_extent_flag = 0” indicates that the subregion is smaller than the region.

  “Subregion_position_flag” is 1-bit flag information indicating whether or not the following data includes information on the region (position and size) of the subregion. “Subregion_position_flag = 1” indicates that the following data includes information on the region (position and size) of the subregion. On the other hand, “subregion_position_flag = 0” indicates that the following data does not include information on the region (position and size) of the subregion.

  “Target_stereo_frame” is 1-bit information that specifies the target frame (display target frame) of the corresponding subregion. This “target_stereo_frame” constitutes target frame information. “Target_stereo_frame = 0” indicates that the corresponding subregion is displayed in frame 0 (for example, the left eye frame or the base view frame). On the other hand, “target_stereo_frame = 1” indicates that the corresponding subregion is displayed in frame 1 (for example, a right eye frame or a non-base view frame).

  “Rendering_level” indicates a level corresponding to disparity information (disparity) that is essential on the reception side (decoder side) when displaying captions. “00” indicates that subtitle three-dimensional display using parallax information is optional. “01” indicates that subtitle three-dimensional display using disparity information (default_disparity) commonly used within the subtitle display period is essential. “10” indicates that subtitle three-dimensional display using disparity information (disparity_update) sequentially updated within the subtitle display period is essential.

  “Temporal_extension_flag” is 1-bit flag information indicating the presence / absence of disparity information (disparity_update) that is sequentially updated within the caption display period. In this case, “1” indicates that it exists, and “0” indicates that it does not exist. “Shared_disparity” indicates whether to perform common disparity information (disparity) control over all regions. “1” indicates that one common disparity information (disparity) is applied to all subsequent regions. “0” indicates that the disparity information (Disparity) is applied to only one region.

  An 8-bit field of “subregion_disparity” indicates default disparity information. This disparity information is disparity information when updating is not performed, that is, disparity information that is commonly used within a caption display period. When “subregion_position_flag = 1”, information on the following subregion regions (position and size) is included.

  “Subregion_horizontal_position” is 16-bit information indicating the position of the left end of the subregion which is a rectangular region. “Subregion_vertical_position” is 16-bit information indicating the position of the upper end of the subregion which is a rectangular region. “Subregion_width” is 16-bit information indicating the horizontal size (number of pixels) of a subregion which is a rectangular region. “Subregion_height” is 16-bit information indicating the vertical size (number of pixels) of a subregion which is a rectangular region. These position information and size information constitute subregion region information.

  When “temporal_extension_flag” is “1”, “disparity_temporal_extension ()” is included. Here, basically, disparity information to be updated for each base segment period (BSP) is stored. FIG. 20 illustrates an example of updating disparity information for each base segment period (BSP). Here, the base segment period means an update frame interval. As is clear from this figure, the disparity information sequentially updated within the caption display period includes the disparity information of the first frame in the caption display period, and the disparity information of the frame for each subsequent base segment period (updated frame interval). It is made up of.

  FIG. 21 shows a structural example (syntax) of “disparity_temporal_extension ()”. FIG. 22 shows the main data definition contents (semantics). A 2-bit field of “temporal_division_size” indicates the number of frames included in the base segment period (update frame interval). “00” indicates 16 frames. “01” indicates 25 frames. “10” indicates 30 frames. Further, “11” indicates 32 frames.

  A 5-bit field of “temporal_division_count” indicates the number of base segments included in the caption display period. “Disparity_curve_no_update_flag” is 1-bit flag information indicating whether or not disparity information is updated. “1” indicates that the disparity information is not updated at the edge of the corresponding base segment, that is, skipping, and “0” indicates that the disparity information is updated at the edge of the corresponding base segment.

  FIG. 23 illustrates an example of updating disparity information for each base segment period (BSP). In the figure, the disparity information is not updated at the edge of the base segment to which “skip” is attached. When the period in which the change in the frame direction of the disparity information is similar continues for a long time due to the presence of this flag information, disparity information is not updated, and transmission of disparity information within that period can be omitted. It becomes possible to suppress the data amount of information.

  When “disparity_curve_no_update_flag” is “0” and the disparity information is updated, “shifting_interval_counts” of the corresponding base segment is included. On the other hand, when “disparity_curve_no_update_flag” is “1” and disparity information is not updated, “disparity_update” is not included. A 6-bit field of “shifting_interval_counts” indicates a draw factor for adjusting the base segment period (update frame interval), that is, the number of subtracted frames.

  In the example of updating the disparity information for each base segment period (BSP) in FIG. 23, the base segment period is adjusted by the draw factor with respect to the update timing of the disparity information at the time points C to F. The presence of this adjustment information makes it possible to adjust the base segment period (update frame interval), and more accurately convey changes in the time direction (frame direction) of disparity information to the receiving side. .

  In addition, as the adjustment of the base segment period (update frame interval), in addition to the adjustment in the direction of shortening by the number of subtracted frames described above, the adjustment in the direction of increasing by the number of additional frames may be considered. For example, bi-directional adjustment is possible by setting a 5-bit field of “shifting_interval_counts” as a signed integer.

  The 8-bit field of “disparity_update” indicates disparity information of the corresponding base segment. Note that “disparity_update” at k = 0 is an initial value of disparity information that is sequentially updated at update frame intervals within the caption display period, that is, disparity information of the first frame in the caption display period.

  FIG. 24 shows the flow of stereoscopic image data and subtitle data (including display control information) from the broadcast station 100 to the television receiver 300 via the set top box 200 or directly from the broadcast station 100 to the television receiver 300. Is shown schematically. In this case, the broadcast station 100 generates stereoscopic image subtitle data in accordance with a side-by-side format. The stereoscopic image data is included in the video data stream and transmitted, and the stereoscopic image subtitle data is included in the subtitle data stream and transmitted.

  First, a case where stereoscopic image data and subtitle data (including display control information) are sent from the broadcasting station 100 to the set top box 200, and the set top box 200 is a legacy 2D-compatible device (Legacy 2D STB) will be described. To do. The set top box 200 generates region display data for displaying the left eye subtitle and the right eye subtitle based on the subtitle data (excluding subregion display control information), and converts the display data into stereoscopic image data. The output stereoscopic image data is obtained by superimposing. In this case, the overlapping position is the position of the region.

  The set top box 200 transmits the output stereoscopic image data to the television receiver 300 through, for example, an HDMI digital interface. In this case, the transmission format of the stereoscopic image data from the set top box 200 to the television receiver 300 is, for example, a side-by-side system.

  When the television receiver 300 is a 3D-compatible device (3D TV), the left eye on which side-by-side stereoscopic image data sent from the set-top box 200 is subjected to 3D signal processing and a subtitle is superimposed. Image and right eye image data are generated. Then, the television receiver 300 displays binocular parallax images (a left-eye image and a right-eye image) for allowing the user to recognize a stereoscopic image on a display panel such as an LCD.

  Next, a case will be described in which stereoscopic image data and subtitle data (including display control information) are sent from the broadcasting station 100 to the set top box 200, and the set top box 200 is a 3D-compatible device (3D STB). The set top box 200 generates region display data for displaying the left-eye subtitle and the right-eye subtitle based on the subtitle data (excluding subregion display control information). The set top box 200 extracts display data corresponding to the left eye SR and display data corresponding to the right eye SR from the display data of this region.

  Then, the set top box 200 obtains output stereoscopic image data by superimposing display data corresponding to the left eye SR and right eye SR on the stereoscopic image data. In this case, the display data corresponding to the left eye SR is superimposed on the frame portion (left eye image frame portion) indicated by frame0 which is the target frame information of the left eye SR. The display data corresponding to the right eye SR is superimposed on a frame portion (right eye image frame portion) indicated by frame1 which is target frame information of the right eye SR.

  In this case, the display data corresponding to the left eye SR is the position indicated by Position1 which is the region information of the left eye SR in the side-by-side stereoscopic image data, and the disparity1 which is the disparity information of the left eye SR. It is superimposed at a position shifted by half of. The display data corresponding to the left eye SR is the position indicated by Position2 which is the area information of the right eye SR in the side-by-side stereoscopic image data, and the disparity2 which is the disparity information of the left eye SR. It is superimposed at a position shifted by half.

  Then, the set top box 200 transmits the output stereoscopic image data obtained as described above to the television receiver 300 through, for example, an HDMI digital interface. In this case, the transmission format of the stereoscopic image data from the set top box 200 to the television receiver 300 is, for example, a side-by-side system.

  When the television receiver 300 is a 3D-compatible device (3D TV), the left eye on which side-by-side stereoscopic image data sent from the set-top box 200 is subjected to 3D signal processing and a subtitle is superimposed. Image and right eye image data are generated. Then, the television receiver 300 displays binocular parallax images (a left-eye image and a right-eye image) for allowing the user to recognize a stereoscopic image on a display panel such as an LCD.

  Next, a case where stereoscopic image data and subtitle data (including display control information) are sent from the broadcast station 100 to the television receiver 300, and the television receiver 300 is a 3D-compatible device (3D TV) will be described. The television receiver 300 generates region display data for displaying the left-eye subtitle and the right-eye subtitle based on the subtitle data (excluding subregion display control information). Then, the television receiver 300 extracts display data corresponding to the left eye SR and display data (right eye display data) corresponding to the right eye SR from the display data of this region.

  The television receiver 300 obtains full-resolution left-eye display data by scaling display data corresponding to the left eye SR twice in the horizontal direction. Then, the television receiver 300 superimposes the left-eye display data on full-resolution left-eye image data corresponding to frame0 that is target frame information of the left eye SR. That is, the television receiver 300 performs full-resolution left-eye image data obtained by scaling the left-eye display data by horizontally doubling the left-eye image portion of the side-by-side stereoscopic image data. The left eye image data with the subtitle superimposed is generated.

  The television receiver 300 scales the display data corresponding to the right eye SR twice in the horizontal direction to obtain right-eye display data corresponding to full resolution. Then, the television receiver 300 superimposes the right-eye display data on the full-resolution right-eye image data corresponding to the frame 1 that is the target frame information of the right eye SR. That is, the television receiver 300 uses the right-eye display data obtained by scaling the right-eye image portion of the stereoscopic image data of the side-by-side method twice in the horizontal direction in the horizontal direction. The right eye image data on which the subtitle is superimposed is generated.

  In this case, in the left-eye display data, the position where Position1 that is the region information of the left eye SR is doubled in the left-eye image data of the full resolution is shifted by Disparity1 that is the disparity information of the left eye SR. It is superimposed on the position. Further, in this case, the right-eye display data is the parallax of the left-eye SR at a position that is doubled by subtracting H / 2 from Position2 that is the area information of the right-eye SR of the full-resolution right-eye image data. It is superimposed at a position shifted by the information Disparity2

  The television receiver 300 uses a binocular parallax image for allowing a user to recognize a stereoscopic image on a display panel such as an LCD based on the left-eye image data and the right-eye image data on which the subtitles generated as described above are superimposed. (Left eye image and right eye image) are displayed.

  FIG. 25 shows a flow of stereoscopic image data and subtitle data (including display control information) from the broadcasting station 100 to the television receiver 300 via the set top box 200 or directly from the broadcasting station 100 to the television receiver 300. Is shown schematically. In this case, the broadcast station 100 generates stereoscopic image subtitle data that conforms to the MVC (Multi-view Video Coding) method. In this case, stereoscopic image data is composed of base-view image data (left-eye image data) and non-base-view image data (right-eye image data). The stereoscopic image data is included in the video data stream and transmitted, and the stereoscopic image subtitle data is included in the subtitle data stream and transmitted.

  First, a case where stereoscopic image data and subtitle data (including display control information) are sent from the broadcasting station 100 to the set top box 200, and the set top box 200 is a legacy 2D-compatible device (Legacy 2D STB) will be described. To do. The set top box 200 generates region display data for displaying the left-eye subtitle and the right-eye subtitle based on the subtitle data (excluding subregion display control information), and uses this display data as a base view (left The output image data is obtained by superimposing it on the eye image data. In this case, the overlapping position is the position of the region.

  The set top box 200 transmits the output image data to the television receiver 300 through, for example, an HDMI digital interface. The television receiver 300 displays a 2D image on the display panel regardless of whether the device is a 2D compatible device (2D TV) or a 3D compatible device (3D TV).

  Next, a case will be described in which stereoscopic image data and subtitle data (including display control information) are sent from the broadcasting station 100 to the set top box 200, and the set top box 200 is a 3D-compatible device (3D STB). The set top box 200 generates region display data for displaying the left-eye subtitle and the right-eye subtitle based on the subtitle data (excluding subregion display control information). The set top box 200 extracts display data corresponding to the left eye SR and display data corresponding to the right eye SR from the display data of this region.

  The set top box 200 superimposes the display data corresponding to the left eye SR on the image data of the base view (left eye image) indicated by the frame 0 that is the target frame information of the left eye SR, and superimposes the left eye subtitle. Output image data of the base view (left eye image) is obtained. In this case, the display data corresponding to the left eye SR is the position indicated by Position1 which is the region information of the left eye SR in the image data of the base view (left eye image), and Disparity1 which is the disparity information of the left eye SR. It is superimposed at a position shifted by the amount.

  Further, the set-top box 200 superimposes display data corresponding to the right eye SR on the image data of the non-base view (right eye image) indicated by the frame 1 that is the target frame information of the right eye SR, so that the right eye Output image data of a non-base view (left eye image) on which a subtitle is superimposed is obtained. In this case, the display data corresponding to the right eye SR is the disparity information of the right eye SR at the position indicated by Position2 which is the area information of the right eye SR in the image data of the non-base view (right eye image). It is superimposed at a position shifted by Disparity2.

  The set-top box 200 transmits the image data of the base view (left eye image) and the non-base view (right eye image) obtained as described above to the television receiver 300 through, for example, an HDMI digital interface. To do. In this case, the transmission format of the stereoscopic image data from the set top box 200 to the television receiver 300 is, for example, a frame packing system.

  When the television receiver 300 is a 3D-compatible device (3D TV), the left-eye image and the right image on which the subtitle is superimposed by performing 3D signal processing on the frame packing type stereoscopic image data transmitted from the set-top box 200. Generate eye image data. Then, the television receiver 300 displays binocular parallax images (a left-eye image and a right-eye image) for allowing the user to recognize a stereoscopic image on a display panel such as an LCD.

  Next, a case where stereoscopic image data and subtitle data (including display control information) are sent from the broadcast station 100 to the television receiver 300, and the television receiver 300 is a 3D-compatible device (3D TV) will be described. The television receiver 300 generates region display data for displaying the left-eye subtitle and the right-eye subtitle based on the subtitle data (excluding subregion display control information). Then, the television receiver 300 extracts display data corresponding to the left eye SR and display data corresponding to the right eye SR from the display data of this region.

  The television receiver 300 superimposes the display data corresponding to the left eye SR on the image data of the base view (left eye image) indicated by frame0 that is the target frame information of the left eye SR, and superimposes the left eye subtitle. Output image data of the base view (left eye image) is obtained. In this case, the display data corresponding to the left eye SR is the position indicated by Position1 which is the region information of the left eye SR in the image data of the base view (left eye image), and Disparity1 which is the disparity information of the left eye SR. It is superimposed at a position shifted by the amount.

  Also, the television receiver 300 superimposes display data corresponding to the right eye SR on the image data of the non-base view (right eye image) indicated by frame1 that is the target frame information of the right eye SR, so that the right eye Output image data of a non-base view (left eye image) on which a subtitle is superimposed is obtained. In this case, the display data corresponding to the right eye SR is the disparity information of the right eye SR at the position indicated by Position2 which is the area information of the right eye SR in the image data of the non-base view (right eye image). It is superimposed at a position shifted by Disparity2.

  Based on the image data of the base view (left eye image) and the non-base view (right eye image) on which the subtitles generated as described above are superimposed, the television receiver 300 displays a stereoscopic image to the user on the display panel such as an LCD. A binocular parallax image (left eye image and right eye image) for recognizing the image is displayed.

  In the above description, the example in which the display control information (region information, target frame information, parallax information) for the left eye SR and the right eye SR is created separately has been shown. However, it is also conceivable to create only display control information for one of the left eye SR and the right eye SR, for example, the left eye SR. In this case, the display control information for the left eye SR does not include region information among the region information, target frame information, and disparity information for the right eye SR, but includes target frame information and disparity information.

  FIG. 26 illustrates stereoscopic image data and subtitle data (display control information) from the broadcasting station 100 to the television receiver 300 via the set-top box 200 or directly from the broadcasting station 100 to the television receiver 300 in that case. An example of the flow of In this case, the broadcast station 100 generates stereoscopic image subtitle data in accordance with a side-by-side format. The stereoscopic image data is included in the video data stream and transmitted, and the stereoscopic image subtitle data is included in the subtitle data stream and transmitted.

  First, a case where stereoscopic image data and subtitle data (including display control information) are sent from the broadcasting station 100 to the set top box 200, and the set top box 200 is a legacy 2D-compatible device (Legacy 2D STB) will be described. To do. The set top box 200 generates region display data for displaying the left eye subtitle and the right eye subtitle based on the subtitle data (excluding subregion display control information), and converts the display data into stereoscopic image data. The output stereoscopic image data is obtained by superimposing. In this case, the overlapping position is the position of the region.

  The set top box 200 transmits the output stereoscopic image data to the television receiver 300 through, for example, an HDMI digital interface. In this case, the transmission format of the stereoscopic image data from the set top box 200 to the television receiver 300 is, for example, a side-by-side system.

  When the television receiver 300 is a 3D-compatible device (3D TV), the left eye on which side-by-side stereoscopic image data sent from the set-top box 200 is subjected to 3D signal processing and a subtitle is superimposed. Image and right eye image data are generated. Then, the television receiver 300 displays binocular parallax images (a left-eye image and a right-eye image) for allowing the user to recognize a stereoscopic image on a display panel such as an LCD.

  Next, a case will be described in which stereoscopic image data and subtitle data (including display control information) are sent from the broadcasting station 100 to the set top box 200, and the set top box 200 is a 3D-compatible device (3D STB). The set top box 200 generates region display data for displaying the left-eye subtitle and the right-eye subtitle based on the subtitle data (excluding subregion display control information). Then, the set top box 200 extracts display data corresponding to the left eye SR from the display data of this region.

  Then, the set top box 200 obtains output stereoscopic image data by superimposing display data corresponding to the left eye SR on the stereoscopic image data. In this case, the display data corresponding to the left eye SR is superimposed on the frame portion (left eye image frame portion) indicated by frame0 which is the target frame information of the left eye SR. Further, the display data corresponding to the left eye SR is superimposed on a frame portion (right eye image frame portion) indicated by frame1 which is target frame information of the right eye SR.

  In this case, the display data corresponding to the left eye SR shifts the position indicated by Position, which is the region information, of the side-by-side stereoscopic image data by half of Disparity1, which is the parallax information of the left eye SR. Is superimposed on the selected position. In addition, the display data corresponding to the left eye SR shifts the position indicated by Position + H / 2, which is the region information, of the side-by-side stereoscopic image data by half of Disparity2, which is the parallax information of the right eye SR. Is superimposed on the selected position.

  Then, the set top box 200 transmits the output stereoscopic image data obtained as described above to the television receiver 300 through, for example, an HDMI digital interface. In this case, the transmission format of the stereoscopic image data from the set top box 200 to the television receiver 300 is, for example, a side-by-side system.

  When the television receiver 300 is a 3D-compatible device (3D TV), the left eye on which side-by-side stereoscopic image data sent from the set-top box 200 is subjected to 3D signal processing and a subtitle is superimposed. Image and right eye image data are generated. Then, the television receiver 300 displays binocular parallax images (a left-eye image and a right-eye image) for allowing the user to recognize a stereoscopic image on a display panel such as an LCD.

  Next, a case where stereoscopic image data and subtitle data (including display control information) are sent from the broadcast station 100 to the television receiver 300, and the television receiver 300 is a 3D-compatible device (3D TV) will be described. The television receiver 300 generates region display data for displaying the left-eye subtitle and the right-eye subtitle based on the subtitle data (excluding subregion display control information). Then, the television receiver 300 extracts display data corresponding to the left eye SR from the display data of this region.

  The television receiver 300 obtains full-resolution left-eye display data by scaling display data corresponding to the left eye SR twice in the horizontal direction. Then, the television receiver 300 superimposes the left-eye display data on the full-resolution left-eye image data corresponding to the frame 0 that is target frame information. That is, the television receiver 300 performs full-resolution left-eye image data obtained by scaling the left-eye display data by horizontally doubling the left-eye image portion of the side-by-side stereoscopic image data. The left eye image data with the subtitle superimposed is generated.

  In addition, the television receiver 300 scales the display data corresponding to the left eye SR twice in the horizontal direction to obtain right-eye display data corresponding to full resolution. Then, the television receiver 300 superimposes the right-eye display data on the full-resolution right-eye image data corresponding to the target frame information frame1. That is, the television receiver 300 uses the right-eye display data obtained by scaling the right-eye image portion of the stereoscopic image data of the side-by-side method twice in the horizontal direction in the horizontal direction. The right eye image data on which the subtitle is superimposed is generated.

  In this case, the left-eye display data is superimposed on a position where the position that is the region information of the full-resolution left-eye image data is doubled by the Disparity 1 that is the disparity information. In this case, the right-eye display data is superimposed at a position where the position of the position information, which is the region information, is doubled by the amount of Disparity 2 of the disparity information in the right-eye image data of full resolution.

  The television receiver 300 uses a binocular parallax image for allowing a user to recognize a stereoscopic image on a display panel such as an LCD based on the left-eye image data and the right-eye image data on which the subtitles generated as described above are superimposed. (Left eye image and right eye image) are displayed.

  In the transmission data generation unit 110 illustrated in FIG. 2, the bit stream data BSD output from the multiplexer 119 is a multiplexed data stream including a video data stream and a subtitle data stream. The video data stream includes stereoscopic image data. Further, the subtitle data stream includes subtitle data for stereoscopic images (for 3D images) corresponding to the transmission format of the stereoscopic image data.

  The stereoscopic image subtitle data includes left-eye subtitle data and right-eye subtitle data. Therefore, on the receiving side, based on the subtitle data, the display data of the left eye subtitle superimposed on the left eye image data included in the stereoscopic image data and the display of the right eye subtitle superimposed on the right eye image data included in the stereoscopic image data are displayed. Data can be generated easily. This facilitates processing.

  In the transmission data generation unit 110 illustrated in FIG. 2, the bit stream data BSD output from the multiplexer 119 includes display control information in addition to stereoscopic image data and stereoscopic image subtitle data. This display control information includes display control information (region information, target frame information, parallax information) related to the left eye SR and the right eye SR.

  Therefore, on the receiving side, it is easy to superimpose and display only the left eye subtitle in the left eye SR and the subtitle in the right eye SR on the target frame. Then, parallax can be given to the display positions of the left eye subtitle in the left eye SR and the subtitle in the right eye SR, and in the display of the subtitle (caption), the consistency of perspective with each object in the image is improved. It becomes possible to maintain the optimum state.

  In the transmission data generation unit 110 shown in FIG. 2, the subtitle processing unit 123 can transmit the SCS segment including the disparity information sequentially updated in the subtitle display period, so the left eye subtitle and the right eye in the left eye SR can be transmitted. The display position of the right-eye subtitle in SR can be dynamically controlled. As a result, on the receiving side, the parallax provided between the left eye subtitle and the right eye subtitle can be dynamically changed in conjunction with the change in the image content.

  In the transmission data generation unit 110 illustrated in FIG. 2, the disparity information included in the SCS segment created by the subtitle processing unit 116 includes disparity information of the first frame in the subtitle display period and each update frame interval thereafter. It is assumed to consist of disparity information of the frame. Therefore, the amount of transmission data can be reduced, and the memory capacity for holding the parallax information can be greatly saved on the receiving side.

  Further, in the transmission data generation unit 110 illustrated in FIG. 2, the disparity information of the frame for each update frame interval included in the SCS segment created by the subtitle processing unit 116 is not an offset value from the previous disparity information, but the disparity information. Information itself. Therefore, even if an error occurs in the interpolation process on the receiving side, it is possible to recover from the error within a certain delay time.

  In addition, in the transmission data generation unit 110 illustrated in FIG. 2, the disparity information included in the SCS segment created by the subtitle processing unit 116 has integer pixel accuracy. For this reason, a difference in capability between receivers is unlikely to occur, and therefore, a difference does not open between different receivers as time elapses. In addition, since interpolation between update frame intervals is given by the receiver performance, the degree of freedom in receiver design is increased.

[Description of Set Top Box]
Returning to FIG. 1, the set-top box 200 receives bit stream data (transport stream) BSD transmitted from the broadcasting station 100 on broadcast waves. The bit stream data BSD includes stereoscopic image data and audio data including left eye image data and right eye image data. The bit stream data BSD includes stereoscopic image subtitle data (including display control information) for displaying a subtitle (caption).

  The set top box 200 has a bit stream processing unit 201. The bit stream processing unit 201 extracts stereoscopic image data, audio data, and subtitle data from the bit stream data BSD. The bit stream processing unit 201 generates stereoscopic image data on which the subtitle is superimposed using stereoscopic image data, subtitle data, and the like.

  In this case, parallax can be given between the left eye subtitle superimposed on the left eye image and the right eye subtitle superimposed on the right eye image. For example, as described above, the subtitle data for stereoscopic images transmitted from the broadcast station 100 can be generated so that parallax is given between the left eye subtitle and the right eye subtitle. Further, for example, as described above, the display control information added to the stereoscopic image subtitle data transmitted from the broadcast station 100 includes disparity information, and based on this disparity information, Parallax can be given between the eye subtitle and the right eye subtitle. In this manner, by providing parallax between the left eye subtitle and the right eye subtitle, the user can recognize the subtitle (caption) in front of the image.

  FIG. 27A shows a display example of a subtitle (caption) on an image. In this display example, captions are superimposed on an image composed of a background and a foreground object. FIG. 27B shows the perspective of the background, the foreground object, and the caption, and indicates that the caption is recognized most forward.

  FIG. 28A shows a display example of subtitles (captions) on the same image as FIG. 27A. FIG. 28B shows a left-eye caption LGI superimposed on the left-eye image and a right-eye caption RGI superimposed on the right-eye image. FIG. 28C shows that a parallax is given between the left-eye caption LGI and the right-eye caption RGI because the caption is recognized most forward.

[Configuration example of set-top box]
A configuration example of the set top box 200 will be described. FIG. 29 shows a configuration example of the set top box 200. The set top box 200 includes a bit stream processing unit 201, an HDMI terminal 202, an antenna terminal 203, a digital tuner 204, a video signal processing circuit 205, an HDMI transmission unit 206, and an audio signal processing circuit 207. ing. The set top box 200 includes a CPU 211, a flash ROM 212, a DRAM 213, an internal bus 214, a remote control receiving unit 215, and a remote control transmitter 216.

  The antenna terminal 203 is a terminal for inputting a television broadcast signal received by a receiving antenna (not shown). The digital tuner 204 processes the television broadcast signal input to the antenna terminal 203 and outputs predetermined bit stream data (transport stream) BSD corresponding to the user's selected channel.

  As described above, the bit stream processing unit 201 extracts stereoscopic image data, audio data, stereoscopic image subtitle data (including display control information), and the like from the bit stream data BSD. The bit stream processing unit 201 outputs audio data. Further, the bit stream processing unit 201 combines the display data of the left eye subtitle and the right eye subtitle with the stereoscopic image data, and obtains output stereoscopic image data on which the subtitle is superimposed. The display control information includes left eye SR and right eye SR area information, target frame information, and parallax information.

  In this case, the bitstream processing unit 201 generates region display data for displaying the left-eye subtitle and the right-eye subtitle based on the subtitle data (excluding subregion display control information). Then, the bit stream processing unit 201 extracts display data corresponding to the left eye SR and display data corresponding to the right eye SR from the display data of this region based on the region information of the left eye SR and the right eye SR. .

  Then, the bit stream processing unit 201 superimposes display data corresponding to the left eye SR and right eye SR on the stereoscopic image data to obtain output stereoscopic image data (display stereoscopic image data). In this case, the display data corresponding to the left eye SR is superimposed on the frame portion (left eye image frame portion) indicated by frame0 which is the target frame information of the left eye SR. The display data corresponding to the right eye SR is superimposed on a frame portion (right eye image frame portion) indicated by frame1 which is target frame information of the right eye SR. At this time, the bit stream processing unit 201 shift-adjusts the display positions (superimposition positions) of the left eye subtitle in the left eye SR and the right eye subtitle in the right eye SR based on the parallax information.

  The video signal processing circuit 205 performs image quality adjustment processing on the output stereoscopic image data obtained by the bit stream processing unit 201 as necessary, and supplies the processed output stereoscopic image data to the HDMI transmission unit 206. The audio signal processing circuit 207 performs sound quality adjustment processing or the like on the audio data output from the bit stream processing unit 201 as necessary, and supplies the processed audio data to the HDMI transmission unit 206.

  The HDMI transmission unit 206 sends, for example, uncompressed image data and audio data from the HDMI terminal 202 by communication conforming to HDMI. In this case, since transmission is performed using an HDMI TMDS channel, image data and audio data are packed and output from the HDMI transmission unit 206 to the HDMI terminal 202.

  For example, when the transmission format of stereoscopic image data from the broadcasting station 100 is a side-by-side format, the TMDS transmission format is a side-by-side format (see FIG. 24). For example, when the transmission format of stereoscopic image data from the broadcasting station 100 is the top-and-bottom method, the TMDS transmission format is the top-and-bottom method. For example, when the transmission format of the stereoscopic image data from the broadcasting station 100 is the MVC method, the TMDS transmission format is a frame packing method (see FIG. 25).

  The CPU 211 controls the operation of each part of the set top box 200. The flash ROM 212 stores control software and data. The DRAM 213 constitutes a work area for the CPU 211. The CPU 211 develops software and data read from the flash ROM 212 on the DRAM 213 to activate the software, and controls each part of the set top box 200.

  The remote control receiving unit 215 receives the remote control signal (remote control code) transmitted from the remote control transmitter 216 and supplies it to the CPU 211. The CPU 211 controls each part of the set top box 200 based on the remote control code. The CPU 211, flash ROM 212 and DRAM 213 are connected to the internal bus 214.

  The operation of the set top box 200 will be briefly described. A television broadcast signal input to the antenna terminal 203 is supplied to the digital tuner 204. The digital tuner 204 processes the television broadcast signal and outputs predetermined bit stream data (transport stream) BSD corresponding to the user's selected channel.

  The bit stream data BSD output from the digital tuner 204 is supplied to the bit stream processing unit 201. The bit stream processing unit 201 extracts stereoscopic image data, audio data, stereoscopic image subtitle data (including display control information), and the like from the bit stream data BSD. In the bit stream processing unit 201, display data (bitmap data) of the left eye subtitle and the right eye subtitle is combined with the stereoscopic image data, and output stereoscopic image data on which the subtitle is superimposed is obtained.

  The output stereoscopic image data obtained by the bit stream processing unit 201 is supplied to the video signal processing circuit 205. In the video signal processing circuit 205, image quality adjustment processing or the like is performed on the output stereoscopic image data as necessary. The processed output stereoscopic image data output from the video signal processing circuit 205 is supplied to the HDMI transmission unit 206.

  Also, the audio data obtained by the bit stream processing unit 201 is supplied to the audio signal processing circuit 207. The audio signal processing circuit 207 performs processing such as sound quality adjustment processing on the audio data as necessary. The processed audio data output from the audio signal processing circuit 207 is supplied to the HDMI transmission unit 206. The stereoscopic image data and audio data supplied to the HDMI transmission unit 206 are transmitted from the HDMI terminal 202 to the HDMI cable 400 via the HDMI TMDS channel.

[Configuration example of bit stream processing unit]
FIG. 30 shows a configuration example of the bit stream processing unit 201. The bit stream processing unit 201 has a configuration corresponding to the transmission data generation unit 110 shown in FIG. The bit stream processing unit 201 includes a demultiplexer 221, a video decoder 222, and an audio decoder 229. The bit stream processing unit 201 also includes a subtitle decoder 223, a stereoscopic image subtitle generation unit 224, a display control unit 225, a display control information acquisition unit 226, a parallax information processing unit 227, and a video superimposition unit 228. Have.

  The demultiplexer 221 extracts video, audio, and subtitle packets from the bit stream data BSD, and sends them to each decoder. The demultiplexer 221 extracts information such as PMT and EIT inserted in the bit stream data BSD, and sends the information to the CPU 211. As described above, Stream_content ('0x03' = DVB subtitles) & Component_type (for 3Dtarget) is described in the component descriptor under the EIT. Thereby, it can be identified that the subtitle data stream includes subtitle data for stereoscopic images. Therefore, the CPU 211 can identify that the subtitle data stream includes the subtitle data for stereoscopic images based on this description.

  The video decoder 222 performs processing reverse to that of the video encoder 112 of the transmission data generation unit 110 described above. In other words, a video data stream is reconstructed from the video packets extracted by the demultiplexer 221, and decoding processing is performed to obtain stereoscopic image data including left eye image data and right eye image data. The transmission format of the stereoscopic image data is, for example, a side-by-side method, a top-and-bottom method, a frame-sequential method, an MVC method, or the like.

  The subtitle decoder 223 performs processing reverse to that of the subtitle encoder 118 of the transmission data generation unit 110 described above. That is, the subtitle decoder 223 reconstructs a subtitle data stream from the subtitle packet extracted by the demultiplexer 221 and performs a decoding process to obtain stereoscopic image subtitle data (including display control information). The stereoscopic image subtitle generating unit 224 generates display data (bitmap data) of the left eye subtitle and the right eye subtitle to be superimposed on the stereoscopic image data based on the stereoscopic image subtitle data (excluding display control information). . The stereoscopic image subtitle generating unit 224 constitutes a display data generating unit.

  The display control unit 225 controls display data to be superimposed on stereoscopic image data based on display control information (left eye SR, right eye SR region information, target frame information, and parallax information). That is, the display control unit 225 corresponds to the left eye SR from the display data (bitmap data) of the left eye subtitle and the right eye subtitle superimposed on the stereoscopic image data based on the region information of the left eye SR and the right eye SR. And display data corresponding to the right eye SR is extracted.

  In addition, the display control unit 225 supplies display data corresponding to the left eye SR and right eye SR to the video superimposing unit 228 and superimposes them on the stereoscopic image data. In this case, the display data corresponding to the left eye SR is superimposed on the frame portion (left eye image frame portion) indicated by frame0 which is the target frame information of the left eye SR. The display data corresponding to the right eye SR is superimposed on a frame portion (right eye image frame portion) indicated by frame1 which is target frame information of the right eye SR. At this time, the display control unit 225 shifts and adjusts the display positions (superimposition positions) of the left eye subtitle in the left eye SR and the right eye subtitle in the right eye SR based on the parallax information, Parallax is given between right-eye subtitles.

  The display control information extraction unit 226 acquires display control information (region information, target frame information, parallax information) from the subtitle data stream. This display control information includes disparity information (see “subregion_disparity” in FIG. 18) that is commonly used within the caption display period. Further, the display control information may further include disparity information (see “disparity_update” in FIG. 21) that is sequentially updated within the caption display period. As described above, the disparity information sequentially updated within the caption display period is composed of the disparity information of the first frame in the caption display period and the disparity information of the frame for each subsequent base segment period (update frame interval). ing.

  The disparity information processing unit 227 sends the region information and target frame information included in the display control information, as well as the disparity information used in common within the caption display period, to the display control unit 225 as it is. On the other hand, the disparity information processing unit 227 performs interpolation processing on the disparity information sequentially updated within the caption display period, and generates disparity information at an arbitrary frame interval, for example, one frame interval within the caption display period. To the display control unit 225.

  As the interpolation processing, the disparity information processing unit 227 performs not the linear interpolation processing but the interpolation processing with the low-pass filter (LPF) processing in the time direction (frame direction), and the disparity information at a predetermined frame interval after the interpolation processing. The time direction (frame direction) is made gentle. FIG. 31 illustrates an example of interpolation processing with the above-described LPF processing in the parallax information processing unit 227. This example corresponds to the parallax information update example of FIG. 23 described above.

  Here, when only the disparity information (disparity vector) used in common within the caption display period is sent from the disparity information processing unit 227, the display control unit 225 described above uses the disparity information. Further, the display control unit 225 uses one of the parallax information that is sequentially updated within the caption display period from the parallax information processing unit 227.

  Which one is used is, for example, as described above, information indicating the disparity information (disparity) correspondence level that is included in the extended display control data unit and is essential on the reception side (decoder side) when displaying captions. (See “rendering_level” in FIG. 18). In this case, for example, when “00”, it depends on the user setting. By using the disparity information sequentially updated within the caption display period, it is possible to dynamically change the disparity to be given to the left eye subtitle and the right eye subtitle in conjunction with the change of the image content.

  The video superimposing unit 228 obtains output stereoscopic image data Vout. In this case, the video superimposing unit 228 corresponds to the display data (bitmap data) of the left eye SR and the right eye SR that are shift-adjusted by the display control unit 225 with respect to the stereoscopic image data obtained by the video decoder 222. Superimpose on the target frame part. Then, the video superimposing unit 228 outputs the output stereoscopic image data Vout to the outside of the bit stream processing unit 201.

  Also, the audio decoder 229 performs a process reverse to that of the audio encoder 113 of the transmission data generation unit 110 described above. That is, the audio decoder 229 reconstructs an audio elementary stream from the audio packet extracted by the demultiplexer 221 and performs a decoding process to obtain audio data Aout. The audio decoder 229 outputs the audio data Aout to the outside of the bit stream processing unit 201.

  The operation of the bit stream processing unit 201 illustrated in FIG. 30 will be briefly described. The bit stream data BSD output from the digital tuner 204 (see FIG. 29) is supplied to the demultiplexer 221. In the demultiplexer 221, video, audio, and subtitle packets are extracted from the bit stream data BSD and supplied to each decoder.

  In the video decoder 222, a video data stream is reconstructed from the video packet extracted by the demultiplexer 221, and further, decoding processing is performed to obtain stereoscopic image data including left eye image data and right eye image data. . The stereoscopic image data is supplied to the video superimposing unit 226.

  Further, the subtitle decoder 223 reconstructs a subtitle data stream from the subtitle packet extracted by the demultiplexer 221 and further performs a decoding process to obtain stereoscopic image subtitle data (including display control information). It is done. This subtitle data is supplied to the stereoscopic image subtitle generating unit 224.

  The stereoscopic image subtitle generation unit 224 generates display data (bitmap data) of the left eye subtitle and the right eye subtitle to be superimposed on the stereoscopic image data based on the stereoscopic image subtitle data (excluding display control information). The This display data is supplied to the display control unit 225.

  In addition, the display control information acquisition unit 226 acquires display control information (region information, target frame information, parallax information) from the subtitle data stream. This display control information is supplied to the display control unit 225 through the parallax information processing unit 227. At this time, the disparity information processing unit 227 performs the following processing on disparity information that is sequentially updated within the caption display period. That is, the disparity information processing unit 227 performs interpolation processing with LPF processing in the time direction (frame direction), and generates disparity information at an arbitrary frame interval, for example, one frame interval within the caption display period. To the display control unit 225.

  The display control unit 225 controls superimposition of display data on stereoscopic image data based on display control information (left eye SR, right eye SR region information, target frame information, and parallax information). That is, the display data of the left eye SR and the right eye SR is extracted from the display data generated by the stereoscopic image subtitle generation unit 224, and shift adjustment is performed. Thereafter, the display data of the left eye SR and the right eye SR that have been subjected to the shift adjustment is supplied to the video superimposing unit 228 so as to be superimposed on the target frame of the stereoscopic image data.

  The video superimposing unit 228 superimposes the display data shift-adjusted by the display control unit 225 on the stereoscopic image data obtained by the video decoder 222 to obtain output stereoscopic image data Vout. The output stereoscopic image data Vout is output to the outside of the bit stream processing unit 201.

  Also, the audio decoder 229 reconstructs an audio elementary stream from the audio packet extracted by the demultiplexer 221, further performs a decoding process, and the audio data Aout corresponding to the display stereoscopic image data Vout described above. Is obtained. The audio data Aout is output to the outside of the bit stream processing unit 201.

  In the set top box 200 shown in FIG. 29, the bit stream data BSD output from the digital tuner 204 is a multiplexed data stream having a video data stream and a subtitle data stream. The video data stream includes stereoscopic image data. Further, the subtitle data stream includes subtitle data for stereoscopic images (for 3D images) corresponding to the transmission format of the stereoscopic image data.

  The stereoscopic image subtitle data includes left-eye subtitle data and right-eye subtitle data. Therefore, the stereoscopic image subtitle generating unit 224 of the bitstream processing unit 201 can easily generate display data for the left eye subtitle to be superimposed on the left eye image data included in the stereoscopic image data. Further, the stereoscopic image subtitle generating unit 224 of the bitstream processing unit 201 can easily generate display data of the right eye subtitle to be superimposed on the right eye image data included in the stereoscopic image data. This facilitates processing.

  In the set top box 200 shown in FIG. 29, the bit stream data BSD output from the digital tuner 204 includes display control information in addition to stereoscopic image data and stereoscopic image subtitle data. This display control information includes display control information (region information, target frame information, parallax information) related to the left eye SR and the right eye SR. Therefore, it becomes easy to superimpose and display only the left eye subtitle in the left eye SR and the subtitle in the right eye SR on the target frame. Further, parallax can be given to the display positions of the left eye subtitle in the left eye SR and the subtitle in the right eye SR, and the consistency of perspective with each object in the image is displayed in the display of the subtitle (caption). It becomes possible to maintain the optimum state.

  Also, in the set-top box 200 shown in FIG. 29, when the display control information acquired by the display control information acquisition unit 226 of the bitstream processing unit 201 includes disparity information that is sequentially updated within the caption display period, display control is performed. The unit 225 can dynamically control the display positions of the left eye subtitle in the left eye SR and the right eye subtitle in the right eye SR. Thereby, the parallax provided between the left eye subtitle and the right eye subtitle can be dynamically changed in conjunction with the change of the image content.

  Also, in the set-top box 200 shown in FIG. 29, the disparity of a plurality of frames constituting disparity information sequentially updated within the caption display period (a predetermined number of frame periods) by the disparity information processing unit 227 of the bitstream processing unit 201. Interpolation processing is performed on the information. In this case, even when disparity information is transmitted from the transmission side every update frame interval, the disparity provided between the left eye subtitle and the right eye subtitle is controlled at a fine interval, for example, for each frame. Is possible.

  In the set top box 200 shown in FIG. 29, the interpolation processing in the disparity information processing unit 227 of the bit stream processing unit 201 is accompanied by, for example, low-pass filter processing in the time direction (frame direction). Therefore, even when disparity information is transmitted from the transmission side every update frame interval, the change in the time direction of the disparity information after the interpolation process can be gently performed, and is provided between the left eye subtitle and the right eye subtitle. It is possible to suppress a sense of incongruity due to discontinuity in the disparity transitions at every update frame interval.

[Description of TV receiver]
Returning to FIG. 1, the television receiver 300 receives stereoscopic image data sent from the set top box 200 via the HDMI cable 400. The television receiver 300 includes a 3D signal processing unit 301. The 3D signal processing unit 301 performs processing (decoding processing) corresponding to the transmission format on the stereoscopic image data to generate left-eye image data and right-eye image data.

[Configuration example of TV receiver]
A configuration example of the television receiver 300 will be described. FIG. 32 illustrates a configuration example of the television receiver 300. The television receiver 300 includes a 3D signal processing unit 301, an HDMI terminal 302, an HDMI receiving unit 303, an antenna terminal 304, a digital tuner 305, and a bit stream processing unit 306.

  The television receiver 300 includes a video / graphic processing circuit 307, a panel drive circuit 308, a display panel 309, an audio signal processing circuit 310, an audio amplification circuit 311, and a speaker 312. In addition, the television receiver 300 includes a CPU 321, a flash ROM 322, a DRAM 323, an internal bus 324, a remote control receiving unit 325, and a remote control transmitter 326.

  The antenna terminal 304 is a terminal for inputting a television broadcast signal received by a receiving antenna (not shown). The digital tuner 305 processes the television broadcast signal input to the antenna terminal 304 and outputs predetermined bit stream data (transport stream) BSD corresponding to the user's selected channel. The bit stream processing unit 306 extracts stereoscopic image data, audio data, stereoscopic image subtitle data (including display control information), and the like from the bit stream data BSD.

  The bit stream processing unit 306 is configured in the same manner as the bit stream processing unit 201 of the set top box 200. The bit stream processing unit 306 combines the display data of the left eye subtitle and the right eye subtitle with the stereoscopic image data, and generates and outputs output stereoscopic image data on which the subtitle is superimposed. Note that the bit stream processing unit 306 performs scaling processing, for example, when the transmission format of the stereoscopic image data is a side-by-side method or a top-and-bottom method, and performs full-resolution left-eye image data and Right-eye image data is output (see the portion of the television receiver 300 in FIGS. 24 to 26). The bit stream processing unit 306 outputs audio data.

  The HDMI receiving unit 303 receives uncompressed image data and audio data supplied to the HDMI terminal 302 via the HDMI cable 400 by communication conforming to HDMI. The HDMI receiving unit 303 has a version of, for example, HDMI 1.4a, and can handle stereoscopic image data.

  The 3D signal processing unit 301 performs decoding processing on the stereoscopic image data received by the HDMI receiving unit 303 to generate full-resolution left-eye image data and right-eye image data. The 3D signal processing unit 301 performs a decoding process corresponding to the TMDS transmission data format. Note that the 3D signal processing unit 301 does nothing with the full-resolution left-eye image data and right-eye image data obtained by the bit stream processing unit 306.

  The video / graphic processing circuit 307 generates image data for displaying a stereoscopic image based on the left-eye image data and the right-eye image data generated by the 3D signal processing unit 301. The video / graphic processing circuit 307 performs image quality adjustment processing on the image data as necessary. Further, the video / graphic processing circuit 307 synthesizes superimposition information data such as a menu and a program guide with the image data as necessary. The panel drive circuit 308 drives the display panel 309 based on the image data output from the video / graphic processing circuit 307. The display panel 309 includes, for example, an LCD (Liquid Crystal Display), a PDP (Plasma Display Panel), and the like.

  The audio signal processing circuit 310 performs necessary processing such as D / A conversion on the audio data received by the HDMI receiving unit 303 or obtained by the bit stream processing unit 306. The audio amplification circuit 311 amplifies the audio signal output from the audio signal processing circuit 310 and supplies the amplified audio signal to the speaker 312.

  The CPU 321 controls the operation of each unit of the television receiver 300. The flash ROM 322 stores control software and data. The DRAM 323 constitutes a work area for the CPU 321. The CPU 321 develops software and data read from the flash ROM 322 on the DRAM 323 to activate the software, and controls each unit of the television receiver 300. The remote control receiving unit 325 receives the remote control signal (remote control code) transmitted from the remote control transmitter 326 and supplies it to the CPU 321. The CPU 321 controls each part of the television receiver 300 based on the remote control code. The CPU 321, flash ROM 322, and DRAM 323 are connected to the internal bus 324.

  The operation of the television receiver 300 shown in FIG. 32 will be briefly described. The HDMI receiving unit 303 receives stereoscopic image data and audio data transmitted from the set top box 200 connected to the HDMI terminal 302 via the HDMI cable 400. The stereoscopic image data received by the HDMI receiving unit 303 is supplied to the 3D signal processing unit 301. The audio data received by the HDMI receiving unit 303 is supplied to the audio signal processing circuit 310.

  A television broadcast signal input to the antenna terminal 304 is supplied to the digital tuner 305. The digital tuner 305 processes the television broadcast signal and outputs predetermined bit stream data (transport stream) BSD corresponding to the user's selected channel.

  The bit stream data BSD output from the digital tuner 305 is supplied to the bit stream processing unit 306. The bit stream processing unit 306 extracts stereoscopic image data, audio data, stereoscopic image subtitle data (including display control information), and the like from the bit stream data BSD. Further, the bit stream processing unit 306 combines the display data of the left eye subtitle and the right eye subtitle with the stereoscopic image data, and outputs the stereoscopic image data (full resolution left eye image data and Right eye image data) is generated. The output stereoscopic image data is supplied to the video / graphic processing circuit 307 through the 3D signal processing unit 301.

  In the 3D signal processing unit 301, the stereoscopic image data received by the HDMI receiving unit 303 is decoded, and full-resolution left eye image data and right eye image data are generated. The left eye image data and right eye image data are supplied to the video / graphic processing circuit 307. In the video / graphic processing circuit 307, image data for displaying a stereoscopic image is generated based on the left eye image data and the right eye image data, and image quality adjustment processing, OSD (on-screen display) is performed as necessary. The superimposing information data is synthesized.

  Image data obtained by the video / graphic processing circuit 307 is supplied to the panel drive circuit 308. Therefore, a stereoscopic image is displayed on the display panel 309. For example, the left eye image based on the left eye image data and the right eye image based on the right eye image data are alternately displayed on the display panel 309 in a time division manner. For example, the viewer can see only the left-eye image with the left eye and the right eye with the shutter glasses by alternately opening the left-eye shutter and the right-eye shutter in synchronization with the display on the display panel 309. Only the right eye image can be seen, and a stereoscopic image can be perceived.

  Also, the audio data obtained by the bit stream processing unit 306 is supplied to the audio signal processing circuit 310. In the audio signal processing circuit 310, necessary processing such as D / A conversion is performed on the audio data received by the HDMI receiving unit 303 or obtained by the bit stream processing unit 306. The audio data is amplified by the audio amplification circuit 311 and then supplied to the speaker 312. Therefore, sound corresponding to the display image on the display panel 309 is output from the speaker 312.

[Another example of configuration of transmission data generation unit and bit stream processing unit (1)]
"Configuration example of transmission data generator"
FIG. 33 illustrates a configuration example of the transmission data generation unit 110A in the broadcast station 100 (see FIG. 1). The transmission data generation unit 110A transmits disparity information (disparity vector) with a data structure that can be easily linked to an ARIB (Association of Radio Industries and Businesses) system, which is one of the existing broadcasting standards. The transmission data generation unit 110A includes a data extraction unit (archive unit) 121, a video encoder 122, an audio encoder 123, a caption generation unit 124, a disparity information creation unit 125, a caption encoder 126, and a multiplexer 127. is doing.

  A data recording medium 121a is detachably attached to the data extraction unit 121, for example. The data recording medium 121a includes audio data, stereo data including left-eye image data and right-eye image data, as in the data recording medium 111a in the data extraction unit 111 of the transmission data generation unit 110 shown in FIG. Parallax information is recorded in association with each other. The data extracting unit 121 extracts and outputs stereoscopic image data, audio data, parallax information, and the like from the data recording medium 121a. The data recording medium 121a is a disk-shaped recording medium, a semiconductor memory, or the like.

  Returning to FIG. 33, the caption generation unit 124 generates caption data (ARIB-style caption text data). The caption encoder 126 generates a caption data stream (caption elementary stream) including the caption data generated by the caption generation unit 124. FIG. 34A shows a configuration example of a caption data stream. In this example, as shown in FIG. 34B, three caption units (captions) of “1st Caption Unit”, “2nd Caption Unit”, and “3rd Caption Unit” are displayed on the same screen. Is shown.

  The caption data of each caption unit is inserted into the caption data stream as the caption text data (caption code) of the caption text data group. Although not shown, setting data such as the display area of each caption unit is inserted into the caption data stream as data of a caption management data group. The display areas of the caption units of “1st Caption Unit”, “2nd Caption Unit”, and “3rd Caption Unit” are indicated by (x1, y1), (x2, y2), and (x3, y3), respectively. .

  The parallax information creation unit 125 has a viewer function. The disparity information creating unit 125 performs a downsizing process on the disparity information output from the data extracting unit 121, that is, the disparity vector for each pixel (pixel), and generates disparity vectors belonging to a predetermined region. Although the detailed description is omitted, the disparity information creating unit 125 performs the same downsizing process as the disparity information creating unit 115 of the transmission data generating unit 110 illustrated in FIG. 2 described above.

  The disparity information creating unit 125 creates disparity vectors corresponding to a predetermined number of caption units (captions) displayed on the same screen by the above-described downsizing process. In this case, the disparity information creating unit 125 creates a disparity vector (individual disparity vector) for each caption unit, or creates a disparity vector common to each caption unit (common disparity vector). This selection depends on, for example, user settings.

  When creating an individual disparity vector, the disparity information creating unit 125 obtains a disparity vector belonging to the display area by the above-described downsizing process based on the display area of each caption unit. Further, when creating the common disparity vector, the disparity information creating unit 125 obtains the disparity vector of the entire picture (entire image) by the above-described downsizing process (see FIG. 9D). Note that the disparity information creating unit 125 may obtain a disparity vector belonging to the display area of each caption unit and select the disparity vector having the largest value when creating a common disparity vector.

  The caption encoder 126 includes the disparity vector (disparity information) created by the disparity information creating unit 125 as described above in the caption data stream. In this case, the caption data of each caption unit displayed on the same screen is inserted as the caption text data (caption code) in the PES stream of the caption text data group. Also, disparity vectors (disparity information) are inserted into the caption data stream as caption display control information in the caption management data PES stream or in the caption text data group PES stream.

  Here, a case where an individual disparity vector is created by the disparity information creating unit 125 and a disparity vector (disparity information) is inserted into the PES stream of caption management data will be described. Here, an example is shown in which three caption units (captions) of “1st Caption Unit”, “2nd Caption Unit”, and “3rd Caption Unit” are displayed on the same screen.

  The disparity information creating unit 125 creates an individual disparity vector corresponding to each caption unit, as shown in FIG. “Disparity 1” is an individual disparity vector corresponding to “1st Caption Unit”. “Disparity 2” is a disparity vector corresponding to “2nd Caption Unit”. “Disparity 3” is an individual disparity vector corresponding to “3rd Caption Unit”.

  FIG. 35A shows a configuration example of a caption data stream (PES stream) generated by the caption encoder 126. The caption text information of each caption unit and the extended display control information (data unit ID) associated with each caption text information are inserted into the PES stream of the caption text data group. Further, extended display control information (disparity information) corresponding to the caption text information of each caption unit is inserted into the PES stream of the caption management data group.

  The extended display control information (data unit ID) of the caption text data group is required to associate each extended display control information (disparity information) of the caption management data group with each caption text information of the caption text data group. The In this case, the disparity information as each extended display control information of the caption management data group is an individual disparity vector of the corresponding caption unit. Although not shown, setting data such as the display area of each caption unit is inserted as caption management data (control code) in the PES stream of the caption management data group. The display areas of the caption units of “1st Caption Unit”, “2nd Caption Unit”, and “3rd Caption Unit” are indicated by (x1, y1), (x2, y2), and (x3, y3), respectively. .

  FIG. 35 (c) shows a first view (1st View) in which each caption unit (caption) is superimposed, for example, a right eye image. FIG. 35D shows a second view (1st View) in which each caption unit is superimposed, for example, a left eye image. As shown in the figure, the individual disparity vector corresponding to each caption unit is used to give disparity between each caption unit superimposed on the right eye image and each caption unit superimposed on the left eye image, for example. Used for.

  Next, a case where a common disparity vector is created by the disparity information creating unit 125 and a disparity vector (disparity information) is inserted into the PES stream of caption management data will be described. Here, an example is shown in which three caption units (captions) of “1st Caption Unit”, “2nd Caption Unit”, and “3rd Caption Unit” are displayed on the same screen. The disparity information creating unit 125 creates a common disparity vector “Disparity” common to the caption units, as shown in FIG.

  FIG. 36A shows a configuration example of a caption data stream (PES stream) generated by the caption encoder 126. The caption sentence information of each caption unit is inserted into the PES stream of the caption sentence data group. Further, extended display control information (disparity information) corresponding to the caption text information of each caption unit is inserted into the PES stream of the caption management data group. In this case, the disparity information as the extended display control information of the caption management data group is a common disparity vector of each caption unit.

  Although not shown, setting data such as the display area of each caption unit is inserted as caption management data (control code) in the PES stream of the caption management data group. The display areas of the caption units of “1st Caption Unit”, “2nd Caption Unit”, and “3rd Caption Unit” are indicated by (x1, y1), (x2, y2), and (x3, y3), respectively. .

  FIG. 36C shows a first view (1st View) in which each caption unit (caption) is superimposed, for example, a right eye image. FIG. 36D shows a second view (1st View) in which the caption units are superimposed, for example, a left eye image. As shown in the figure, the common disparity vector common to each caption unit is used to give disparity between each caption unit superimposed on the right eye image and each caption unit superimposed on the left eye image, for example. Used for.

  Next, a case where an individual disparity vector is created by the disparity information creating unit 125 and a case where a disparity vector (disparity information) is inserted into a PES stream of a caption text data group will be described. Here, an example is shown in which three caption units (captions) of “1st Caption Unit”, “2nd Caption Unit”, and “3rd Caption Unit” are displayed on the same screen.

  The disparity information creating unit 125 creates an individual disparity vector corresponding to each caption unit, as shown in FIG. “Disparity 1” is an individual disparity vector corresponding to “1st Caption Unit”. “Disparity 2” is a disparity vector corresponding to “2nd Caption Unit”. “Disparity 3” is an individual disparity vector corresponding to “3rd Caption Unit”.

  FIG. 37A shows a configuration example of the PES stream of the caption text data group in the caption data stream (PES stream) generated by the caption encoder 126. The caption sentence information (caption sentence data) of each caption unit is inserted into the PES stream of this caption sentence data group. Also, display control information (disparity information) corresponding to the caption text information of each caption unit is inserted into the PES stream of this caption text data group. In this case, the disparity information as each display control information is the individual disparity vector created by the disparity information creating unit 125 as described above.

  Although not shown, setting data such as the display area of each caption unit is inserted as caption management data (control code) in the PES stream of the caption management data group. In addition, the display areas of the caption units of “1st Caption Unit”, “2nd Caption Unit”, and “3rd Caption Unit” are indicated by (x1, y1), (x2, y2), and (x3, y3), respectively. ing.

  FIG. 37 (c) shows a first view (1st View) in which each caption unit (caption) is superimposed, for example, a right eye image. FIG. 37 (d) shows a second view (1st View) in which the caption units are superimposed, for example, a left eye image. As shown in the figure, the individual disparity vector corresponding to each caption unit is used to give disparity between each caption unit superimposed on the right eye image and each caption unit superimposed on the left eye image, for example. Used for.

  Next, a case where a common disparity vector is created by the disparity information creating unit 125 and a disparity vector (disparity information) is inserted into the PES stream of the caption text data group will be described. Here, an example is shown in which three caption units (captions) of “1st Caption Unit”, “2nd Caption Unit”, and “3rd Caption Unit” are displayed on the same screen. As shown in FIG. 38B, the disparity information creating unit 125 creates a common disparity vector “Disparity” common to the caption units.

  FIG. 38A shows a configuration example of the PES stream of the caption text data group in the caption data stream (PES stream) generated by the caption encoder 126. The caption sentence information (caption sentence data) of each caption unit is inserted into the PES stream of this caption sentence data group. Also, display control information (disparity information) corresponding to the caption text information of each caption unit is inserted into the PES stream of this caption text data group. In this case, the disparity information as the display control information is the common disparity vector created by the disparity information creating unit 125 as described above.

  Although not shown, setting data such as the display area of each caption unit is inserted into the PES stream of the caption management data group as caption management information (control code). In addition, the display areas of the caption units of “1st Caption Unit”, “2nd Caption Unit”, and “3rd Caption Unit” are indicated by (x1, y1), (x2, y2), and (x3, y3), respectively. ing.

  FIG. 38 (c) shows a first view (1st View) in which each caption unit (caption) is superimposed, for example, a right eye image. FIG. 38D shows a second view (1st View) in which each caption unit is superimposed, for example, a left eye image. As shown in the figure, the common disparity vector common to each caption unit is used to give disparity between each caption unit superimposed on the right eye image and each caption unit superimposed on the left eye image, for example. Used for.

  Note that the examples of FIGS. 35 (c), (d), 36 (c), (d), 37 (c), (d), 38 (c), (d) are shown in the second view ( For example, only the position of each caption unit superimposed on the left eye image) is shifted. However, it is conceivable to shift only the position of each caption unit superimposed on the first view (for example, the right eye image), or to shift the position of each caption unit superimposed on both views.

  FIGS. 39A and 39B show a case where the positions of both the caption unit superimposed on the first view and the second view are shifted. In this case, the shift value (offset value) D [i] of each caption unit in the first view and the second view from the value “disparity [i]” of the disparity vector “Disparity” corresponding to each caption unit. However, it is required as follows.

  That is, when disparity [i] is an even number, “D [i] = − disparity [i] / 2” is obtained in the first view, and “D [i] = disparity” is obtained in the second view. [i] / 2 ”. Thereby, the position of each caption unit to be superimposed on the first view (for example, the right eye image) is shifted to the left by “disparity [i] / 2”. Further, the position of each caption unit to be superimposed on the second view (for example, the left eye image) is shifted to the right by (disparity [i] / 2).

  When disparity (i) is an odd number, “D [i] = − (disparity [i] +1) / 2” is obtained in the first view, and “D [i] is obtained in the second view. i] = (disparity [i] -1) / 2 ". Thereby, the position of each caption unit to be superimposed on the first view (for example, the right eye image) is shifted to the left by “(disparity [i] +1) / 2”. Further, the position of each caption unit to be superimposed on the second view (for example, the left eye image) is shifted to the right by “(disparity [i] −1) / 2”.

  Here, the packet structure of the caption code and the control code will be briefly described. First, a basic packet structure of a caption code included in a PES stream of a caption text data group will be described. FIG. 40 shows a packet structure of the caption code. “Data_group_id” indicates data group identification, and here indicates a caption sentence data group. Note that “Data_group_id” indicating a caption text data group further specifies a language. For example, “Data_group_id == 0x21” is set, indicating that the subtitle text data group is a subtitle text (first language).

  “Data_group_size” indicates the number of bytes of subsequent data group data. In the case of a caption text data group, this data group data is caption text data (caption_data). One or more data units are arranged in the caption text data. Each data unit is separated by a data unit separation code (unit_parameter). A caption code is arranged as data unit data (data_unit_data) in each data unit.

  Next, the packet structure of the control code will be described. FIG. 41 illustrates a packet structure of control codes included in the PES stream of the caption management data group. “Data_group_id” indicates data group identification. Here, it indicates a caption management data group, and “Data_group_id == 0x20” is set. “Data_group_size” indicates the number of bytes of subsequent data group data. In the case of a caption management data group, this data group data is caption management data (caption_management_data).

  One or more data units are arranged in the caption management data. Each data unit is separated by a data unit separation code (unit_parameter). A control code is arranged as data unit data (data_unit_data) in each data unit. In this embodiment, the value of the disparity vector is given as an 8-unit code. “TCS” is 2-bit data and indicates a character encoding method. Here, “TCS == 00” is set, which indicates an 8-unit code.

  FIG. 42 shows the structure of a data group in a caption data stream (PES stream). A 6-bit field of “data_group_id” indicates data group identification, and identifies the types of caption management data and caption text data. A 16-bit field of “data_group_size” indicates the number of bytes of subsequent data group data in this data group field. Data group data is stored in “data_group_data_byte”. “CRC — 16” is a 16-bit cyclic redundancy check code. The encoding section of the CRC code is from the beginning of “data_group_id” to the end of “data_group_data_byte”.

  In the case of a caption management data group, “data_group_data_byte” in the data group structure of FIG. 42 is caption management data (caption_management_data). In the case of a caption text data group, “data_group_data_byte” in the data group structure of FIG. 42 is caption data (caption_data).

  FIG. 43 schematically shows the structure of caption management data when disparity vectors (disparity information) are inserted into the PES stream of caption management data. “Advanced_rendering_version” is 1-bit flag information newly defined in this embodiment and indicating whether or not the extended display of subtitles is supported. On the receiving side, based on the flag information arranged in the management information layer as described above, it is possible to easily grasp whether or not the extended display of subtitles is supported. A 24-bit field of “data_unit_loop_length” indicates the number of bytes of the subsequent data unit in the caption management data field. In “data_unit”, a data unit to be transmitted in this caption management data field is stored.

  FIG. 44 schematically shows the structure of caption data when disparity vectors (disparity information) are inserted into the PES stream of caption management data. A 24-bit field of “data_unit_loop_length” indicates the number of bytes of the subsequent data unit in the caption data field. In “data_unit”, a data unit to be transmitted in the caption data field is stored. Note that there is no flag information of “advanced_rendering_version” in the structure of the caption data.

  FIG. 45 schematically illustrates the structure of caption data when disparity vectors (disparity information) are inserted into the PES stream of the caption text data group. “Advanced_rendering_version” is 1-bit flag information newly defined in this embodiment and indicating whether it corresponds to extended display of subtitles. On the receiving side, based on the flag information arranged in the upper layer of the data unit as described above, it is possible to easily grasp whether or not the extended display of subtitles is supported. The 24-bit field of “data_unit_loop_length” indicates the number of bytes of the subsequent data unit in the caption text data field. In “data_unit”, a data unit to be transmitted in the caption text data field is stored.

  FIG. 46 schematically shows the structure of caption management data when disparity vectors (disparity information) are inserted into the PES stream of the caption text data group. A 24-bit field of “data_unit_loop_length” indicates the number of bytes of the subsequent data unit in the caption management data field. In “data_unit”, a data unit to be transmitted in this caption management data field is stored. Note that there is no flag information of “advanced_rendering_version” in the structure of the caption management data.

  FIG. 47 illustrates the structure (Syntax) of the data unit (data_unit) included in the caption data stream. An 8-bit field of “unit_separator” indicates a data unit separation code and is “0x1F”. The 8-bit field of “data_unit_parameter” is a data unit parameter that identifies the type of data unit.

  FIG. 48 shows the types of data units, data unit parameters, and functions. For example, the data unit parameter indicating the data unit of the main text is “0x20”. For example, the data unit parameter indicating the geometric data unit is “0x28”. For example, a data unit parameter indicating a bitmap data unit is set to “0x35”. In this embodiment, a data unit for extended display control for storing display control information (extended display control information) is newly defined, and a data unit parameter indicating this data unit is set to, for example, “0x4F”.

  The 24-bit field of “data_unit_size” indicates the number of bytes of subsequent data unit data in this data unit field. Data unit data is stored in “data_unit_data_byte”. FIG. 49 shows the structure (Syntax) of the data unit (data_unit) for extended display control. In this case, the data unit parameter is “0x4F”, and display control information is stored in “Advanced_Rendering_Control” as “data_unit_data_byte”.

  FIG. 50 illustrates the structure (Syntax) of “Advanced_Rendering_Control” in the data unit for extended display control included in the PES stream of the caption management data group in the examples of FIGS. 35 and 36 described above. Further, FIG. 50 illustrates the structure (Syntax) of “Advanced_Rendering_Control” in the data unit of the extended display control included in the PES stream of the caption data group in the example of FIGS. 37 and 38 described above. That is, FIG. 50 shows a structure in the case of inserting stereo video parallax information as display control information.

  The 8-bit field of “start_code” indicates the start of “Advanced_Rendering_Control”. A 16-bit field of “data_unit_id” indicates a data unit ID. The 16-bit field of “data_length” indicates the number of subsequent data bytes in this advanced rendering control field. The 8-bit field of “Advanced_rendering_type” is an advanced rendering type that specifies the type of display control information. Here, the data unit parameter is, for example, “0x01”, indicating that the display control information is “stereo video parallax information”. Disparity information is stored in “disparity_information”.

  FIG. 51 illustrates the structure (Syntax) of “Advanced_Rendering_Control” in the extended display control data unit included in the PES stream of the caption data group in the example of FIG. 35 described above. That is, FIG. 51 shows a structure when a data unit ID is inserted as display control information.

  The 8-bit field of “start_code” indicates the start of “Advanced_Rendering_Control”. A 16-bit field of “data_unit_id” indicates a data unit ID. The 16-bit field of “data_length” indicates the number of subsequent data bytes in this advanced rendering control field. The 8-bit field of “Advanced_rendering_type” is an advanced rendering type that specifies the type of display control information. Here, the data unit parameter is “0x00”, for example, and the display control information is “data unit ID”.

  FIG. 53 shows the main data definition contents in the structure of “Advanced_Rendering_Control” described above and further in the structure of “disparity_information” shown in FIG. 52 described later.

  FIG. 52 shows the structure (Syntax) of “disparity_information” in “Advanced_Rendering_Control” in the data unit (data_unit) of extended display control included in the caption text data group. The 8-bit field of “sync_byte” is identification information of “disparity_information”, and indicates the beginning of this “disparity_information”. “Interval_PTS [32..0]” specifies the frame period (interval of one frame) in the update frame interval of disparity information (disparity) in units of 90 KHz. That is, “interval_PTS [32..0]” represents a value obtained by measuring the frame period with a clock of 90 KHz in 33-bit length.

  In disparity information, the frame period is designated by “interval_PTS [32..0]”, so that the update frame interval of disparity information intended on the transmission side can be correctly transmitted to the reception side. When this information is not added, for example, the video frame period is referred to on the receiving side.

  “Rendering_level” indicates a level corresponding to disparity information (disparity) that is essential on the reception side (decoder side) when displaying captions. “00” indicates that subtitle three-dimensional display using parallax information is optional. “01” indicates that subtitle three-dimensional display using disparity information (default_disparity) commonly used within the subtitle display period is essential. “10” indicates that subtitle three-dimensional display using disparity information (disparity_update) sequentially updated within the subtitle display period is essential.

  “Temporal_extension_flag” is 1-bit flag information indicating the presence / absence of disparity information (disparity_update) that is sequentially updated within the caption display period. In this case, “1” indicates that it exists, and “0” indicates that it does not exist. The 8-bit field “default_disparity” indicates default disparity information. This disparity information is disparity information when updating is not performed, that is, disparity information that is commonly used within a caption display period.

  “Shared_disparity” indicates whether to perform common disparity information (disparity) control across data units (Data_unit). “1” indicates that one common disparity information (disparity) is applied to a plurality of subsequent data units (Data_unit). “0” indicates that the disparity information (Disparity) is applied to only one data unit (data_unit).

  When “temporal_extension_flag” is “1”, the disparity information includes “disparity_temporal_extension ()”. Since the structural example (Syntax) of this “disparity_temporal_extension ()” is the same as described above, the description thereof is omitted here (see FIGS. 21 and 22).

  Note that “interval_PTS [32..0]” is added to the structure (Syntax) of “disparity_information” shown in FIG. However, a structure (Syntax) of “disparity_information” without “interval_PTS [32..0]” may be considered. In this case, the structure of “disparity_information” is as shown in FIG.

  Returning to FIG. 33, the video encoder 122 performs encoding such as MPEG4-AVC, MPEG2, VC-1, etc. on the stereoscopic image data supplied from the data extraction unit 121 to generate a video elementary stream. The audio encoder 123 performs encoding such as MPEG-2 Audio AAC on the audio data supplied from the data extraction unit 121 to generate an audio elementary stream.

  The multiplexer 127 multiplexes the elementary streams output from the video encoder 122, the audio encoder 123, and the caption encoder 126. The multiplexer 127 outputs bit stream data (transport stream) BSD as transmission data (multiplexed data stream).

  The operation of the transmission data generation unit 110A shown in FIG. 33 will be briefly described. The stereoscopic image data output from the data extraction unit 121 is supplied to the video encoder 122. In the video encoder 122, the stereoscopic image data is encoded by MPEG4-AVC, MPEG2, VC-1, or the like, and a video elementary stream including the encoded video data is generated. This video elementary stream is supplied to the multiplexer 127.

  The caption generation unit 124 generates ARIB format caption data. This caption data is supplied to the caption encoder 126. The caption encoder 126 generates a caption elementary stream (caption data stream) including the caption data generated by the caption generator 124. This subtitle elementary stream is supplied to the multiplexer 127.

  Also, the disparity vector for each pixel (pixel) output from the data extraction unit 121 is supplied to the disparity information creation unit 125. The disparity information creating unit 125 creates disparity vectors (horizontal disparity vectors) corresponding to a predetermined number of caption units (captions) displayed on the same screen by downsizing processing. In this case, the disparity information creating unit 125 creates a disparity vector (individual disparity vector) for each caption unit or a disparity vector (common disparity vector) common to all caption units.

  The disparity vector created by the disparity information creating unit 125 is supplied to the caption encoder 126. In the caption encoder 126, the disparity vector is included in the caption data stream (see FIGS. 35 to 38). In the caption data stream, caption data of each caption unit displayed on the same screen is inserted as caption text data (caption code) in the PES stream of the caption text data group. In addition, a disparity vector (disparity information) is inserted into the caption data stream as the caption display control information in the PES stream of the caption management data group or in the PES stream of the caption data group. In this case, the disparity vector is inserted into a data unit for extended display control that transmits newly defined display control information (see FIG. 49).

  The audio data output from the data extraction unit 121 is supplied to the audio encoder 123. In the audio encoder 123, encoding such as MPEG-2 Audio AAC is performed on the audio data, and an audio elementary stream including the encoded audio data is generated. This audio elementary stream is supplied to the multiplexer 127.

  As described above, the elementary stream from the video encoder 122, the audio encoder 123, and the caption encoder 126 is supplied to the multiplexer 127. In the multiplexer 127, the elementary streams supplied from the encoders are packetized and multiplexed to obtain bit stream data (transport stream) BSD as transmission data.

  FIG. 55 illustrates a configuration example of a general transport stream (multiplexed data stream) including a video elementary stream, an audio elementary stream, and a caption elementary stream. This transport stream includes PES packets obtained by packetizing each elementary stream. In this configuration example, a PES packet “Video PES” of a video elementary stream is included. In addition, in this configuration example, the PES packet “Audio PES” of the audio elementary stream and the PES packet “SubtitlePES” of the caption elementary stream are included.

  In addition, the transport stream includes a PMT (Program Map Table) as PSI (Program Specific Information). This PSI is information describing to which program each elementary stream included in the transport stream belongs. Further, the transport stream includes an EIT (Event Information Table) as SI (Serviced Information) for managing each event.

  The PMT includes a program descriptor (Program Descriptor) that describes information related to the entire program. The PMT includes an elementary loop having information related to each elementary stream. In this configuration example, there are a video elementary loop, an audio elementary loop, and a subtitle elementary loop. In each elementary loop, information such as a packet identifier (PID) and a stream type (Stream_Type) is arranged for each stream, and although not shown, there is also a descriptor that describes information related to the elementary stream. Be placed.

  In this embodiment, in the transport stream (multiplexed data stream) output from the multiplexer 127 (see FIG. 33), flag information indicating whether or not the caption data stream corresponds to extended display control of captions. Has been inserted. Here, the extended display control of caption is, for example, three-dimensional caption display using parallax information. In this case, on the receiving side (set top box 200), it is possible to grasp whether or not this subtitle data stream supports extended display control of subtitles without opening the data in the subtitle data stream.

  The multiplexer 127 inserts this flag information, for example, under the above-mentioned EIT. In the configuration example of FIG. 55, a data content descriptor is inserted under the EIT. This data content descriptor includes flag information “Advanced_Rendering_support”. FIG. 56 shows a structure example (Syntax) of the data content descriptor. “Descriptor_tag” is 8-bit data indicating the type of descriptor (descriptor), and here indicates a data content descriptor. “Descriptor_length” is 8-bit data indicating the length (size) of the descriptor. This data indicates the number of bytes after “descriptor_length” as the length of the descriptor.

  “Component_tag” is 8-bit data for associating with a subtitle elementary stream. After this “component_tag”, “arib_caption_info” is defined. FIG. 57A shows a structure example (Syntax) of this “arib_caption_info”. As shown in FIG. 57B, “Advanced_Rendering_support” is 1-bit flag information indicating whether or not the caption data stream is compatible with caption extended display control. “1” indicates that subtitle expansion display is supported. “0” indicates that subtitle extended display control is not supported.

  The multiplexer 127 can also insert the flag information described above under the PMT. FIG. 58 shows a configuration example of a transport stream (multiplexed data stream) in that case. In this configuration example, a data encoding scheme descriptor is inserted under the subtitle ES loop of the PMT. The data encoding scheme descriptor includes flag information “Advanced_Rendering_support”.

  FIG. 59 shows a structure example (Syntax) of the data encoding scheme descriptor. “Descriptor_tag” is 8-bit data indicating the type of descriptor (descriptor), and here indicates a data content descriptor. “Descriptor_length” is 8-bit data indicating the length (size) of the descriptor. This data indicates the number of bytes after “descriptor_length” as the length of the descriptor.

  “Component_tag” is 8-bit data for associating with a subtitle elementary stream. Here, “data_component_id” is “0x0008” indicating caption data. After “data_component_id”, “additional_arib_caption_info” is defined. FIG. 60 shows a structural example (Syntax) of this “additional_arib_caption_info”. “Advanced_Rendering_support” is 1-bit flag information indicating whether or not the caption data stream supports extended display control of captions, as shown in FIG. 57 (b) described above. “1” indicates that subtitle expansion display is supported. “0” indicates that subtitle extended display control is not supported.

  As described above, in the transmission data generation unit 110A illustrated in FIG. 33, the bit stream data BSD output from the multiplexer 127 is a multiplexed data stream including a video data stream and a caption data stream. The video data stream includes stereoscopic image data. Also, the caption data stream includes ARIB caption data (caption unit) and disparity vectors (disparity information).

  Also, disparity information is inserted into a data unit for transmitting caption display control information in the PES stream of the caption management data group or in the PES stream of the caption sentence data group, and the caption sentence data (subtitle sentence information) and the disparity information are Correspondence has been made. Therefore, on the reception side (set top box 200), appropriate parallax can be given to caption units (captions) superimposed on the left-eye image and right-eye image using the corresponding disparity vector (disparity information). Therefore, in the display of caption units (captions), perspective consistency with each object in the image can be maintained in an optimum state.

  33, disparity information (see “default_disparity” in FIG. 52) commonly used in the caption display period is inserted into the newly defined extended display control data unit. Is done. Also, disparity information (see “disparity_update” in FIG. 21) that is sequentially updated within the caption display period can be inserted into this data unit. Then, flag information indicating the presence of disparity information that is sequentially updated within the caption display period is inserted into the data unit for extended display control (see “temporal_extension_flag” in FIG. 52).

  For this reason, it is possible to select whether to transmit only disparity information that is commonly used within the caption display period, or to transmit disparity information that is sequentially updated within the caption display period. By transmitting disparity information that is sequentially updated within the caption display period, it is possible to dynamically change the disparity to be added to the superimposition information on the reception side (set top box 200) in conjunction with the change in the image content. It becomes.

  In addition, in the transmission data generation unit 110A illustrated in FIG. 33, the disparity information included in the extended display control data unit includes disparity information of the first frame in the subtitle display period, and disparity information of frames for each subsequent update frame interval. It is supposed to consist of Therefore, the amount of transmission data can be reduced, and the memory capacity for holding the parallax information can be greatly saved on the receiving side.

  In addition, in the transmission data generation unit 110A illustrated in FIG. 33, “disparity_temporal_extension ()” inserted into the extended display control data unit has the same structure as “disparity_temporal_extension ()” included in the SCS segment described above. (See FIG. 21). Therefore, although detailed description is omitted, the transmission data generation unit 110A illustrated in FIG. 33 can obtain the same effect as the transmission data generation unit 110 illustrated in FIG. 2 by the structure of “disparity_temporal_extension ()”.

"Configuration example of the bitstream processing unit"
FIG. 61 illustrates a configuration example of the bit stream processing unit 201A of the set top box 200 corresponding to the transmission data generation unit 110A illustrated in FIG. This bit stream processing unit 201A has a configuration corresponding to the transmission data generation unit 110A shown in FIG. The bit stream processing unit 201A includes a demultiplexer 231, a video decoder 232, and a caption decoder 233. Further, the bit stream processing unit 201A includes a stereoscopic image subtitle generating unit 234, a parallax information extracting unit 235, a parallax information processing unit 236, a video superimposing unit 237, and an audio decoder 238.

  The demultiplexer 231 extracts video, audio, and subtitle packets from the bit stream data BSD, and sends them to each decoder. The video decoder 232 performs processing reverse to that of the video encoder 122 of the transmission data generation unit 110A described above. That is, a video elementary stream is reconstructed from the video packet extracted by the demultiplexer 231 and decoding processing is performed to obtain stereoscopic image data including left-eye image data and right-eye image data. The transmission method of the stereoscopic image data is, for example, the first transmission method (“Top & Bottom” method) described above, the second transmission method (“Side By Side” method), and the third transmission method (“Frame”). Sequential ”method) (see FIG. 4).

  The caption decoder 223 performs the reverse process of the caption encoder 133 of the transmission data generation unit 110 described above. That is, the caption decoder 233 reconstructs a caption elementary stream (caption data stream) from the caption packet extracted by the demultiplexer 231, performs decoding processing, and caption data (ARIB format data) of each caption unit. Subtitle data).

  The disparity information extracting unit 235 extracts disparity vectors (disparity information) corresponding to each caption unit from the caption stream obtained through the caption decoder 233. In this case, a disparity vector (individual disparity vector) for each caption unit or a disparity vector common to each caption unit (common disparity vector) is obtained (see FIGS. 35 to 38).

  As described above, the caption data stream includes ARIB-style caption (caption unit) data and disparity information (disparity vector). The disparity information is inserted into a data unit that transmits subtitle display control information. Therefore, the disparity information extracting unit 235 can extract disparity information (disparity vector) in association with the caption data of each caption unit.

  The disparity information extracting unit 235 acquires disparity information (see “default_disparity” in FIG. 52) that is commonly used in the caption display period. Further, the disparity information extracting unit 235 may acquire disparity information (see “disparity_update” in FIG. 21) that is sequentially updated within the caption display period. The disparity information extracting unit 235 sends the disparity information (disparity vector) to the stereoscopic image caption generating unit 234 through the disparity information processing unit 236. As described above, the disparity information sequentially updated within the caption display period is composed of the disparity information of the first frame in the caption display period and the disparity information of the frame for each subsequent base segment period (update frame interval). ing.

  The parallax information processing unit 236 sends the parallax information used in common during the caption display period to the stereoscopic image caption generation unit 234 as it is. On the other hand, the disparity information processing unit 236 performs interpolation processing on the disparity information sequentially updated within the caption display period, and generates disparity information at an arbitrary frame interval, for example, one frame interval within the caption display period. To the stereoscopic image caption generation unit 234. The disparity information processing unit 236 performs not the linear interpolation process but the interpolation process with the low-pass filter (LPF) process in the time direction (frame direction) as the interpolation process, and the disparity information at a predetermined frame interval after the interpolation process. In the time direction (frame direction) is smoothed (see FIG. 31).

  The stereoscopic image caption generation unit 234 generates left-eye caption data and right-eye caption data to be superimposed on the left-eye image and the right-eye image, respectively. This generation process is performed based on the caption data of each caption unit obtained by the caption decoder 233 and the disparity information (disparity vector) supplied through the disparity information processing unit 236. The stereoscopic image caption generation unit 234 then outputs left-eye caption and left-eye caption data (bitmap data).

  In this case, the left eye caption and the left eye caption (caption unit) are the same information. However, the superimposed position in the image is shifted in the horizontal direction by, for example, the parallax vector between the left-eye caption and the right-eye caption. As a result, the same subtitle superimposed on the left eye image and the right eye image can be used with parallax adjusted according to the perspective of each object in the image. The perspective consistency between each object within is maintained.

  Here, when only the disparity information (disparity vector) that is commonly used in the caption display period is transmitted from the disparity information processing unit 236, the stereoscopic image caption generation unit 234 uses the disparity information. In addition, the stereoscopic image caption generation unit 234 uses either one of the parallax information sequentially updated within the caption display period from the parallax information processing unit 236.

  Which one is used is, for example, as described above, information indicating the disparity information (disparity) correspondence level that is included in the extended display control data unit and is essential on the reception side (decoder side) when displaying captions. (See “rendering_level” in FIG. 52). In this case, for example, when “00”, it depends on the user setting. By using the disparity information that is sequentially updated within the caption display period, it is possible to dynamically change the disparity to be given to the left eye and the right eye in conjunction with the change in the image content.

  The video superimposing unit 237 performs the left-eye and left-eye caption data generated by the stereoscopic image caption generation unit 234 with respect to the stereoscopic image data (left-eye image data and right-eye image data) obtained by the video decoder 232. (Bitmap data) is superimposed to obtain display stereoscopic image data Vout. Then, the video superimposing unit 237 outputs the display stereoscopic image data Vout to the outside of the bit stream processing unit 201A.

  Further, the audio decoder 238 performs a process reverse to that of the audio encoder 123 of the transmission data generation unit 110A described above. That is, the audio decoder 238 reconstructs an audio elementary stream from the audio packet extracted by the demultiplexer 231 and performs a decoding process to obtain audio data Aout. The audio decoder 238 then outputs the audio data Aout to the outside of the bit stream processing unit 201A.

  The operation of the bit stream processing unit 201A shown in FIG. 61 will be briefly described. The bit stream data BSD output from the digital tuner 204 (see FIG. 29) is supplied to the demultiplexer 231. The demultiplexer 231 extracts video, audio, and subtitle packets from the bit stream data BSD and supplies them to each decoder.

  In the video decoder 232, a video elementary stream is reconstructed from the video packet extracted by the demultiplexer 231, and further, decoding processing is performed so that stereoscopic image data including left eye image data and right eye image data is converted. can get. The stereoscopic image data is supplied to the video superimposing unit 237.

  Also, the subtitle decoder 233 reconstructs a subtitle elementary stream from the subtitle packet extracted by the demultiplexer 231, further performs decoding processing, and subtitle data of each caption unit (ARIB method subtitle data) Is obtained. The caption data of each caption unit is supplied to the stereoscopic image caption generation unit 234.

  Also, the disparity information extracting unit 235 extracts disparity vectors (disparity information) corresponding to each caption unit from the caption stream obtained through the caption decoder 233. In this case, a disparity vector (individual disparity vector) for each caption unit, or a disparity vector common to each caption unit (common disparity vector) is obtained.

  In addition, the disparity information extracting unit 235 acquires disparity information that is commonly used within the caption display period, or disparity information that is sequentially updated within the caption display period. The disparity information (disparity vector) extracted by the disparity information extracting unit 235 is sent to the stereoscopic image caption generating unit 234 through the disparity information processing unit 236. In the disparity information processing unit 236, the following processing is performed on the disparity information that is sequentially updated within the caption display period. That is, the disparity information processing unit 236 performs interpolation processing with LPF processing in the time direction (frame direction), and generates disparity information at an arbitrary frame interval, for example, one frame interval within the caption display period. And sent to the stereoscopic image caption generation unit 234.

  In the stereoscopic image caption generation unit 234, based on the caption data of each caption unit and the disparity vector corresponding to each caption unit, the left-eye caption and the right-eye caption to be superimposed on the left-eye image and the right-eye image, respectively. Data (bitmap data) is generated. In this case, the superimposed position in the image is shifted in the horizontal direction by the amount of the parallax vector, for example, with respect to the left-eye caption. The left-eye caption and left-eye caption data are supplied to the video superimposing unit 237.

  The video superimposing unit 237 superimposes the left-eye caption data and the right-eye caption data (bitmap data) generated by the stereoscopic image caption generation unit 234 on the stereoscopic image data obtained by the video decoder 232 for display. Stereoscopic image data Vout is obtained. The display stereoscopic image data Vout is output to the outside of the bit stream processing unit 201A.

  Also, the audio decoder 238 reconstructs an audio elementary stream from the audio packet extracted by the demultiplexer 231, further performs decoding processing, and audio data Aout corresponding to the display stereoscopic image data Vout described above. Is obtained. The audio data Aout is output to the outside of the bit stream processing unit 201A.

  As described above, the caption data stream included in the bit stream data BSD supplied to the bit stream processing unit 201A illustrated in FIG. 61 includes caption (caption unit) data and disparity vectors (disparity information). . A disparity vector (disparity information) is inserted into a data unit for transmitting caption display control information in the PES stream of the caption text data group, and the caption data and the disparity vector are associated with each other.

  Therefore, the bit stream processing unit 201A can give appropriate parallax to the caption unit (caption) superimposed on the left eye image and the right eye image using the corresponding parallax vector (disparity information). Therefore, in the display of caption units (captions), perspective consistency with each object in the image can be maintained in an optimum state.

  In addition, the disparity information extracting unit 235 of the bitstream processing unit 201A illustrated in FIG. 61 acquires disparity information that is commonly used within the caption display period, or the disparity information that is sequentially updated within the caption display period. The The stereoscopic image caption generation unit 234 uses disparity information that is sequentially updated within the caption display period, so that the disparity to be given to the left-eye and right-eye captions is dynamically linked with changes in the image content. It can be changed.

  In addition, in the disparity information processing unit 236 of the bitstream processing unit 201A illustrated in FIG. 61, the disparity information sequentially updated within the caption display period is subjected to interpolation processing, and the disparity at an arbitrary frame interval within the caption display period. Information is generated. In this case, even when the disparity information is transmitted from the transmission side (broadcast station 100) every base segment period (update frame interval) such as 16 frames, the disparity given to the left-eye and right-eye captions It is possible to control at fine intervals, for example, for each frame.

  In addition, the disparity information processing unit 236 of the bit stream processing unit 201A illustrated in FIG. 61 performs an interpolation process with a low-pass filter process in the time direction (frame direction). Therefore, even when disparity information is transmitted from the transmission side (broadcast station 100) every base segment period (updated frame interval), the change in the time direction (frame direction) of the disparity information after the interpolation processing is gently performed. Yes (see FIG. 31). Therefore, it is possible to suppress a sense of incongruity due to the disparity transitions given to the left-eye and right-eye captions being discontinuous at each update frame interval.

[Another example of configuration of transmission data generation unit and bit stream processing unit (2)]
"Configuration example of transmission data generator"
FIG. 62 shows a configuration example of the transmission data generation unit 110B in the broadcast station 100 (see FIG. 1). The transmission data generation unit 110B transmits disparity information (disparity vector) with a data structure that can be easily linked to the CEA system, which is one of existing broadcast standards. The transmission data generation unit 110B includes a data extraction unit (archive unit) 131, a video encoder 132, and an audio encoder 133. In addition, the transmission data generation unit 110B includes a closed caption encoder (CC encoder) 134, a disparity information creation unit 135, and a multiplexer 136.

  A data recording medium 131a is detachably attached to the data extraction unit 131, for example. Similar to the data recording medium 111a in the data extraction unit 111 of the transmission data generation unit 110 shown in FIG. 2, the data recording medium 131a includes audio data, stereo image data including left eye image data and right eye image data, Parallax information is recorded in association with each other. The data extracting unit 131 extracts and outputs stereoscopic image data, audio data, parallax information, and the like from the data recording medium 131a. The data recording medium 131a is a disk-shaped recording medium, a semiconductor memory, or the like.

  The CC encoder 134 is an encoder compliant with CEA-708, and outputs CC data (closed caption information data) for displaying closed caption captions. In this case, the CC encoder 134 sequentially outputs CC data of each closed caption information displayed in time series.

  The disparity information creation unit 135 performs downsizing processing on the disparity vector output from the data extraction unit 131, that is, the disparity vector for each pixel (pixel), and each window included in the CC data output from the CC encoder 134 described above. Disparity information (disparity vector) associated with the ID (WindowID) is output. Although the detailed description is omitted, the disparity information creating unit 135 performs the same downsizing process as the disparity information creating unit 115 of the transmission data generating unit 110 shown in FIG. 2 described above.

  The disparity information creating unit 135 creates disparity vectors corresponding to a predetermined number of caption units (captions) displayed on the same screen by the downsizing process described above. In this case, the disparity information creating unit 135 creates a disparity vector (individual disparity vector) for each caption unit, or creates a disparity vector common to each caption unit (common disparity vector). This selection depends on, for example, user settings. The disparity information includes shift target designation information for designating closed caption information to be shifted based on the disparity information among the closed caption information superimposed on the left eye image and the closed caption information superimposed on the right eye image. It has been added.

  When creating an individual disparity vector, the disparity information creating unit 135 obtains a disparity vector belonging to the display area by the above-described downsizing process based on the display area of each caption unit. In addition, when creating the common disparity vector, the disparity information creating unit 135 obtains the disparity vector of the entire picture (entire image) by the above-described downsizing process (see FIG. 9D). Note that when creating the common parallax vector, the parallax information creating unit 135 may obtain a parallax vector belonging to the display area of each caption unit and select the parallax vector having the largest value.

  This disparity information is, for example, disparity information that is commonly used within a predetermined number of frame periods (caption display periods) in which closed caption information is displayed, or disparity information that is sequentially updated within this caption display period. . The disparity information that is sequentially updated within the caption display period is composed of disparity information of the first frame in a predetermined number of frame periods and disparity information of frames for each subsequent update frame interval.

  The video encoder 132 performs encoding such as MPEG4-AVC, MPEG2, or VC-1 on the stereoscopic image data supplied from the data extraction unit 131 to obtain encoded video data. Also, the video encoder 132 generates a video elementary stream including encoded video data in the payload portion by the stream formatter 132a provided in the subsequent stage.

  The CC data output from the CC encoder 134 and the disparity information created by the disparity information creating unit 135 are supplied to the stream formatter 132a in the video encoder 132. The stream formatter 132a embeds CC data and disparity information as user data in a video elementary stream. That is, stereoscopic image data is included in the payload portion of the video elementary stream, and CC data and disparity information are included in the user data area of the header portion.

  As shown in FIG. 63, a sequence header portion including parameters in units of sequences is arranged at the head of a video elementary stream. Subsequent to the sequence header portion, a picture header including parameters and user data in units of pictures is arranged. Following this picture header portion, a payload portion including picture data is arranged. Hereinafter, the picture header part and the payload part are repeatedly arranged. The CC data and disparity information described above are embedded in, for example, the user data area of the picture header part. Details of the method of embedding (inserting) the parallax information into the user data area will be described later.

  The audio encoder 133 performs encoding such as MPEG-2Audio AAC on the audio data extracted from the data extraction unit 131 to generate an audio elementary stream. The multiplexer 136 multiplexes the elementary streams output from the video encoder 132 and the audio encoder 133. The multiplexer 136 outputs bit stream data (transport stream) BSD as transmission data (multiplexed data stream).

  The operation of the transmission data generation unit 110B shown in FIG. 62 will be briefly described. The stereoscopic image data output from the data extraction unit 131 is supplied to the video encoder 132. In the video encoder 132, the stereoscopic image data is encoded by MPEG4-AVC, MPEG2, VC-1, or the like, and a video elementary stream including the encoded video data is generated. This video elementary stream is supplied to the multiplexer 136.

  Also, the CC encoder 134 outputs CC data (closed caption information data) for displaying closed caption captions. In this case, the CC encoder 134 sequentially outputs CC data of each closed caption information displayed in time series.

  Further, the disparity vector for each pixel (pixel) output from the data extracting unit 131 is supplied to the disparity information creating unit 135. In the disparity information creating unit 135, the disparity vector is subjected to downsizing processing or the like, and disparity information (disparity information) associated with each window ID (WindowID) included in the CC data output from the CC encoder 134 described above. Vector) is output.

  The CC data output from the CC encoder 134 and the disparity information created by the disparity information creating unit 135 are supplied to the stream formatter 132a of the video encoder 132. In this stream formatter 132a, CC data and disparity information are inserted into the user data area of the header portion of the video elementary stream. In this case, embedding or insertion of disparity information has been skipped as, for example, (A) a method of expanding within the range of an existing table (CEA table), and (B) padding bytes. This is done by, for example, a method for defining a new byte extension.

  The audio data output from the data extraction unit 131 is supplied to the audio encoder 133. The audio encoder 133 performs encoding such as MPEG-2Audio AAC on the audio data, and generates an audio elementary stream including the encoded audio data. This audio elementary stream is supplied to the multiplexer 136. In the multiplexer 136, the elementary streams supplied from the encoders are multiplexed to obtain bit stream data BSD as transmission data.

[Method of embedding (inserting) disparity information in user area]
Next, details of a method for embedding disparity information in a user data area will be described. (A) A method of performing extension within the range of an existing table (CEA table), (B) a method of newly defining a byte skipped as a padding byte, and the like can be considered. The method (A) is a method in which the extension command EXT1 and the subsequent value indicate the number of extension bytes and a parameter is subsequently inserted. Each method will be described below.

"(A) Method of extending within the range of existing table (table) (1)"
FIG. 64 schematically shows a CEA table. When performing expansion in this CEA table, after declaring the start of the extended command with the 0x10 (EXT1) command in the C0 table, the C2 table (C2 Table) and C3 table (C3 Table) are determined according to the byte length of the extended command. , G2 table (G2 Table) and G3 table (G3 Table) addresses are designated. Here, since a 3-byte command is configured, the following byte string indicating that 3 bytes follow in the C2 table is defined. It is determined by the CEA standard that the address space of 0x18 to 0x1F in the C2 table indicates that 3 bytes follow.

The total extended command in this case is as follows.
Extended command: EXT1 (0x10) + 0x18 (following 3 bytes)
+ [Byte1] + [Byte2] + [Byte3]

  FIG. 65 shows a structure example of a 3-byte field of “Byte1”, “Byte2”, and “Byte3”. In the 3-bit field from the seventh bit to the fifth bit of “Byte1”, “window_id” is arranged. With this “window_id”, association with the window to which the information of the extended command is applied is performed. In addition, “temporal_division_count” is arranged in the 5-bit field from the 4th bit to the 0th bit of “Byte1”. This “temporal_division_count” indicates the number of base segments included in the caption display period (see FIG. 22).

  “Temporal_division_size” is arranged in the 7-bit and 6-bit 2-bit fields of “Byte2”. The “temporal_division_size” indicates the number of frames included in the base segment period (update frame interval). “00” indicates 16 frames. “01” indicates 25 frames. “10” indicates 30 frames. Further, “11” indicates 32 frames (see FIG. 22).

  “Shared_disparity” is arranged in the 1-bit field of the fifth bit of “Byte2”. This “shared_disparity” indicates whether or not to perform common disparity information (disparity) control over all windows. “1” indicates that one common disparity information (disparity) is applied to all subsequent windows. “0” indicates that the disparity information (Disparity) is applied to only one window (see FIG. 19).

  “Shifting_interval_counts” is arranged in the 5-bit field from the 4th bit to the 0th bit of “Byte2”. This “shifting_interval_counts” indicates a draw factor for adjusting the base segment period (update frame interval), that is, the number of subtracted frames (see FIG. 22).

  In the example of updating the disparity information for each base segment period (BSP) in FIG. 66, the base segment period is adjusted by the draw factor with respect to the update timing of the disparity information at time points C to F. The presence of this adjustment information makes it possible to adjust the base segment period (update frame interval), and more accurately convey changes in the time direction (frame direction) of disparity information to the receiving side. .

  In addition, as the adjustment of the base segment period (update frame interval), in addition to the adjustment in the direction of shortening by the number of subtracted frames described above, the adjustment in the direction of increasing by the number of additional frames may be considered. For example, bi-directional adjustment is possible by setting a 5-bit field of “shifting_interval_counts” as a signed integer.

  In the 8-bit field from the 7th bit to the 0th bit of “Byte3”, “disparity_update” is arranged. This “disparity_update” indicates the disparity information of the corresponding base segment. Note that “disparity_update” at k = 0 is an initial value of disparity information that is sequentially updated at update frame intervals within the caption display period, that is, disparity information of the first frame in the caption display period.

  By repeatedly transmitting the 5-byte extended command described above in the user data area, transmission (transmission) of disparity information that is sequentially updated in the caption display period and adjustment information of the update frame interval added thereto is possible. Become.

“(A) Method to extend within the range of existing table (table) (2)”
FIG. 67 schematically shows the CEA table. When performing expansion in this CEA table, after declaring the start of the extended command with the 0x10 (EXT1) command in the C0 table, the C2 table (C2 Table) and C3 table (C3 Table) are determined according to the byte length of the extended command. , G2 table (G2 Table) and G3 table (G3 Table) addresses are designated. Here, since a variable length command is configured, the following byte sequence is defined in the C3 table. Note that the CEA standard determines that the address space of 0x90 to 0x9F in the C3 table indicates the subsequent 3 bytes.

The total extended command in this case is as follows.
Extended command: EXT1 (0x10) + EXTCode (0x90)
+ [Header (Byte1)] + [Byte2] + ... + [ByteN]

  FIG. 68 shows a structure example of a 4-byte field of “Header (Byte 1)”, “Byte 2”, “Byte 3”, and “Byte 4”. “Type_field” is arranged in the 7-bit and 6-bit 2-bit fields of “Header (Byte1)”. This “type_field” indicates a command type. “00” indicates the start of a command (BOC: Beginning of Comand). “01” indicates continuation of command (COC). “10” indicates the end of command (EOC: End Of Command).

  In a 5-bit field from the 4th bit to the 0th bit of “Header (Byte1)”, “Length_field” is arranged. This “Length_field” indicates the number of bytes after this extended command. A maximum of 28 bytes is determined in one service block. Within this range, disparity information (disparity) can be updated by repeating Byte2 to Byte4 in a loop. In this case, a maximum of nine sets of disparity information can be updated in one service block.

  In the 3-bit field from the seventh bit to the fifth bit of “Byte2”, “window_id” is arranged. With this “window_id”, association with the window to which the information of the extended command is applied is performed. Also, “temporal_division_count” is arranged in the 5-bit field from the 4th bit to the 0th bit of “Byte2”. This “temporal_division_count” indicates the number of base segments included in the caption display period (see FIG. 22).

  “Temporal_division_size” is arranged in the 7-bit and 6-bit 2-bit fields of “Byte3”. The “temporal_division_size” indicates the number of frames included in the base segment period (update frame interval). “00” indicates 16 frames. “01” indicates 25 frames. “10” indicates 30 frames. Further, “11” indicates 32 frames (see FIG. 22).

  In the 1-bit field of the fifth bit of “Byte3”, “shared_disparity” is arranged. This “shared_disparity” indicates whether or not to perform common disparity information (disparity) control over all windows. “1” indicates that one common disparity information (disparity) is applied to all subsequent windows. “0” indicates that the disparity information (Disparity) is applied to only one window (see FIG. 19).

  “Shifting_interval_counts” is arranged in a 5-bit field from the 4th bit to the 0th bit of “Byte3”. This “shifting_interval_counts” indicates a draw factor for adjusting the base segment period (update frame interval), that is, the number of subtracted frames (see FIG. 22).

  In the 8-bit field from the 7th bit to the 0th bit of “Byte4”, “disparity_update” is arranged. This “disparity_update” indicates the disparity information of the corresponding base segment. Note that “disparity_update” at k = 0 is an initial value of disparity information that is sequentially updated at update frame intervals within the caption display period, that is, disparity information of the first frame in the caption display period.

  By transmitting the above-described variable-length extended command in the user data area, transmission (transmission) of disparity information sequentially updated in the caption display period and update frame interval adjustment information added thereto is possible. .

“(B) How to define a new extended padding byte”
FIG. 69 shows a structural example (Syntax) of conventional closed caption data (CC data). In the case of “cc_valid = 0” and “cc_type = 00”, the receiving side (decoder) skips the fields of “cc_data_1” and “cc_data_2”. Here, this space is used to define an extension for disparity information (disparity) transmission.

  FIG. 70 illustrates a structure example (Syntax) of closed caption data (CC data) modified to support disparity information (disparity). The 2-bit field “extended_control” is information for controlling the two fields “cc_data_1” and “cc_data_2”. As shown in FIG. 71 (a), when “cc_valid = 0” and “cc_type = 00”, when the 2-bit field of “extended_control” is “01” and “10”, “cc_data_1” and “cc_data_2” These two fields are used for transmission of disparity information (disparity).

  In this case, as shown in FIG. 71 (b), when “extended_control = 01”, the field of “cc_data_1” means “Start of Extended Packet”, and the first extended packet data (1 byte) is inserted. It will be. At this time, the field of “cc_data_2” means “Extended Packet Data”, and the extended packet data (1 byte) that follows is inserted.

  As shown in FIG. 71 (b), when “extended_control = 10”, each field of “cc_data_1” and “cc_data_2” means “Extended Packet Data”, and the extended packet data (1 byte) is inserted. 71 (b), when “extended_control = 00”, the fields “cc_data_1” and “cc_data_2” mean “Padding”.

  “Extended Packet Data” is defined as a transport of “caption_disparity_data ()”. 72 and 73 show a structure example (syntax) of “caption_disparity_data ()”. FIG. 74 shows main data definition contents (semantics) in the structural example of “caption_disparity_data ()”.

  “Service_number” is 1-bit information indicating a service type. “Shared_windows” indicates whether to perform common disparity information (disparity) control over all windows. “1” indicates that one common disparity information (disparity) is applied to all subsequent windows. “0” indicates that the disparity information (Disparity) is applied to only one window.

  “Caption_window_count” is 3-bit information indicating the number of caption windows. “Caption_window_id” is 3-bit identification information for identifying a caption window. “Temporal_extension_flag” is 1-bit flag information indicating the presence / absence of disparity information (disparity_update) sequentially updated within the caption display period in the corresponding window. In this case, “1” indicates that it exists, and “0” indicates that it does not exist.

  “Rendering_level” indicates a level corresponding to disparity information (disparity) that is essential on the reception side (decoder side) when displaying captions. “00” indicates that subtitle three-dimensional display using parallax information is optional. “01” indicates that subtitle three-dimensional display using disparity information (default_disparity) commonly used within the subtitle display period is essential. “10” indicates that subtitle three-dimensional display using disparity information (disparity_update) sequentially updated within the subtitle display period is essential.

  “Select_view_shift” is 2-bit information constituting shift target designation information. This “select_view_shift” designates closed caption information to be shifted based on disparity information among the closed caption information superimposed on the left eye image and the closed caption information superimposed on the right eye image. “Select_view_shift = 00” is reserved. When “select_view_shift = 01”, this indicates that only the closed caption information to be superimposed on the right eye image is shifted in the horizontal direction by the amount of disparity information (disparity).

  Further, when “select_view_shift = 10”, it indicates that only the closed caption information to be superimposed on the left eye image is shifted in the horizontal direction by the amount of disparity information (disparity). Further, when “select_view_shift = 11”, both the closed caption information superimposed on the left eye image and the closed caption information superimposed on the right eye image are shifted in opposite directions in the horizontal direction.

  The 8-bit field “default_disparity” indicates default disparity information. This disparity information is disparity information when updating is not performed, that is, disparity information that is commonly used within a caption display period. When “temporal_extention_flag = 1” is “1”, “caption_disparity_data ()” has “disparity_temporal_extension ()”. Here, basically, disparity information to be updated for each base segment period (BSP) is stored.

  As described above, FIG. 20 illustrates an example of updating disparity information for each base segment period (BSP). The base segment period means an update frame interval. As is clear from this figure, the disparity information sequentially updated within the caption display period includes the disparity information of the first frame in the caption display period, and the disparity information of the frame for each subsequent base segment period (updated frame interval). It is made up of.

  FIG. 73 illustrates a structure example (syntax) of “disparity_temporal_extension ()”. A 2-bit field of “temporal_division_size” indicates the number of frames included in the base segment period (update frame interval). “00” indicates 16 frames. “01” indicates 25 frames. “10” indicates 30 frames. Further, “11” indicates 32 frames.

  “Temporal_division_count” indicates the number of base segments included in the caption display period. “Disparity_curve_no_update_flag” is 1-bit flag information indicating whether or not disparity information is updated. “1” indicates that the disparity information is not updated at the edge of the corresponding base segment, that is, skipping, and “0” indicates that the disparity information is updated at the edge of the corresponding base segment.

  In the example of updating disparity information for each base segment period (BSP) in FIG. 23 described above, disparity information is not updated at the edge of the base segment to which “skip” is attached. When the period in which the change in the frame direction of the disparity information is similar continues for a long time due to the presence of this flag information, disparity information is not updated, and transmission of disparity information within that period can be omitted. It becomes possible to suppress the data amount of information.

  When “disparity_curve_no_update_flag” is “0” and the disparity information is updated, the disparity information includes “shifting_interval_counts” of the corresponding base segment. Further, when the disparity information is updated when “disparity_curve_no_update_flag” is “0”, the disparity information includes “disparity_update”. A 6-bit field of “shifting_interval_counts” indicates a draw factor for adjusting the base segment period (update frame interval), that is, the number of subtracted frames.

  In the example of updating disparity information for each base segment period (BSP) in FIG. 23 described above, the base segment period is adjusted by the draw factor with respect to the update timing of disparity information at time points C to F. The presence of this adjustment information makes it possible to adjust the base segment period (update frame interval), and more accurately convey changes in the time direction (frame direction) of disparity information to the receiving side. .

  In addition, as the adjustment of the base segment period (update frame interval), in addition to the adjustment in the direction of shortening by the number of subtracted frames described above, the adjustment in the direction of increasing by the number of additional frames may be considered. For example, bi-directional adjustment is possible by setting a 5-bit field of “shifting_interval_counts” as a signed integer.

  As described above, by newly defining a byte that has been skipped as a padding byte, transmission (transmission) of disparity information that is sequentially updated in a caption display period and adjustment information of an update frame interval that is added thereto is transmitted. ) Is possible.

  FIG. 75 illustrates a configuration example of a general transport stream (multiplexed data stream) including a video elementary stream, an audio elementary stream, and a caption elementary stream. This transport stream includes PES packets obtained by packetizing each elementary stream. In this configuration example, a PES packet “Video PES” of a video elementary stream is included. In addition, in this configuration example, the PES packet “Audio PES” of the audio elementary stream and the PES packet “SubtitlePES” of the caption elementary stream are included.

  In addition, the transport stream includes a PMT (Program Map Table) as PSI (Program Specific Information). This PSI is information describing to which program each elementary stream included in the transport stream belongs. Further, the transport stream includes an EIT (Event Information Table) as SI (Serviced Information) for managing each event.

  The PMT includes a program descriptor (Program Descriptor) that describes information related to the entire program. The PMT includes an elementary loop having information related to each elementary stream. In this configuration example, there are a video elementary loop, an audio elementary loop, and a subtitle elementary loop. In each elementary loop, information such as a packet identifier (PID) and a stream type (Stream_Type) is arranged for each stream, and although not shown, there is also a descriptor that describes information related to the elementary stream. Be placed.

  In the transmission data generation unit 110B illustrated in FIG. 62, the disparity information (disparity) is embedded (transmitted) in the user data area of the disparity information of the video elementary stream, as illustrated in FIG.

  In the transmission data generation unit 110B illustrated in FIG. 62, stereoscopic image data including left-eye image data and right-eye image data for displaying a stereoscopic image is included in the payload portion of the video elementary stream and transmitted. Also, disparity information for adding disparity to CC data and closed caption information based on the CC data is inserted into the user data area of the header portion of the video elementary stream and transmitted.

  Therefore, on the receiving side (set top box 200), in addition to acquiring stereoscopic image data from this video elementary stream, CC data and disparity information can be easily acquired. On the receiving side, appropriate parallax can be given to the same closed caption information superimposed on the left eye image and the right eye image using the parallax information. Therefore, in the display of closed caption information, the perspective consistency with each object in the image can be maintained in an optimum state.

  In addition, in the transmission data generation unit 110B illustrated in FIG. 62, disparity information (see “disparity_update” in FIGS. 65, 68, and 73) that is sequentially updated within the caption display period can be inserted. Therefore, on the receiving side (set top box 200), the parallax to be added to the closed caption information can be dynamically changed in conjunction with the change of the image content.

  In addition, in the transmission data generation unit 110B illustrated in FIG. 62, the disparity information sequentially updated within the caption display period includes the disparity information of the first frame in the caption (closed caption information) display period, and the subsequent update frame interval. It is assumed to be composed of disparity information of each frame. Therefore, the amount of transmission data can be reduced, and the memory capacity for holding the parallax information can be greatly saved on the receiving side.

  62, “disparity_temporal_extension ()” included in “caption_disparity_data ()” has the same structure as “disparity_temporal_extension ()” included in the SCS segment described above (FIG. 62). 21). Therefore, although detailed description is omitted, the transmission data generation unit 110B illustrated in FIG. 62 can obtain the same effect as the transmission data generation unit 110 illustrated in FIG. 2 by the structure of “disparity_temporal_extension ()”.

"Configuration example of transmission data generator"
FIG. 76 illustrates a configuration example of the bit stream processing unit 201B of the set-top box 200 corresponding to the transmission data generation unit 110B illustrated in FIG. 62 described above. The bit stream processing unit 201B has a configuration corresponding to the transmission data generation unit 110B shown in FIG. The bit stream processing unit 201B includes a demultiplexer 241, a video decoder 242, and a CC decoder 243. The bit stream processing unit 201B includes a stereoscopic image CC generating unit 244, a parallax information extracting unit 245, a parallax information processing unit 246, a video superimposing unit 247, and an audio decoder 248.

  The demultiplexer 241 extracts video and audio packets from the bit stream data BSD and sends them to each decoder. The video decoder 242 performs processing opposite to that of the video encoder 132 of the transmission data generation unit 110B described above. That is, the video decoder 242 reconstructs a video elementary stream from the video packet extracted by the demultiplexer 241, performs decoding processing, and generates stereoscopic image data including left-eye image data and right-eye image data. Get.

  The transmission method of the stereoscopic image data is, for example, the first transmission method (“Top & Bottom” method) described above, the second transmission method (“Side By Side” method), and the third transmission method (“Frame”). Sequential ”method) (see FIGS. 4A to 4C). The video decoder 242 sends the stereoscopic image data to the video superimposing unit 247.

  The CC decoder 243 extracts CC data from the video video elementary stream reconstructed by the video decoder 242. Then, the CC decoder 243 acquires closed caption information (caption character code) for each caption window (Caption Window), and control data for the superimposition position and the display time from the CC data.

  The disparity information extracting unit 245 extracts disparity information from the video elementary stream obtained through the video decoder 242. This disparity information is associated with closed caption data (caption character code) for each caption window acquired by the CC decoder 243 described above. This disparity information is a disparity vector (individual disparity vector) for each caption window, or a disparity vector common to each caption window (common disparity vector).

  The disparity information extracting unit 245 acquires disparity information that is commonly used within the caption display period, or disparity information that is sequentially updated within the caption display period. The parallax information extraction unit 245 sends this parallax information to the stereoscopic image CC generation unit 244 via the parallax information processing unit 246. As described above, the disparity information sequentially updated within the caption display period is composed of the disparity information of the first frame in the caption display period and the disparity information of the frame for each subsequent base segment period (update frame interval). ing.

  The parallax information processing unit 246 sends the parallax information used in common during the caption display period to the stereoscopic image CC generation unit 244 as it is. On the other hand, the disparity information processing unit 246 performs interpolation processing on the disparity information sequentially updated within the caption display period, and generates disparity information at an arbitrary frame interval, for example, one frame interval within the caption display period. To the CC generating unit 244 for stereoscopic images. As the interpolation processing, the disparity information processing unit 246 performs not the linear interpolation processing but the interpolation processing accompanied by the low-pass filter (LPF) processing in the time direction (frame direction), and the disparity information at a predetermined frame interval after the interpolation processing. In the time direction (frame direction) is smoothed (see FIG. 31).

  The stereoscopic image CC generating unit 244 generates, for each caption window (Caption Window), left-eye closed caption information (caption) and right-eye closed caption information (caption) to be superimposed on the left-eye image and the right-eye image, respectively. Generate data. This generation process is performed based on the closed caption data and the superimposed position control data obtained by the CC decoder 243 and the disparity information (disparity vector) sent from the disparity information extracting unit 245 through the disparity information 246. Then, the stereoscopic image CC generating unit 244 outputs left-eye caption and left-eye caption data (bitmap data).

  In this case, the left eye caption and the left eye caption are the same information. However, the superimposed position in the image is shifted in the horizontal direction by, for example, the parallax vector between the left-eye caption and the right-eye caption. As a result, the same subtitle superimposed on the left eye image and the right eye image can be used with parallax adjusted according to the perspective of each object in the image. The perspective consistency between each object within is maintained.

  Here, for example, when only the disparity information (disparity vector) that is commonly used within the caption display period is transmitted from the disparity information processing unit 246, the stereoscopic image CC generating unit 244 uses the disparity information. . In addition, for example, when only the disparity information (disparity vector) that is sequentially updated within the caption display period is sent from the disparity information processing unit 246, the stereoscopic image CC generating unit 244 uses the disparity information. Further, the stereoscopic image CC generating unit 244 uses, for example, when the parallax information sequentially updated within the caption display period is also sent from the parallax information processing unit 246.

  Which one is used is constrained by information (see “rendering_level” in FIG. 72) indicating the disparity information (disparity) correspondence level that is essential on the reception side (decoder side), for example, as described above. The In this case, for example, when “00”, it depends on the user setting. By using the disparity information that is sequentially updated within the caption display period, it is possible to dynamically change the disparity to be given to the left eye and the right eye in conjunction with the change in the image content.

  The video superimposing unit 247 performs the left-eye and left-eye caption data generated by the stereoscopic image CC generating unit 244 for the stereoscopic image data (left-eye image data and right-eye image data) obtained by the video decoder 242. (Bitmap data) is superimposed to obtain display stereoscopic image data Vout. Then, the video superimposing unit 247 outputs the display stereoscopic image data Vout to the outside of the bit stream processing unit 201B.

  Also, the audio decoder 248 performs a process reverse to that of the audio encoder 133 of the transmission data generation unit 110B described above. That is, the audio decoder 248 reconstructs an audio elementary stream from the audio packet extracted by the demultiplexer 241 and performs a decoding process to obtain audio data Aout. The audio decoder 248 then outputs the audio data Aout to the outside of the bit stream processing unit 201B.

  The operation of the bit stream processing unit 201B shown in FIG. 76 will be briefly described. The bit stream data BSD output from the digital tuner 204 (see FIG. 29) is supplied to the demultiplexer 241. In the demultiplexer 241, video and audio packets are extracted from the bit stream data BSD and supplied to each decoder. The video decoder 242 reconstructs a video elementary stream from the video packet extracted by the demultiplexer 241, further performs decoding processing, and generates stereoscopic image data including left-eye image data and right-eye image data. can get. The stereoscopic image data is supplied to the video superimposing unit 247.

  The video video elementary stream reconstructed by the video decoder 242 is supplied to the CC decoder 243. The CC decoder 243 extracts CC data from the video elementary stream. In the CC decoder 243, closed caption information (caption character code) for each caption window (caption window), and control data for the superimposed position and display time are obtained from the CC data. The closed caption information and the control data of the superimposed position and the display time are supplied to the stereoscopic image CC generator 244.

  The video video elementary stream reconstructed by the video decoder 242 is supplied to the disparity information extracting unit 245. The disparity information extracting unit 245 extracts disparity information from the video elementary stream. This disparity information is associated with closed caption data (caption character code) for each caption window acquired by the CC decoder 243 described above. The parallax information is supplied to the stereoscopic image CC generating unit 244 through the parallax information processing unit 246.

  The disparity information processing unit 246 performs the following processing on the disparity information that is sequentially updated within the caption display period. That is, the disparity information processing unit 246 performs interpolation processing with LPF processing in the time direction (frame direction), and generates disparity information at an arbitrary frame interval within the caption display period, for example, one frame interval. And sent to the CC generating unit 244 for stereoscopic images.

  In the stereoscopic image CC generating unit 244, for each caption window (Caption Window), left-eye closed caption information (caption) and right-eye closed caption information (caption) to be superimposed on the left-eye image and right-eye image, respectively. Data is generated. This generation process is performed based on the closed caption data and superposition position control data obtained by the CC decoder 243 and the disparity information (disparity vector) supplied from the disparity information extracting unit 245 through the disparity information processing unit 246.

  In the stereoscopic image CC generating unit 244, a shift process for adding parallax is performed on either or both of the left-eye closed caption information and the right-eye closed caption information. In this case, when the disparity information supplied through the disparity information processing unit 246 is disparity information commonly used in each frame, the common caption information superimposed on the left eye image and the right eye image is included in the common caption information. Parallax is given based on the parallax information. Further, when the disparity information is disparity information that is sequentially updated in each frame, the disparity is based on the disparity information updated for each frame in the closed caption information superimposed on the left eye image and the right eye image. Is granted.

  As described above, the data (bitmap data) of the closed caption information of the left eye and the right eye generated for each caption window (Caption Window) by the stereoscopic image CC generating unit 244, together with display time control data, This is supplied to the video superimposing unit 247. The video superimposing unit 247 superimposes the closed caption information data supplied from the stereoscopic image CC generating unit 244 on the stereoscopic image data (left-eye image data and right-eye image data) obtained by the video decoder 242. As a result, display stereoscopic image data Vout is obtained.

  Also, the audio decoder 248 reconstructs an audio elementary stream from the audio packet extracted by the demultiplexer 241, further performs decoding processing, and audio data corresponding to the display stereoscopic image data Vout described above. Aout is obtained. The audio data Aout is output to the outside of the bit stream processing unit 201A.

  In the bit stream processing unit 201B illustrated in FIG. 76, stereoscopic image data can be acquired from the payload portion of the video elementary stream, and CC data and disparity information can be acquired from the user data area of the header portion. Therefore, appropriate disparity can be given to the closed caption information superimposed on the left eye image and the right eye image by using the disparity information suitable for the closed caption information. Therefore, in the display of closed caption information, perspective consistency with each object in the image can be maintained in an optimum state.

  In addition, the disparity information extracting unit 245 of the bitstream processing unit 201B illustrated in FIG. 76 acquires disparity information that is commonly used within the caption display period, or is updated sequentially with the disparity information that is sequentially updated within the caption display period. The The stereoscopic image CC generating unit 244 uses the disparity information that is sequentially updated within the caption display period, so that the disparity to be added to the closed caption information superimposed on the left eye image and the right eye image is included in the image content. It becomes possible to change dynamically in conjunction with the change.

  In addition, in the disparity information processing unit 246 of the bit stream processing unit 201B illustrated in FIG. 76, an interpolation process is performed on disparity information sequentially updated within the caption display period, and the disparity at an arbitrary frame interval within the caption display period. Information is generated. In this case, even when disparity information is transmitted from the transmission side (broadcast station 100) every base segment period (update frame interval) such as 16 frames, the closed The parallax given to the caption information can be controlled at fine intervals, for example, for each frame.

  In addition, the disparity information processing unit 246 of the bit stream processing unit 201B illustrated in FIG. 76 performs an interpolation process with a low-pass filter process in the time direction (frame direction). Therefore, even when disparity information is transmitted from the transmission side (broadcast station 100) every base segment period (updated frame interval), the change in the time direction (frame direction) of the disparity information after the interpolation processing is gently performed. Yes (see FIG. 31). Therefore, it is possible to suppress a sense of incongruity due to discontinuity in the disparity transition of the disparity given to the closed caption information superimposed on the left eye image and the right eye image.

<2. Modification>
Note that FIG. 77 illustrates another example structure (Syntax) of “disparity_temporal_extension ()”. FIG. 78 shows main data defining contents (semantics) related to the structural example. An 8-bit field of “disparity_update_count” indicates the number of updates of disparity information (disparity). There is a for loop regulated by the number of updates of the parallax information.

  The 8-bit field of “interval_count” indicates the update period as a multiple of an interval period (Interval period) indicated by “interval_PTS” described later. The 8-bit field of “disparity_update” indicates disparity information for the corresponding update period. Note that “disparity_update” at k = 0 is an initial value of disparity information that is sequentially updated at update frame intervals within the caption display period, that is, disparity information of the first frame in the caption display period.

  When “disparity_temporal_extension ()” of the structure shown in FIG. 77 is used instead of “disparity_temporal_extension ()” of the structure shown in FIG. 21, for example, substantial information of the SCS (Subregion Composition segment) shown in FIG. A 33-bit field of “interval_PTS” is provided in the included portion. This “interval_PTS” designates an interval period (Interval period) in units of 90 KHz. That is, “interval_PTS” represents a value obtained by measuring this interval period (Interval period) with a 90 KHz clock in a 33-bit length.

  79 and 80 illustrate an example of disparity information update in the case where “disparity_temporal_extension ()” having the structure illustrated in FIG. 77 is used. FIG. 79 shows a case where the interval period (Interval period) indicated by “interval_PTS” is fixed and the period is equal to the update period. In this case, “interval_count” is “1”.

  On the other hand, FIG. 80 is a general example and shows an example of updating disparity information when the interval period (Interval period) indicated by “interval_PTS” is a short period (for example, a frame period may be used). In this case, “interval_count” is M, N, P, Q, and R in each update period. 79 and 80, “A” indicates the start frame (start time) of the caption display period, and “B” to “F” indicate subsequent update frames (update time).

  Even when disparity information sequentially updated within the caption display period is sent to the receiving side (such as the set top box 200) using “disparity_temporal_extension ()” having the structure shown in FIG. 77, the receiving side is the same as described above. Can be processed. That is, in this case as well, on the receiving side, it is possible to generate and use disparity information at an arbitrary frame interval, for example, one frame interval, by performing an interpolation process on the disparity information for each update period.

  FIG. 81A shows a configuration example of a subtitle data stream when “disparity_temporal_extension ()” having the structure shown in FIG. 77 is used. The PES header includes time information (PTS). Moreover, each segment of DDS, PCS, RCS, CDS, ODS, SCS, and EOS is included as PES payload data. These are transmitted all at once before the start of the subtitle display period. Although not described above, the configuration example of the subtitle data stream in the case of using “disparity_temporal_extension ()” having the structure shown in FIG. 21 is the same.

  Note that disparity information that is sequentially updated within the caption display period can be sent to the receiving side (such as the set top box 200) without including “disparity_temporal_extension ()” in the SCS segment. In this case, “temporal_extension_flag = 0” is set, and only “subregion_disparity” is encoded in the SCS segment (see FIG. 18). In this case, an SCS segment is inserted into the subtitle data stream at every update timing. In this case, although not shown, a time difference value (delta_PTS) is added to the SCS segment at each update timing as time information.

  FIG. 81 (b) shows a configuration example of the subtitle data stream in that case. First, each segment of DDS, PCS, RCS, CDS, ODS, and SCS is transmitted as PES payload data. Thereafter, a predetermined number of SCS segments in which the time difference value (delta_PTS) and the disparity information are updated are transmitted at the timing of updating. Finally, an EOS segment is transmitted along with the SCS segment.

  FIG. 82 illustrates an example of updating disparity information when sequentially transmitting SCS segments as described above. In FIG. 82, “A” indicates the start frame (start point) of the caption display period, and “B” to “F” indicate subsequent update frames (update point).

  Even when the SCS segment is sequentially transmitted and the disparity information sequentially updated within the caption display period is transmitted to the receiving side (such as the set top box 200), the receiving side can perform the same processing as described above. That is, in this case as well, on the receiving side, it is possible to generate and use disparity information at an arbitrary frame interval, for example, one frame interval, by performing an interpolation process on the disparity information for each update period.

  Note that the use of “disparity_temporal_extension ()” having the structure shown in FIG. 77 is performed using the description of the transmission data generation unit 110 shown in FIG. 2 (FIG. 21 and the like). However, although detailed explanation is omitted, it is needless to say that this is possible not only in the DVB system but also in the ARIB system and the CEA system.

  FIG. 83 shows an example of updating disparity information (disparity) similar to FIG. 80 described above. The update frame interval is represented by a multiple of an interval period (ID: Interval Duration) as a unit period. For example, the update frame interval “DivisionPeriod 1” is represented by “ID * M”, the update frame interval “Division Period 2” is represented by “ID * N”, and the following update frame intervals are similarly represented. In the example of updating disparity information shown in FIG. 83, the update frame interval is not fixed, and the update frame interval is set according to the disparity information curve.

  Also, in this update example of disparity information (disparity), on the receiving side, the start frame (start time) T1_0 of the caption display period is a PTS (Presentation Time Stamp) inserted in the header of the PES stream including this disparity information. Given in. Then, on the receiving side, each update time of the disparity information is obtained based on information on interval periods (information on unit periods) that is information on each update frame interval and information on the number of the interval periods.

  In this case, each update time is sequentially obtained from the start frame (start time) T1_0 of the caption display period based on the following equation (1). In this equation (1), “interval_count” indicates the number of interval periods, and is a value corresponding to M, N, P, Q, R, and S in FIG. In this equation (1), “interval_time” is a value corresponding to the interval period (ID) in FIG.

    Tm_n = Tm_ (n-1) + (interval_time * interval_count) (1)

  For example, in the update example shown in FIG. 83, each update time is obtained as follows based on the equation (1). That is, the update time T1_1 is obtained as “T1_1 = T1_0 + (ID * M)” using the start time (T1_0), the interval period (ID), and the number (M). Further, the update time T1_2 is obtained as “T1_2 = T1_1 + (ID * N)” using the update time (T1_1), the interval period (ID), and the number (N). Each subsequent update time is obtained in the same manner.

  In the update example shown in FIG. 83, on the reception side, interpolation processing is performed on the disparity information sequentially updated within the caption display period, and disparity information at an arbitrary frame interval, for example, one frame interval within the caption display period. Generated and used. For example, the interpolation process is not a linear interpolation process but an interpolation process with a low-pass filter (LPF) process in the time direction (frame direction), so that the time direction of the disparity information at a predetermined frame interval after the interpolation process is performed. The (frame direction) changes gently. A broken line a in FIG. 83 shows an example of LPF output.

  FIG. 84 illustrates a configuration example of the subtitle data stream. The PES header includes time information (PTS). The PES payload data includes DDS, PCS, RCS, CDS, ODS, DSS (Display Signaling Segment), and EOS segments. These are transmitted all at once before the start of the subtitle display period.

  The DSS segment includes disparity information for realizing disparity information update as shown in FIG. 83 described above. That is, this DSS includes disparity information of the start frame (start time) of the caption display period, and disparity information of frames for each subsequent update frame interval. In addition, information on the interval period (unit period information) and information on the number of the interval periods are added to the disparity information as the update frame interval information. Thereby, on the receiving side, each update frame interval can be easily obtained by calculating “unit period * number”.

  In addition, in the DSS segment, as disparity information sequentially updated in the caption display period, either disparity information in units of regions or subregions included in this region and disparity information in units of pages including all regions are included. Or both are selectively included. Also, the DSS includes disparity information in units of regions or subregions included in this region and disparity information in units of pages including all regions as disparity information fixed in the caption display period.

  FIG. 85 shows a display example of subtitles as subtitles. In this display example, the page area (Area for Page_default) includes two regions (Region 1 and Region 2) as subtitle display areas. A region includes one or more subregions. Here, it is assumed that the region includes one subregion, and the region region and the subregion region are equal.

  FIG. 86 shows the case where each region and page disparity is included when the DSS segment includes both disparity information in units of regions and disparity information in units of pages as disparity information (Disparity) sequentially updated during the caption display period. An example of the parallax information curve is shown. Here, the parallax information curve of the page is configured to take the minimum value of the parallax information curves of the two regions.

  Regarding the region 1 (Region1), there are seven pieces of disparity information of T1_0 which is a start time and T1_1, T1_2, T1_3,. For region 2 (Region2), there are eight pieces of disparity information of T2_0 which is a start time and T2_1, T2_2, T2_3,..., T2_7 which are update times thereafter. Further, regarding the page (Page_default), there are seven pieces of disparity information, that is, T0_0 that is a start time and T0_1, T0_2, T0_3,.

  FIG. 87 shows a structure in which the disparity information of the page and each region shown in FIG. 86 is sent. First, the page layer will be described. In this page layer, “page_default_disparity” that is a fixed value of disparity information is arranged. For the disparity information sequentially updated in the caption display period, “interval_count” indicating the number of interval periods and “disparity_page_updete” indicating disparity information corresponding to the start time and each subsequent update time are sequentially arranged. The The start time “interval_count” is set to “0”.

  Next, the region layer will be described. For region 1 (subregion 1), “subregion_disparity_integer_part” and “subregion_disparity_fractional_part” which are fixed values of disparity information are arranged. Here, “subregion_disparity_integer_part” indicates an integer part of disparity information, and “subregion_disparity_fractional_part” indicates a decimal part of disparity information.

  For the disparity information sequentially updated in the caption display period, “interval_count” indicating the number of interval periods, “disparity_region_updete_integer_part” and “disparity_region_updete_fractional_part” indicating the disparity information are associated with the start time and each subsequent update time. Are arranged sequentially. Here, “disparity_region_updete_integer_part” indicates an integer part of disparity information, and “disparity_region_updete_fractional_part” indicates a decimal part of disparity information. The start time “interval_count” is set to “0”.

  Region 2 (subregion 2) is the same as region 1 described above, and “subregion_disparity_integer_part” and “subregion_disparity_fractional_part”, which are fixed values of disparity information, are arranged. For the disparity information sequentially updated in the caption display period, “interval_count” indicating the number of interval periods, “disparity_region_updete_integer_part” and “disparity_region_updete_fractional_part” indicating the disparity information are associated with the start time and each subsequent update time. Are arranged sequentially.

  88 to 91 show a structure example (syntax) of DSS (Disparity_Signaling_Segment). 92 and 93 show the main data definition contents (semantics) of the DSS. This structure includes information of “Sync_byte”, “segment_type”, “page_id”, “segment_length”, and “dss_version_number”. “Segment_type” is 8-bit data indicating a segment type, and is a value indicating DSS here. “Segment_length” is 8-bit data indicating the number of subsequent bytes.

  The 1-bit flag of “disparity_page_update_sequence_flag” indicates whether or not there is disparity information that is sequentially updated during the caption display period as disparity information in units of pages. “1” indicates that it exists, and “0” indicates that it does not exist. The 1-bit flag of “disparity_region_update_sequence_present_flag” indicates whether or not there is disparity information that is sequentially updated in the caption display period as disparity information in units of regions (subregions). “1” indicates that it exists, and “0” indicates that it does not exist. Note that “disparity_region_update_sequence_present_flag” is sent for the purpose of easily determining whether “disparity update” relating to at least one region (region) exists outside the while loop. Whether or not to send “disparity_region_update_sequence_present_flag” is up to the sender.

  The 8-bit field of “page_default_disparity” indicates fixed disparity information in units of pages, that is, disparity information that is commonly used in the caption display period. When the “disparity_page_update_sequence_flag” flag is “1”, “disparity_page_update_sequence ()” is read.

  FIG. 90 illustrates a structure example (Syntax) of “disparity_page_update_sequence ()”. “Disparity_page_update_sequence_length” is 8-bit data indicating the number of subsequent bytes. “Segment_NOT_continued_flag” indicates whether the packet is completed in the current packet. “1” indicates that the current packet is completed. “0” indicates that the current packet is not completed, and there is a subsequent part in the next packet.

  A 24-bit field of “interval_time [23..0]” specifies an interval period (Interval Duration) (see FIG. 83) as a unit period in units of 90 KHz. That is, “interval_time [23..0]” represents a value obtained by measuring the interval duration with a 90 KHz clock in a 24-bit length.

  The reason why the PTS inserted in the header portion of the PES is 33 bits long is 24 bits long for the following reason. That is, a time exceeding 24 hours can be expressed with a 33-bit length, but this interval period (Interval Duration) within the caption display period is an unnecessary length. In addition, by using 24 bits, the data size can be reduced and compact transmission can be performed. Further, 24 bits are 8 × 3 bits, and byte alignment is easy.

  The 8-bit field of “division_period_count” indicates the number of periods (Division Period) for transmitting disparity information. For example, in the case of the update example shown in FIG. 83, this number is “7” corresponding to the start time T1_0 and the subsequent update times T1_1 to T1_6. The following for loop is repeated by the number indicated by the 8-bit field of “division_period_count”.

  An 8-bit field of “interval_count” indicates the number of interval periods. For example, in the case of the update example shown in FIG. 83, M, N, P, Q, R, and S correspond. An 8-bit field of “disparity_page_update” indicates disparity information. “Interval_count” is set to “0” corresponding to the disparity information at the start time (the initial value of the disparity information). That is, when “interval_count” is “0”, “disparity_page_update” indicates disparity information (initial value of disparity information) at the start time.

  The while loop of FIG. 89 is repeated when the data length processed so far (processed_length) has not reached the segment data length (segment_length). In this while loop, disparity information in units of regions or sub-regions in the region is arranged. Here, the region includes one or more subregions, and the subregion region and the region region may be the same.

  In this while loop, information of “region_id” and “subregion_id” is included. When the subregion region is the same as the region region, “subregion_id” is set to “0”. Therefore, when “subregion_id” is not “0”, position information of “subregion_horizontal_position” and width information of “subregion_width” indicating the subregion region are included in this while loop.

  The 1-bit flag of “disparity_region_update_sequence_flag” indicates whether or not there is disparity information that is sequentially updated in the caption display period as disparity information in units of regions (subregions). “1” indicates that it exists, and “0” indicates that it does not exist. The 8-bit field of “subregion_disparity_integer_part” indicates fixed disparity information in units of regions (subregions), that is, an integer part of disparity information that is commonly used in the caption display period. A 4-bit field of “subregion_disparity_fractional_part” indicates fixed disparity information in units of regions (subregions), that is, a decimal part of disparity information that is commonly used in a caption display period.

  When the “disparity_region_update_sequence_flag” flag is “1”, “disparity_region_update_sequence ()” is read. FIG. 91 illustrates a structure example (Syntax) of “disparity_page_update_sequence ()”. “Disparity_region_update_sequence_length” is 8-bit data indicating the number of subsequent bytes. “Segment_NOT_continued_flag” indicates whether the packet is completed in the current packet. “1” indicates that the current packet is completed. “0” indicates that the current packet is not completed, and there is a subsequent part in the next packet.

  A 24-bit field of “interval_time [23..0]” specifies an interval period (Interval Duration) (see FIG. 83) as a unit period in units of 90 KHz. That is, “interval_time [23..0]” represents a value obtained by measuring the interval duration with a 90 KHz clock in a 24-bit length. The 24-bit length is the same as that described in the structure example (Syntax) of “disparity_page_update_sequence ()” described above.

  The 8-bit field of “division_period_count” indicates the number of periods (Division Period) for transmitting disparity information. For example, in the case of the update example shown in FIG. 83, this number is “7” corresponding to the start time T1_0 and the subsequent update times T1_1 to T1_6. The following for loop is repeated by the number indicated by the 8-bit field of “division_period_count”.

  An 8-bit field of “interval_count” indicates the number of interval periods. For example, in the case of the update example shown in FIG. 83, M, N, P, Q, R, and S correspond. An 8-bit field of “disparity_region_update_integer_part” indicates an integer part of disparity information. A 4-bit field of “disparity_region_update_fractional_part” indicates a decimal part of disparity information. “Interval_count” is set to “0” corresponding to the disparity information at the start time (the initial value of the disparity information). That is, when “interval_count” is “0”, “disparity_region_update_integer_part” and “disparity_region_update_fractional_part” indicate disparity information (initial value of disparity information) at the start time.

  In the above-described embodiment, the image transmission / reception system 10 includes the broadcasting station 100, the set-top box 200, and the television receiver 300. However, the television receiver 300 includes a bit stream processing unit 306 that functions in the same manner as the bit stream processing unit 201 (201A, 201B) in the set top box 200, as shown in FIG. Therefore, as shown in FIG. 94, an image transmission / reception system 10A including a broadcasting station 100 and a television receiver 300 is also conceivable.

  In the above-described embodiment, an example in which a data stream (bit stream data) including stereoscopic image data is broadcast from the broadcast station 100 has been described. However, the present invention can be similarly applied to a system in which the data stream is distributed to the receiving terminal using a network such as the Internet.

  In the above-described embodiment, the set top box 200 and the television receiver 300 are connected via an HDMI digital interface. However, the present invention can be similarly applied even when these are connected by a digital interface (including wireless as well as wired) similar to the HDMI digital interface.

  Further, in the above-described embodiment, a case where a subtitle (caption) is handled as superimposition information is shown. However, the present invention can be similarly applied to other information that handles superimposition information such as graphics information and text information.

  The present invention can be applied to an image transmission / reception system capable of displaying superimposition information such as a subtitle (caption) superimposed on a stereoscopic image.

DESCRIPTION OF SYMBOLS 10,10A ... Image transmission / reception system 100 ... Broadcasting station 110, 110A, 110B ... Transmission data generation part 111,121,131 ... Data extraction part 112,122,132 ... Video encoder 132a. .. Stream formatter 113, 123, 133 ... audio encoder 114 ... subtitle generation unit 115, 125, 135 ... disparity information creation unit 116 ... subtitle processing unit 117 ... display control information generation unit 118 ... Subtitle encoders 119, 127, 136 ... Multiplexer 124 ... Subtitle generator 126 ... Subtitle encoder 134 ... CC encoder 126 ... Multiplexer 200 ... Set top box (STB)
201, 201A, 201B ... bit stream processing unit 202 ... HDMI terminal 203 ... antenna terminal 204 ... digital tuner 205 ... video signal processing circuit 206 ... HDMI transmission unit 207 ... audio Signal processing circuit 211 ... CPU
215: Remote control reception unit 216: Remote control transmitter 221, 231, 241 ... Demultiplexer 222, 232, 242 ... Video decoder 223 ... Subtitle decoder 224 ... Stereo image subtitle generation unit 225 ... Display control unit 226 ... Display control information acquisition unit 227, 236, 246 ... Parallax information processing unit 228, 237, 247 ... Video superposition unit 229, 238, 248 ... Audio decoder 233 ... Subtitle decoder 234: Stereo image caption generation unit 235, 245 ... Parallax information extraction unit 243 ... CC decoder 244 ... Stereo image CC generation unit 300 ... Television receiver (TV)
DESCRIPTION OF SYMBOLS 301 ... 3D signal processing part 302 ... HDMI terminal 303 ... HDMI receiving part 304 ... Antenna terminal 305 ... Digital tuner 306 ... Bit stream processing part 307 ... Video / graphic processing circuit 308 ... Panel drive circuit 309 ... Display panel 310 ... Audio signal processing circuit 311 ... Audio amplification circuit 312 ... Speaker 321 ... CPU
325 ... Remote control receiver 326 ... Remote control transmitter 400 ... HDMI cable

Claims (22)

An image data output unit for outputting stereoscopic image data having left-eye image data and right-eye image data;
A superimposition information data output unit for outputting superimposition information data to be superimposed on an image based on the left eye image data and the right eye image data;
A parallax information output unit for outputting parallax information for shifting the superimposition information to be superimposed on the image based on the left-eye image data and the right-eye image data and providing parallax;
A stereoscopic image data output from the image data output unit, a superimposition information data output from the superimposition information data output unit, and a data transmission unit that transmits parallax information output from the parallax information output unit,
The disparity information is disparity information that is sequentially updated within a predetermined number of frame periods in which the superimposition information is displayed, and the disparity information of the first frame of the predetermined number of frame periods and every subsequent update frame interval. A three-dimensional image data transmission apparatus comprising frame parallax information. The stereoscopic image data transmission device according to claim 1, wherein information of a unit period and information on the number of unit periods are added to the disparity information as information of each update frame interval. The stereoscopic image data transmission device according to claim 2, wherein the information of the unit period is information in which a value obtained by measuring the unit period with a 90 KHz clock is represented by a 24-bit length. The stereoscopic image data transmission device according to claim 1, wherein flag information indicating whether or not the disparity information is updated is added to the disparity information for each frame at each update frame interval. The stereoscopic image data transmission device according to claim 1, wherein information for adjusting the update frame interval is added to the disparity information for each frame at each update frame interval. The stereoscopic image data transmission device according to claim 1, wherein information specifying a frame period is added to the parallax information. The stereoscopic image according to claim 1, wherein the disparity information is disparity information corresponding to specific superimposition information displayed on the same screen and / or disparity information corresponding to a plurality of superimposition information displayed on the same screen. Data transmission device. The stereoscopic image data transmission device according to claim 1, wherein the parallax information is added with information indicating a level corresponding to the parallax information, which is essential when the superimposition information is displayed. The superimposition information data is DVB subtitle data,
The stereoscopic image data transmission device according to claim 1, wherein the data transmission unit transmits the disparity information by including the parallax information in a subtitle data stream including the subtitle data. The stereoscopic image data transmission device according to claim 9, wherein the disparity information is disparity information in units of regions or subregions included in the regions. The stereoscopic image data transmission device according to claim 9, wherein the disparity information is disparity information in units of pages including all regions. The superimposition information data is ARIB subtitle data,
The stereoscopic image data transmission device according to claim 1, wherein the data transmission unit transmits the disparity information by including the parallax information in a caption data stream including the caption data. The superimposition information data is CEA closed caption data,
The stereoscopic image data transmission device according to claim 1, wherein the data transmission unit transmits the disparity information by including the disparity information in a user data area of a video data stream including the closed caption data. The stereoscopic image data transmission device according to claim 13, wherein the superimposition information data is inserted into an extended command based on a CEA table arranged in the user data area. The stereoscopic image data transmission device according to claim 13, wherein the superimposition information data is inserted into the closed caption data arranged in the user data area. An image data output step for outputting stereoscopic image data having left eye image data and right eye image data;
A superimposition information data output step for outputting superimposition information data to be superimposed on an image based on the left eye image data and the right eye image data;
A disparity information output step for outputting disparity information for shifting the superimposition information to be superimposed on the image based on the left eye image data and the right eye image data to give disparity;
A data transmission step for transmitting stereoscopic image data output in the image data output step, superimposition information data output in the superimposition information data output step, and disparity information output in the disparity information output step;
The disparity information is disparity information that is sequentially updated within a predetermined number of frame periods in which the superimposition information is displayed, and the disparity information of the first frame of the predetermined number of frame periods and every subsequent update frame interval. A method for transmitting stereoscopic image data comprising disparity information of a frame. Three-dimensional image data including left-eye image data and right-eye image data, superimposition information data to be superimposed on an image based on the left-eye image data and the right-eye image data, and the left-eye image data and the right-eye image data A data receiving unit that receives parallax information for shifting the superimposition information to be superimposed on the image to give parallax;
The disparity information received by the receiving unit is disparity information that is sequentially updated within a predetermined number of frame periods in which the superimposition information is displayed, and the disparity information of the first frame of the predetermined number of frame periods; It consists of the disparity information of the frame for each subsequent update frame interval,
Using the left eye image data and the right eye image data received by the data receiving unit, the superimposition information data, and the parallax information, the same superimposition information to be superimposed on the left eye image and the right eye image An image data processing unit is further provided that provides parallax and obtains left eye image data on which the superimposition information is superimposed and right eye image data on which the superposition information is superimposed. Stereo image data receiving device.
The image data processing unit performs an interpolation process on disparity information of a plurality of frames that are sequentially updated within the predetermined number of frame periods, and generates disparity information at arbitrary frame intervals within the predetermined number of frame periods. The stereoscopic image data receiving device according to claim 17. The stereoscopic image data receiving device according to claim 18, wherein the interpolation processing includes a low-pass filter processing in a time direction. In the disparity information, information on the unit period and information on the number of unit periods are added as information on the update frame interval.
The image data processing unit
18. The solid according to claim 17, wherein each update time of the parallax information is obtained based on the information of the unit period and the number of pieces of information that are information of the update frame intervals, with the display start time of the superimposition information as a reference. Image data transmission device. The stereoscopic image data transmission device according to claim 20, wherein the display start time of the superimposition information is given by a PTS inserted in a header part of a PES stream including the parallax information. Stereoscopic image data including left-eye image data and right-eye image data, superimposition information data to be superimposed on an image based on the left-eye image data, and superimposition information to be superimposed on an image based on the left-eye image data and the right-eye image data A data receiving step of receiving parallax information for shifting the image to give parallax,
The disparity information received in the reception step is disparity information that is sequentially updated within a predetermined number of frame periods in which the superimposition information is displayed, and the disparity information of the first frame of the predetermined number of frame periods; It consists of the disparity information of the frame for each subsequent update frame interval,
Using the left eye image data and the right eye image data received in the data reception step, the superimposition information data, and the parallax information, the same superimposition information to be superimposed on the left eye image and the right eye image A stereoscopic image data receiving method further comprising an image data processing step of obtaining parallax and obtaining data of a left eye image on which the superimposition information is superimposed and right eye image data on which the superimposition information is superimposed.