JP5928118B2 - Transmitting apparatus, transmitting method, receiving apparatus, and receiving method - Google Patents

Description

  The present technology relates to a transmission device, a transmission method, a reception device, and a reception method, and more particularly to a transmission device and the like for favorably displaying a stereoscopic image (three-dimensional image) on the reception side.

  Conventionally, various systems are known as systems for displaying a three-dimensional image (stereoscopic image). For example, as described in Patent Document 1, a left-eye image and a right-eye image having parallax are alternately displayed on a display at a predetermined cycle, and the left-eye image and the right-eye image are synchronized with the display. A method of observing with shutter glasses including a liquid crystal shutter to be driven is known.

JP-A-9-138384

  As a method for enabling naked-eye viewing of a three-dimensional image (stereoscopic image), a method using a multi-view configuration having N views can be considered. In that case, if image data of all views is transmitted, there is a concern that the transmission band increases. Therefore, instead of transmitting image data of all views, image data of one view or more, for example, two views is transmitted, and image data of views other than the view transmitted on the receiving side is generated by interpolation processing. Is also possible.

  FIG. 31 shows a configuration example of the image transmission / reception system 50 in that case. On the transmission side, for example, images of two views are displayed by the view selector 52 from image data of N views (View 1... View N) obtained by imaging with the N cameras 51-1 to 51-N. Data is selected. Then, for example, two video streams (1st video, 2nd video) obtained by encoding the image data of the two views by the encoder 53 are transmitted toward the receiving side.

  On the receiving side, two video streams sent from the transmitting side are decoded by the decoder 54 to obtain image data of two views. Then, the interpolation processing unit 55 performs interpolation processing based on the image data of these two views, and generates image data of other views that have not been transmitted. As a result, N views (View 1...・ Image data of View N) can be obtained. Thereby, on the receiving side, it is possible to view the three-dimensional image (stereoscopic image) with N-view image data with naked eyes.

  For example, as a method of transmitting image data of two views, (1) a method of transmitting image data of two views at both ends of N views, and (2) an inner 2 of N views. Two methods of transmitting image data of one view are conceivable.

  With regard to the transmission method (1), when the number of multi-view views increases, the relative parallax between the two views at both ends of the transmission increases. For this reason, it is difficult to interpolate around the occlusion associated with the processing of fine parts when interpolating view image data that is not transmitted, and the quality of the reproduced image may be a problem.

  FIG. 32 schematically shows a display unit on the receiving side when the number of views is 5 in this transmission method. Here, “View_0” is the center view, “View_1” is the view one right from the center, “View_2” is the view one left from the center, “View_3” is two views right from the center, that is, the rightmost view, “View_4” indicates a view that is two left from the center, that is, the leftmost view. In this case, only the image data of the view “View_3” and “View_4” is transmitted from the transmission side, the image data of the view “View_3” and “View_4” is received on the reception side, and other “View_0” and “View_1” ”And“ View_2 ”view image data is obtained by interpolation processing. On the receiving side, the images of these five views are synthesized and displayed on the display unit for the naked-eye viewing of the three-dimensional image (stereoscopic image). FIG. 32 shows a lenticular lens, but a parallax barrier or the like may be used instead. The same applies to FIG. 33 below.

  Regarding the transmission method (2), image data of a so-called conventional stereo view is transmitted, and image data of a view that is not transmitted is interpolated on the receiving side. At that time, the interpolation of the image data of the views inside the two views constituting the stereo view can be synthesized by interpolation processing. However, the interpolation of the image data of the view outside the stereo view is synthesized by extrapolation processing. In the synthesis by extrapolation processing, it is difficult to maintain high image quality with respect to end point processing such as occlusion, which causes deterioration in image quality.

  FIG. 33 schematically shows a display unit on the receiving side when the number of views is 5 in this transmission method. Here, “View_0” is the center view, “View_1” is the view one right from the center, “View_2” is the view one left from the center, “View_3” is two views right from the center, that is, the rightmost view, “View_4” indicates a view that is two left from the center, that is, the leftmost view. In this case, only the image data of the view “View_1” and “View_2” is transmitted from the transmission side, the image data of the view “View_1” and “View_2” is received on the reception side, and other “View_0” and “View_3” "," View_4 "view image data is obtained by interpolation processing. On the receiving side, the images of these five views are synthesized and displayed on the display unit for the naked-eye viewing of the three-dimensional image (stereoscopic image).

  An object of the present technology is to enable a stereoscopic image display process to be favorably performed on the reception side.

The concept of this technology is
An image data acquisition unit that acquires image data of a predetermined number of views for stereoscopic image display;
An image data transmitting unit that transmits a container of a predetermined format including a video stream obtained by encoding the acquired image data;
A transmission apparatus, comprising: a view configuration information insertion unit that inserts view configuration information including information indicating a relative positional relationship of at least the predetermined number of views into a layer of the video stream.

  In the present technology, the image data transmission unit acquires image data of a predetermined number of views. For example, image data of at least the left end view and the right end view and an intermediate view located between the left end and the right end, for example, image data of the center view, among a plurality of views for stereoscopic image display are acquired. . The image data in this case is, for example, data obtained by being imaged by a camera or data obtained by being read from a storage medium.

  The image data transmission unit transmits a container having a predetermined format including a video stream obtained by encoding the acquired image data. For example, the container may be a transport stream (MPEG-2 TS) adopted in the digital broadcasting standard. Further, for example, the container may be MP4 used for Internet distribution or the like, or a container of other formats.

  For example, in the video stream included in the container, the image data of the left end view and the right end view may be encoded as one picture data. Further, for example, in the video stream included in the container, the image data of the left end view and the right end view may be interleaved and encoded as one picture data.

  Further, for example, a video stream included in a container may include data of one or more pictures. In this case, for example, when the video stream included in the container includes encoded data of a plurality of pictures, information indicating a boundary may be arranged between the encoded data of each picture. By arranging the information indicating the boundary in this way, it is possible to instantaneously access the head data of each picture.

  The view configuration information insertion unit inserts view configuration information including information indicating the relative positional relationship of at least a predetermined number of views into the layer of the video stream. In this technique, view configuration information is inserted and transmitted in the layer of the video stream in this way, so that the relative positional relationship of each view can be easily grasped on the receiving side, and the stereoscopic image display processing is improved. It can be carried out.

  In the present technology, for example, the image data acquisition unit includes at least the left end view and the right end view of the plurality of views for stereoscopic image display, and an intermediate view positioned between the left end and the right end. The image data may be acquired. In this case, not only the image data of the left end view and the right end view but also the image data of the intermediate view is transmitted, so the relative parallax between the views is small, and fine details when interpolating the image data of other views are small. Interpolation around the occlusion associated with the processing of the part becomes easy, and the quality of the reproduced image can be improved. In addition, since the image data of the left end view and the right end view is transmitted, all of the image data of the non-transmitted view can be synthesized by interpolation processing, and it is easy to maintain high image quality with respect to end point processing such as occlusion. Become.

  In the present technology, for example, a view configuration information insertion unit that inserts view configuration information related to image data in the video stream may be further provided in the layer of the video stream. With this view configuration information, the receiving side can perform appropriate and efficient processing for performing naked-eye viewing of a three-dimensional image (stereoscopic image) based on image data of a plurality of views.

  In this case, for example, an identification information insertion unit that inserts identification information for identifying whether or not view configuration information is inserted into the video stream layer may be further provided in the container layer. With this identification information, the reception side can easily identify whether or not view configuration information is inserted in the layer of the video stream.

  For example, when image data of a predetermined view is encoded as data of one picture in a video stream included in a container, the position of the predetermined view is included in the view configuration information inserted in the layer of the video stream. The information to show may be included.

  Further, for example, when image data of two views is interleaved and encoded as data of one picture in a video stream included in a container, the view configuration information inserted in the layer of this video stream includes: Information indicating the positions of these two views may be included. In this case, for example, the view configuration information may further include information indicating the type of interleaving performed on the image data of the two views.

  Further, for example, the view configuration information inserted in the layer of the video stream may include information indicating whether or not data of a plurality of pictures is encoded in one access unit of the video stream. For example, the view configuration information inserted in the layer of the video stream may include information indicating whether or not the image data of the view essential for image display is the encoded video stream. . Further, for example, the view configuration information inserted in the layer of the video stream may include pixel ratio information for a predetermined horizontal and / or vertical resolution.

  Further, in the present technology, for example, a parallax data acquisition unit that acquires parallax data between the views is further provided, and the image data transmission unit includes, in addition to the video stream obtained by encoding the acquired image data. A container of a predetermined format including a disparity stream obtained by encoding the obtained disparity data may be transmitted. In this case, the reception side easily interpolates and synthesizes the image data of each view that is not transmitted based on the received parallax data without performing the process of generating the parallax data from the received image data of each view. It becomes possible.

Other concepts of this technology are
Obtained by encoding image data of at least the left end view and the right end view, and intermediate view image data positioned between the left end and the right end among a plurality of views for stereoscopic image display An image data receiving unit for receiving a container of a predetermined format including a video stream;
An image data acquisition unit that decodes a video stream included in the container to obtain image data of each view;
The reception apparatus includes an interpolation processing unit that acquires image data of a predetermined number of views positioned between the views based on the parallax data of the views by interpolation processing.

  In the present technology, the image data receiving unit includes at least the left-end view and the right-end view image data, and the intermediate view image data positioned between the left-end and the right-end among the plurality of views for stereoscopic image display. A stream of a predetermined format including a video stream obtained by encoding is received. The video data included in the stream is decoded by the image data acquisition unit, and image data of each view is obtained. Then, based on the parallax data between the views, the interpolation processing unit acquires image data of a predetermined number of views located between the views by the interpolation processing.

  For example, the container may include a parallax data obtained by encoding the parallax data, and further include a parallax data acquisition unit that obtains the parallax data by decoding the parallax stream included in the container. In addition, for example, a parallax data generation unit that generates parallax data based on the image data of each view obtained by the image data acquisition unit may be further provided.

  As described above, in the present technology, among the plurality of views for displaying the stereoscopic image, at least the image data of the left end view and the right end view and the image data of the intermediate view positioned between the left end and the right end are received. The other views are obtained by interpolation processing based on the parallax data. Therefore, it is possible to satisfactorily perform autostereoscopic viewing of a stereoscopic image with a multiview configuration.

  In other words, not only the image data of the left end view and the right end view but also the image data of the intermediate view between these views is received, so that the relative parallax between the views is small and the image data of the view that is not transmitted is interpolated. Interpolation around the occlusion associated with the processing of fine details becomes easy, and the quality of the reproduced image can be improved. In addition, since the image data of the left end view and the right end view is received, all of the image data of the view that is not transmitted can be synthesized by interpolation processing, and it is easy to maintain high image quality for end point processing such as occlusion. Become.

  According to the present technology, stereoscopic image display processing can be favorably performed on the reception side.

It is a block diagram which shows the structural example of the image transmission / reception system as embodiment. It is a figure for demonstrating the example in which the image data of each view of a center, a left end, and a right end are each encoded as one picture data. FIG. 4 is a diagram for explaining an example in which image data of a central view is encoded as data of one picture, and image data of two views at the left end and the right end is interleaved and encoded as data of one picture. is there. It is a figure which shows an example of the video stream containing the encoding data of several pictures. It is a figure which shows the example in case the coding data of three pictures coexist in one video stream. FIG. 6 schematically shows a display unit of a receiver when the number of views is 5 in a method of transmitting image data of a left end view and a right end view among N views and a central view located between them. FIG. It is a block diagram which shows the structural example of the transmission data generation part which produces | generates a transport stream. It is a figure which shows the view selection state in the view selector in a transmission data generation part. It is a figure which shows an example of the parallax data (parallax vector) for every block (Block). It is a figure for demonstrating an example of the production | generation method of the parallax data of a block unit. It is a figure for demonstrating the method to produce | generate the parallax data of a pixel unit by the conversion process from a block unit to a pixel unit. It is a figure which shows the structural example of the multi view stream configuration descriptor as identification information. It is a figure which shows the content of the main information in the structural example of a multi view stream configuration descriptor. It is a figure which shows the structural example of the multi view stream configuration info as view structure information. It is a figure which shows the content of the main information in the structural example of multi view stream configuration info. It is a figure which shows the content of the main information in the structural example of multi view stream configuration info. It is a figure which shows the content of the main information in the structural example of multi view stream configuration info. It is a figure which shows an example of the relationship between the number of views which "view_count" shows, and the position of two views which "view_pair_position_id" shows. It is a figure for demonstrating the production | generation example of the parallax data in a transmission side or a receiving side in the case of transmitting the image data of two view pairs inside the both ends together with image data of two view pairs at both ends. It is a figure for demonstrating the example which interpolates and synthesize | combines the image data of the view located between each view on the receiving side based on parallax data. It is a figure for demonstrating that multiview stream configuration info is inserted in the part of "SELs" of an access unit as "Multiview stream configuration SEI message." It is a figure which shows the structural example of "Multiview stream configuration SEI message" and "userdata_for_multiview_stream_configuration ()". FIG. 4 is a diagram illustrating a structure example of “user_data ()”. It is a figure which shows the structural example in case the three video streams are included in the transport stream TS. It is a figure which shows the structural example in case two video streams are contained in the transport stream TS. It is a figure which shows the structural example in case one video stream is contained in transport stream TS. It is a block diagram which shows the structural example of the receiver which comprises an image transmission / reception system. It is a figure which shows the example of calculation of scaling ratio. It is a figure which shows roughly an example of the interpolation synthetic | combination process in a view interpolation part. It is a block diagram which shows the other structural example of the receiver which comprises an image transmission / reception system. It is a figure which shows the structural example of the image transmission / reception system which enables the naked-eye viewing of a three-dimensional image (stereoscopic image). It is a figure which shows roughly the display part of the receiving side when the number of views is set to 5 in the method of transmitting the image data of two views at both ends of N views. It is a figure which shows roughly the display part of the receiving side when the number of views is set to 5 in the method of transmitting the image data of two inner views among N views.

Hereinafter, modes for carrying out the invention (hereinafter referred to as “embodiments”) will be described. The description will be given in the following order.
1. Embodiment 2. FIG. Modified example

<1. Embodiment>
[Image transmission / reception system]
FIG. 1 shows a configuration example of an image transmission / reception system 10 as an embodiment. The image transmission / reception system 10 includes a broadcasting station 100 and a receiver 200. The broadcasting station 100 transmits a transport stream TS as a container on a broadcast wave.

  The transport stream TS includes a video stream obtained by encoding image data of at least the center view, the left end view, and the right end view among a plurality of views for stereoscopic image display. In this case, the central view constitutes an intermediate view located between the left end view and the right end view.

  In the video stream included in the transport stream TS, as shown in FIG. 2, the image data of the center view, the left end view, and the right end view are each encoded as one picture data. Is done. In the example shown in the figure, the data of each picture has a 1920 × 1080 full HD size.

  Alternatively, in the video stream included in the transport stream TS, as shown in FIG. 3A, the image data of the center (Center) view is encoded as data of one picture, and the left view (Left) view and the right end The image data of the (Right) view is interleaved and encoded as one picture data. In the example shown in the figure, the data of each picture has a 1920 × 1080 full HD size.

  When the image data of the left end view and the right end view are interleaved and encoded as one picture data, the image data of each view is thinned by half in the horizontal direction or the vertical direction. It becomes. In the illustrated example, the type of interleaving is side-by-side, and the size of each view is 960 * 1080. Although not shown, a top-and-bottom may be considered as an interleave type. In this case, the size of each view is 1920 * 540.

  In this way, when the image data of the left end view and the right end view are interleaved and encoded as one picture data, the receiving side performs scaling processing as shown in FIG. The size of the image data of the right view and the rightmost view is returned to the full HD size of 1920 * 1080.

  The video stream included in the transport stream TS includes data of one or a plurality of pictures. For example, the transport stream TS includes the following three video streams (video elementary streams). That is, it is a video stream obtained by encoding the image data of the center view, the left end view, and the right end view as one picture.

  For example, the transport stream TS includes the following two video streams (video elementary streams). That is, the video stream obtained by encoding the image data of the central view as one picture and the image data of the left end view and the right end view are interleaved and encoded as one picture. A video stream.

  Further, for example, the transport stream TS includes the following one video stream (video elementary stream). That is, this one video stream includes data obtained by encoding the image data of the center view, the left end view, and the right end view as data of one picture.

  4A and 4B show an example of a video stream including encoded data of a plurality of pictures. The encoded data of each picture is sequentially arranged in each access unit. In this case, the encoded data of the first picture is composed of “SPS to Coded Slice”, and the encoded data of the second and subsequent pictures is composed of “Subset SPS to Coded Slice”. This example is an example in which MPEG4-AVC encoding is performed, but other encoding schemes are also applicable. The hexadecimal numbers in the figure indicate “NAL unit type”.

  When the encoded data of each picture coexists in one video stream, it is required that the boundary of each picture can be identified instantaneously. However, an AUD (access unit delimiter) can be attached only to the head of one access unit. Therefore, as shown in FIG. 4B, it is conceivable to define and arrange a new “NAL unit” indicating a boundary “View Separation Marker” between encoded data of each picture. This makes it possible to instantly access the top data of each picture. FIG. 4A shows an example in which “View Separation Marker” is not arranged between data of two views.

  FIGS. 5A and 5B show an example in which encoded data of three pictures coexist in one video stream. Here, the encoded data of each picture is shown as a sub stream. FIG. 5A shows the top access unit of the GOP (Group Of Pictures), and FIG. 5B shows the access unit other than the top of the GOP.

  View configuration information relating to image data in the video stream is inserted into a layer (picture layer, sequence layer, etc.) of the video stream. This view configuration information indicates information indicating which view image data is included in the video stream, and indicates whether data of a plurality of pictures is encoded in one access unit of the video stream. Information etc. are included. This view configuration information is inserted into, for example, a user data area of a picture header or a sequence header of a video stream. With this view configuration information, the receiving side can perform appropriate and efficient processing for performing naked-eye viewing of a three-dimensional image (stereoscopic image) based on image data of a plurality of views. Details of this view configuration information will be described later.

  Also, identification information for identifying whether or not view configuration information is inserted into the layer of the video stream is inserted into the layer of the transport stream TS. This identification information is, for example, subordinate to a video elementary loop (Video ESloop) of a program map table (PMT) included in the transport stream TS, or an event information table (EIT: Event Information Table). ). With this identification information, the reception side can easily identify whether or not view configuration information is inserted in the layer of the video stream. Details of this identification information will be described later.

  The receiver 200 receives the transport stream TS transmitted from the broadcast station 100 on a broadcast wave. Further, the receiver 200 decodes the video stream included in the transport stream TS, and acquires, for example, image data of the center view, the left end view, and the right end view. At this time, the receiver 200 can know which view position the image data included in each video stream is based on the view configuration information included in the layer of the video stream.

  In addition, the receiver 200 may determine between the center view and the left end view, between the center view and the left end view, based on the disparity data between the center view and the left end view, and between the center view and the left end view. Image data of a predetermined number of views located between the right end views is acquired by interpolation processing. At this time, the receiver 200 can know the number of views based on the view configuration information included in the layer of the video stream, and can easily grasp which position view has not been transmitted.

  The receiver 200 decodes the parallax data stream sent together with the video stream from the broadcast station 100, and acquires the above-described parallax data. Alternatively, the receiver 200 generates the above-described parallax data based on the acquired image data of the center view, the left end view, and the right end view.

  The receiver 200 is a three-dimensional image (stereoscopic image) based on the image data of each view at the center, the left end, and the right end sent from the broadcast station 100 and the image data of each view acquired by the above-described interpolation processing. The images of each view are synthesized and displayed on the display unit for viewing with the naked eye.

  FIG. 6 schematically shows a display unit of the receiver 200 when the number of views is five. Here, “View_0” is the center view, “View_1” is the view one right from the center, “View_2” is the view one left from the center, “View_3” is two views right from the center, that is, the rightmost view, “View_4” indicates a view that is two left from the center, that is, the leftmost view. In this case, only the image data of the views “View_0”, “View_3”, and “View_4” are transmitted from the broadcast station 100, and the receiver 200 receives the image data of the views “View_0”, “View_3”, and “View_4”. The image data of the other views “View_1” and “View_2” are obtained by interpolation processing. Then, in the receiver 200, the images of these five views are synthesized and displayed on the display unit for viewing the three-dimensional image (stereoscopic image) with the naked eye. Although FIG. 6 shows a lenticular lens, a parallax barrier or the like may be used instead.

"Configuration example of transmission data generator"
FIG. 7 illustrates a configuration example of the transmission data generation unit 110 that generates the above-described transport stream TS in the broadcast station 100. The transmission data generation unit 110 includes N image data output units 111-1 to 111-N, a view selector 112, scalers 113-1, 113-2, and 113-3, and video encoders 114-1 and 114. -2, 114-3 and a multiplexer 115. The transmission data generation unit 110 includes a parallax data generation unit 116, a parallax encoder 117, a graphics data output unit 118, a graphics encoder 119, an audio data output unit 120, and an audio encoder 121.

  The image data output units 111-1 to 111-N output image data of N views (View 1... View N) for stereoscopic image display. The image data output unit includes, for example, a camera that images a subject and outputs image data, or an image data reading unit that reads and outputs image data from a storage medium. Note that the image data of the view that is not transmitted may not actually be present.

  In addition, the view selector 112 selects at least the image data of the left end view and the right end view and the intermediate view (between the left end and the right end) from the image data of N views (View 1... View N). One or more image data are selectively extracted. In this embodiment, the view selector 112 extracts the image data VL of the left end view and the image data VR of the right end view, and extracts the image data VC of the center view. FIG. 8 shows a view selection state in the view selector 112.

  Also, the scalers 113-1, 113-2, and 113-3 perform scaling processing on the image data VC, VL, and VR, respectively, and, for example, 1920 * 1080 full HD size image data VC ′, VL ′ and VR ′ are obtained. In this case, when the image data VC, VL, and VR are 1920 * 1080 full HD size, they are output as they are. If the image data VC, VL, VR is larger than the size of 1920 * 1080, the image data is scaled down and output.

  The video encoder 114-1 performs encoding such as MPEG4-AVC (MVC) or MPEG2 video on the image data VC ′ of the central view to obtain encoded video data. Then, the video encoder 114-1 generates a video stream including the encoded data as a sub stream (sub stream 1) by a stream formatter (not shown) provided in the subsequent stage.

  In addition, the video encoder 114-2 performs encoding such as MPEG4-AVC (MVC) or MPEG2 video on the image data VL ′ of the leftmost view to obtain encoded video data. Then, the video encoder 114-2 generates a video stream including the encoded data as a substream (substream 2) by a stream formatter (not shown) provided in the subsequent stage.

  Furthermore, the video encoder 114-3 performs encoding such as MPEG4-AVC (MVC) or MPEG2 video on the image data VR ′ of the rightmost view to obtain encoded video data. Then, the video encoder 114-3 generates a video stream including the encoded data as a substream (substream 3) by a stream formatter (not shown) provided in the subsequent stage.

  The video encoders 114-1, 114-2, and 114-3 insert the above-described view configuration information into the layer of the video stream. As described above, this view configuration information is information indicating which view image data is included in the video stream, and data of a plurality of pictures is encoded in one access unit of the video stream. It includes information indicating whether or not This view configuration information is inserted into, for example, a user data area of a picture header or a sequence header of a video stream.

  The disparity data generation unit 116 generates disparity data (disparity data) based on the image data of the center, left end, and right end views output from the view selector 112. The disparity data includes, for example, disparity data between the center view and the left end view and disparity data between the center view and the right end view. In this case, parallax data is generated in pixel units or block units. FIG. 9 illustrates an example of disparity data (disparity vector) for each block (Block).

  FIG. 10 shows an example of a method for generating disparity data in units of blocks. In this example, parallax data indicating the j-th view from the i-th view is obtained. In this case, pixel blocks (parallax detection blocks) such as 4 * 4, 8 * 8, or 16 * 16 are set in the picture of the i-th view.

  As shown in the figure, the i-th view picture is the detected image, the j-th view picture is the reference image, and the sum of absolute differences between pixels is minimum for each block of the i-th view picture. The block search of the picture of the j-th view is performed so that the disparity data is obtained.

That is, the parallax data DPn of the Nth block is obtained by block search so that the sum of absolute differences in the Nth block is minimized, for example, as shown in the following equation (1). In this equation (1), Dj represents the pixel value in the picture of the jth view, and Di represents the pixel value in the picture of the ith view.
DPn = min (Σabs (differ (Dj-Di))) (1)

  FIG. 11 shows an example of a method for generating disparity data in units of pixels. This example is a method of generating disparity data in pixel units by conversion processing from block units to pixel units. In FIG. 11A, “A”, “B”, “C”, “D”, and “X” indicate block areas, respectively.

  From the parallax data of these blocks, as shown in FIG. 11B, the parallax data of each region obtained by dividing the “X” block into four is obtained by the following equation (2). For example, the parallax data X (A, B) of the divided areas adjacent to “A” and “B” is the median value of the parallax data of the blocks “A”, “B”, and “X”. In other divided areas, parallax data is similarly obtained.

X (A, B) = median (X, A, B)
X (A, C) = median (X, A, C)
X (B, D) = median (X, B, D)
X (C, D) = median (X, C, D)
... (2)

  With the above-described one-time conversion, the area occupied by the parallax data is narrowed to ½ of the original vertical and horizontal size. By repeating this conversion a predetermined number of times depending on the block size, parallax data in pixel units is obtained. If the texture contains an edge and the complexity of the in-screen object is higher than the other parts, the block size should be reduced as appropriate and the disparity data itself in the initial block unit itself will be textured. It is also possible to improve.

  The disparity encoder 117 encodes the disparity data generated by the disparity data generation unit 116 to generate a disparity stream (disparity data elementary stream). The disparity stream includes disparity data in units of pixels or blocks. When the parallax data is in units of pixels, it can be compressed and transmitted in the same manner as the pixel data.

  Note that when the disparity stream includes disparity data in units of blocks, the reception side can perform conversion in units of pixels by performing the above-described conversion processing. Further, when there is no transmission of such a parallax stream, on the receiving side, it is possible to obtain parallax data in units of blocks between views as described above, and further convert them into units of pixels.

  The graphics data output unit 118 outputs data of graphics (including subtitles as subtitles) to be superimposed on the image. The graphics encoder 119 generates a graphics stream (graphics elementary stream) including the graphics data output from the graphics data output unit 118. Here, the graphics constitute superimposition information, and are, for example, a logo, subtitles, and the like.

  Note that the graphics data output from the graphics data output unit 118 is, for example, graphics data to be superimposed on the center view image. Based on the disparity data generated by the disparity data generation unit 116, the graphics data 119 may generate graphics data to be superimposed on the left end and right end views, and generate a graphics stream including these graphics data. . In this case, it is not necessary to create graphics data to be superimposed on the left end and right end views on the receiving side.

  The graphics data is mainly bitmap data. The graphics data is added with idling offset information indicating the superimposed position on the image. This idling offset information indicates, for example, offset values in the vertical and horizontal directions from the upper left origin of the image to the upper left pixel of the graphics superimposition position. Note that the standard for transmitting caption data as bitmap data is standardized and operated as “DVB_Subtitling” in DVB, which is a European digital broadcasting standard, for example.

  The audio data output unit 120 outputs audio data corresponding to the image data. The audio data output unit 120 includes, for example, a microphone or an audio data reading unit that reads out and outputs audio data from a storage medium. The audio encoder 121 performs encoding such as MPEG-2Audio or AAC on the audio data output from the audio data output unit 120 to generate an audio stream (audio elementary stream).

  The multiplexer 115 packetizes and multiplexes the elementary streams generated by the video encoders 114-1, 114-2, 114-3, the parallax encoder 117, the graphics encoder 119, and the audio encoder 121, and generates a transport stream TS. To do. In this case, a PTS (Presentation Time Stamp) is inserted into the header of each PES (Packetized Elementarty Stream) for synchronous playback on the receiving side.

  The multiplexer 115 inserts the identification information described above into the layer of the transport stream TS. This identification information is information for identifying whether or not view configuration information is inserted in the layer of the video stream. This identification information is, for example, subordinate to a video elementary loop (Video ESloop) of a program map table (PMT) included in the transport stream TS, or an event information table (EIT: Event Information Table). ).

  The operation of the transmission data generation unit 110 shown in FIG. 7 will be briefly described. Image data of N views (View 1... View N) for stereoscopic image display output from the N image data output units 111-1 to 111 -N is supplied to the view selector 112. The view selector 112 extracts the image data VC of the center view, the image data VL of the left end view, and the image data VR of the right end view from the image data of N views.

  The image data VC of the central view extracted by the view selector 112 is supplied to the scaler 113-1, and is scaled to a full HD size of, for example, 1920 * 1080. The image data VC ′ after the scaling process is supplied to the video encoder 114-1.

  In the video encoder 114-1, the image data VC ′ is encoded to obtain encoded video data, and a video stream including the encoded data as a substream (substream 1) is generated. Also, in this video encoder 114-1, a view configuration having information indicating which image data the image data included in the video stream is in, for example, the user data area of the picture header or sequence header of the video stream Information is inserted. This video stream is supplied to the multiplexer 115.

  Further, the image data VL of the leftmost view extracted by the view selector 112 is supplied to the scaler 113-2, and is scaled to, for example, a full HD size of 1920 * 1080. The image data VL ′ after the scaling process is supplied to the video encoder 114-2.

  In the video encoder 114-2, the image data VL ′ is encoded to obtain encoded video data, and a video stream including the encoded data as a substream (substream 2) is generated. Also, in this video encoder 114-2, a view configuration having information indicating which image data the image data included in the video stream is in the user data area of the picture header or sequence header of the video stream Information is inserted. This video stream is supplied to the multiplexer 115.

  Further, the image data VR of the right end view extracted by the view selector 112 is supplied to the scaler 113-3, and is scaled to, for example, a full HD size of 1920 * 1080. The image data VR ′ after the scaling processing is supplied to the video encoder 114-3.

  In the video encoder 114-3, the image data VR ′ is encoded to obtain encoded video data, and a video stream including the encoded data as a substream (substream 3) is generated. Also, in this video encoder 114-3, a view configuration having information indicating which image data the image data included in the video stream is in, for example, the user data area of the picture header or sequence header of the video stream Information is inserted. This video stream is supplied to the multiplexer 115.

  In addition, the image data of the center, left end, and right end views output from the view selector 112 is supplied to the parallax data generation unit 116. The disparity data generation unit 116 generates disparity data (disparity data) based on the image data of each view. The disparity data includes disparity data between the center view and the left end view, and disparity data between the center view and the right end view. In this case, parallax data is generated in pixel units or block units.

  The parallax data generated by the parallax data generation unit 116 is supplied to the parallax encoder 117. In the parallax encoder 117, the parallax data is encoded, and a parallax stream is generated. This parallax stream is supplied to the multiplexer 115.

  Further, graphics data (including subtitle data) output from the graphics data output unit 118 is supplied to the graphics encoder 119. The graphics encoder 119 generates a graphics stream including graphics data. This graphics stream is supplied to the multiplexer 115.

  The audio data output from the audio data output unit 118 is supplied to the audio encoder 121. The audio encoder 121 performs encoding such as MPEG-2Audio or AAC on the audio data to generate an audio stream. This audio stream is supplied to the multiplexer 115.

  In the multiplexer 115, the elementary streams supplied from each encoder are packetized and multiplexed to generate a transport stream TS. In this case, a PTS is inserted into each PES header for synchronous reproduction on the receiving side. Also, in the multiplexer 115, identification information for identifying whether or not view configuration information is inserted in the layer of the video stream is inserted under the PMT or the EIT.

  Note that the transmission data generation unit 110 illustrated in FIG. 7 illustrates a case where three video streams are included in the transport stream TS. That is, the transport stream TS includes three video streams obtained by encoding the image data of each view at the center, the left end, and the right end as one picture.

  Although detailed description is omitted, as described above, the same configuration can be made when two or one video stream is included in the transport stream TS. In the case where two video streams are included in the transport stream TS, for example, the following video streams are included. That is, the video stream obtained by encoding the image data of the central view as one picture and the image data of the left end view and the right end view are interleaved and encoded as one picture. Video stream is included.

  Further, when one video stream is included in the transport stream TS, for example, the following video streams are included. That is, a video stream including data in which image data of each view at the center, the left end, and the right end is encoded as one picture data is included.

[Structure of identification information and view configuration information and TS configuration]
As described above, identification information for identifying whether or not view configuration information is inserted into the layer of the video stream is inserted into the layer of the transport stream TS. FIG. 12 shows a structural example (Syntax) of the multiview stream configuration descriptor (multiview_stream_configuration_descriptor) as the identification information. FIG. 13 shows the contents (Semantics) of main information in the structural example shown in FIG.

  “Multiview_stream_configuration_tag” is 8-bit data indicating a descriptor type, and here indicates that it is a multi-view stream configuration descriptor. “Multiview_stream_configuration_length” is 8-bit data indicating the length (size) of the descriptor. This data indicates the number of subsequent bytes as the length of the descriptor.

  A 1-bit field of “multiview_stream_checkflag” indicates whether or not view configuration information is inserted in the layer of the video stream. “1” indicates that there is insertion of view configuration information in the layer of the video stream, and “0” indicates that there is no insertion. When it is “1”, the receiving side (decoder) checks the view configuration information existing in the user data area.

  Further, as described above, view configuration information having information indicating which view image data is included in the video stream is inserted into the layer of the video stream. FIG. 14 shows a structural example (Syntax) of multi-view stream configuration information (multiview_stream_configuration_info ()) as the view configuration information. 15, FIG. 16, and FIG. 17 show the contents (Semantics) of main information in the structural example shown in FIG.

  The 1-bit field of “3D_flag” indicates whether or not the image data included in the encoded video stream is image data of a part of views constituting 3D. “1” indicates that the image data is part of the view, and “0” indicates that the image data is not part of the image data.

  When “3D_flag = 1”, each information of “view_count”, “single_view_es_flag”, and “view_interleaving_flag” exists. A 4-bit field of “view_count” indicates the number of views constituting the 3D service. The minimum value is 1 and the maximum value is 15. A 1-bit field of “single_view_es_flag” indicates whether or not data of a plurality of pictures is encoded in one access unit of the video stream. “1” indicates that only data of one picture is encoded, and “0” indicates that data of two or more pictures is encoded.

  The 1-bit field of “view_interleaving_flag” indicates whether or not image data of two views is interleaved and encoded as data of one picture in the video stream. “1” indicates that the interleave process is performed and the screen is split, and “0” indicates that the interleave process is not performed.

  When “view_interleaving_flag = 0”, information of “view_allocation” exists. The 4-bit field of “view_allocation” indicates which view image data the image data included in the video stream is, that is, view allocation. For example, “0000” indicates a center view. Further, for example, “0001” indicates that the view is one view left next to the center (1st left view next to center). Further, for example, “0010” indicates that the view is one view right next to the center (1st right view next to center).

  When “view_interleaving_flag = 1”, information of “view_pair_position_id” and “view_interleaving_type” exists. A 3-bit field of “view_pair_position_id” indicates a relative view position of two views in all views. In this case, for example, an early position in the scan order is left (left), and a late position is right (right). For example, “000” indicates that there are two view pairs at both ends. Further, for example, “001” indicates that two view pairs are located one inside from both ends. Further, for example, “010” indicates that two view pairs are located one inside from both ends.

  A 1-bit field of “view_interleaving_type” indicates an interleaving type (type). “1” indicates that the interleaving type is Side-by-Side, and “0” indicates that the interleaving type is Top & Bottom.

  Further, when “3D_flag = 1”, there is information of “display_flag”, “indication_of_picture_size_scaling_horizontal”, and “indication_of_picture_size_scaling_vertical”. The 1-bit field of “display_flag” indicates whether or not the view is indispensable when image display is performed. “1” indicates that display is mandatory. On the other hand, “0” indicates that display is not essential.

  A 4-bit field of “indication_of_picture_size_scaling_horizontal” indicates the horizontal pixel ratio of the decoded image with respect to full HD (1920). “0000” is 100%, “0001” is 80%, “0010” is 75%, “0011” is 66%, “0100” is 50%, “0101” is 33%, “0110” is 25%, “ “0111” indicates 20%.

  A 4-bit field of “indication_of_picture_size_scaling_vertical” indicates a vertical pixel ratio of a decoded image with respect to full HD (1080). “0000” is 100%, “0001” is 80%, “0010” is 75%, “0011” is 66%, “0100” is 50%, “0101” is 33%, “0110” is 25%, “0111” "" Indicates 20%.

  FIG. 18 illustrates an example of the relationship between the number of views indicated by “view_count” and the positions of two views indicated by “view_pair_position_id” (here, “View 1” and “View 2”). The example of (1) is a case where the number of views indicated by “view_count” is 2, and “view_pair_position_id = 000”, indicating that the two views are at both ends. The example (2) is a case where the number of views indicated by “view_count” is 4, and “view_pair_position_id = 000” indicates that the two views are at both ends.

  The example (3) is a case in which the number of views indicated by “view_count” is 4, and “view_pair_position_id = 001” indicates that the two views are one inside from both ends. The example (4) is a case where the number of views indicated by “view_count” is 5, and “view_pair_position_id = 000”, indicating that the two views are at both ends.

  The example (5) is a case where the number of views indicated by “view_count” is 9, and “view_pair_position_id = 000”, indicating that the two views are at both ends. Furthermore, the example of (6) is a case where the number of views indicated by “view_count” is 9, and “view_pair_position_id = 010”, indicating that the two views are two inward from both ends.

  In order to improve the performance of interpolation synthesis, view pairs inside the both ends can be added to the view pairs at both ends in order to improve the performance of interpolation synthesis when the image quality cannot be satisfied with the two views at both ends. Can be transmitted. At this time, the encoded video data of the view pair additionally transmitted may be encoded so as to share the access unit (AccessUnit) in the stream of the view pair at both ends, or may be encoded as another stream. May be used.

  FIG. 19 shows an example of generation of disparity data (disparity data) on the transmission side or the reception side when transmitting image data of two view pairs inside the both ends together with image data of the two view pairs at both ends as described above. Is shown. In the illustrated example, the number of views indicated by “view_count” is nine. Then, a substream (substream1) containing image data of two views (View 1, View 2) at both ends, and a substream containing image data of two views (View 3, View 4) inside it (Substream 2) exists.

  In this case, first, parallax data is calculated for “View 1” and “View 3”. Next, parallax data is calculated for “View 2” and “View 4”. Finally, parallax data is calculated for “View 3” and “View 4”. If the view resolution differs between substreams, the parallax data is calculated after matching either resolution.

  FIG. 20 shows an example in which image data of views located between the views is interpolated and synthesized on the receiving side based on the parallax data calculated as described above. In this case, first, “View_A” positioned between “View 1” and “View 3” is interpolated and synthesized using disparity data between “View 1” and “View 3”.

  Next, using the parallax data between “View 2” and “View 4”, “View_B” located between “View 2” and “View 4” is interpolated and synthesized. Finally, using the parallax data between “View 3” and “View 4”, “View_C”, “View_D”, and “View_E” located between “View 3” and “View 4” are interpolated. To do.

  Next, a case where multi-view stream configuration information (multiview_stream_configuration_info ()) as view configuration information is inserted into a user data area of a video stream (video elementary stream) will be described. In this case, the multi-view stream configuration information is inserted, for example, in picture units or GOP units using the user data area.

  For example, when the encoding method is AVC, the multi-view stream configuration information is inserted as “Multiview stream configuration SEI message” in the “SELs” portion of the access unit. FIG. 21A shows the top access unit of the GOP (Group Of Pictures), and FIG. 21B shows the access unit other than the top of the GOP. When multi-view stream configuration information is inserted in units of GOPs, a “Multiview stream configuration SEI message” is inserted only in the first access unit of the GOP.

  FIG. 22A shows a structural example (Syntax) of “Multiview stream configuration SEI message”. “Uuid_iso_iec_11578” has a UUID value indicated by “ISO / IEC 11578: 1996 Annex A.”. “Userdata_for_multiview_stream_configuration ()” is inserted into the “user_data_payload_byte” field. FIG. 22B shows a structural example (Syntax) of “userdata_for_multiview_stream_configuration ()”. In this, multiview stream configuration information (multiview_stream_configuration_info ()) is inserted (see FIG. 14). “Userdata_id” is an identifier of multi-view stream configuration information indicated by 16 bits without a sign.

  For example, when the encoding method is MPEG2 video, the multi-view stream configuration information is inserted as user data “user_data ()” in the user data area of the picture header portion. FIG. 23A shows a structural example (Syntax) of “user_data ()”. A 32-bit field of “user_data_start_code” is a start code of user data (user_data), and is a fixed value of “0x000001B2”.

  A 32-bit field following the start code is an identifier for identifying the contents of user data. Here, “Stereo_Video_Format_Signaling_identifier” is set, and it is possible to identify that the user data is multi-view stream configuration information. As the data body after this identifier, “Multiview_stream_configuration ()” as stream association information is inserted. FIG. 23B shows a structural example (Syntax) of “Multiview_stream_configuration ()”. In this, multiview stream configuration information (multiview_stream_configuration_info ()) is inserted (see FIG. 14).

  The multi-view stream configuration descriptor (multiview_stream_configuration_descriptor) as the identification information shown in FIG. 12 is inserted in the transport stream TS layer, for example, under the PMT or under the EIT. In other words, this descriptor is placed at an optimum position in an event unit or in a static or dynamic use case in time.

  FIG. 24 illustrates a configuration example of the transport stream TS. In this configuration example, illustration of parallax data, audio, graphics, and the like is omitted for simplification of the drawing. This configuration example shows a case where three video streams are included in the transport stream TS. That is, the transport stream TS includes three video streams obtained by encoding the image data of each view at the center, the left end, and the right end as one picture. Further, this configuration example shows a case where the number of views is five.

  The configuration example of FIG. 24 includes a PES packet “video PES1” of a video stream in which the image data VC ′ of the central view is encoded as one picture. In the multi-view stream configuration information inserted in the user data area of this video stream, it is indicated that the number of views indicated by “View_count” is five.

  In this information, “single_view_es_flag = 1” is set, and it is indicated that only data of one picture in one access unit is encoded in this video stream. Also, in this info, “View_interleaving_flag = 0” is set, and it is indicated that image data of two views is not interleaved and encoded as data of one picture in this video stream. Furthermore, “view_allocation = 0000” is set, indicating that the image data included in this video stream is the image data of the central view.

  In addition, the configuration example of FIG. 24 includes a PES packet “video PES2” of a video stream in which the image data VL ′ of the left end view is encoded as one picture. In the multi-view stream configuration information inserted in the user data area of this video stream, it is indicated that the number of views indicated by “View_count” is five.

  In this information, “single_view_es_flag = 1” is set, and it is indicated that only data of one picture in one access unit is encoded in this video stream. Also, in this info, “View_interleaving_flag = 0” is set, and it is indicated that image data of two views is not interleaved and encoded as data of one picture in this video stream. Furthermore, “view_allocation = 0011” is set, indicating that the image data included in this video stream is the image data of the two adjacent views from the center to the left side, that is, the left end view.

  In addition, the configuration example of FIG. 24 includes a PES packet “video PES3” of a video stream in which the image data VR ′ of the left end view is encoded as one picture. In the multi-view stream configuration information inserted in the user data area of this video stream, it is indicated that the number of views indicated by “View_count” is five.

  In this information, “single_view_es_flag = 1” is set, and it is indicated that only data of one picture in one access unit is encoded in this video stream. Also, in this info, “View_interleaving_flag = 0” is set, and it is indicated that image data of two views is not interleaved and encoded as data of one picture in this video stream. Further, “view_allocation = 0100” is set, indicating that the image data included in this video stream is the image data of two adjacent views from the center to the right side, that is, the right end view.

  The transport stream TS includes a PMT (Program Map Table) as PSI (Program Specific Information). This PSI is information describing to which program each elementary stream included in the transport stream belongs. Further, the transport stream includes an EIT (Event Information Table) as SI (Serviced Information) for managing each event.

  In the PMT, there is an elementary loop having information related to each elementary stream. In this configuration example, a video elementary loop (Video ES loop) exists. In this elementary loop, information such as a packet identifier (PID) is arranged for each stream, and a descriptor describing information related to the elementary stream is also arranged.

  In this configuration example, a multiview stream configuration descriptor (multiview_stream_configuration_descriptor) is inserted in association with each video stream under a PMT video elementary loop (Video ES loop). In this descriptor, “multiview_stream_checkflag = 1” is set, which indicates the presence of multi-view stream configuration information as view configuration information in the user area of the video stream. It is also conceivable to insert this descriptor under the EIT as shown by the broken line.

  FIG. 25 also shows a configuration example of the transport stream TS. Also in this configuration example, illustration of parallax data, audio, graphics, and the like is omitted to simplify the drawing. This configuration example shows a case where two video streams are included in the transport stream TS. That is, the transport stream TS includes a video stream obtained by encoding the image data of the central view as one picture. In addition, the transport stream TS includes a video stream obtained by interleaving the image data of the left end view and the right end view and encoding it as one picture. This configuration example also shows a case where the number of views is five.

  In the configuration example of FIG. 25, the PES packet “video PES1” of the video stream in which the image data VC ′ of the central view is encoded as one picture is included. In the multi-view stream configuration information inserted in the user data area of this video stream, it is indicated that the number of views indicated by “View_count” is five.

  In this information, “single_view_es_flag = 1” is set, and it is indicated that only data of one picture in one access unit is encoded in this video stream. In this info, “View_interleaving_flag = 0” is set, indicating that the image data of two views is not interleaved and encoded as one picture data in this video stream. Yes. Furthermore, “view_allocation = 0000” is set, indicating that the image data included in this video stream is the image data of the central view.

  25 includes a PES packet “video PES2” of a video stream in which the left end view image data VL ′ and the right end view image data VR ′ are encoded as one picture. In the multi-view stream configuration information inserted in the user data area of this video stream, it is indicated that the number of views indicated by “View_count” is five.

  In this information, “single_view_es_flag = 1” is set, and it is indicated that only data of one picture in one access unit is encoded in this video stream. In this information, “View_interleaving_flag = 1” is set, and it is indicated that image data of two views is interleaved and encoded as data of one picture in this video stream. Further, “view_pair_position_id = 000” is set, indicating that the two view pairs are at both ends. Furthermore, “view_interleaving_type = 1” is set, indicating that the type of interleaving is side-by-side.

  In this configuration example, a multiview stream configuration descriptor (multiview_stream_configuration_descriptor) is inserted under the PMT video elementary loop (Video ES loop) in association with each video stream. In this descriptor, “multiview_stream_checkflag = 1” is set, which indicates the presence of multi-view stream configuration information as view configuration information in the user area of the video stream. It is also conceivable to insert this descriptor under the EIT as shown by the broken line.

  FIG. 26 also illustrates a configuration example of the transport stream TS. Also in this configuration example, illustration of parallax data, audio, graphics, and the like is omitted to simplify the drawing. This configuration example shows a case where one video stream is included in the transport stream TS. That is, the transport stream TS includes a video stream including data obtained by encoding image data of each view at the center, the left end, and the right end as data of one picture. This configuration example also shows a case where the number of views is five.

  In the configuration example of FIG. 26, a PES packet “video PES1” of one video stream is included. This video stream includes data in which image data of each view at the center, the left end, and the right end is encoded as data of one picture in one access unit, and a user data area corresponding to each picture Exists. In each case, multi-view stream configuration information is inserted.

  In the information corresponding to the picture data in which the image data of the central view is encoded, it is indicated that the number of views indicated by “View_count” is five. In this information, “single_view_es_flag = 0” is set, and it is indicated that data of a plurality of pictures is encoded in one access unit in this video stream. In this info, “View_interleaving_flag = 0” is set, indicating that this picture data is not encoded by interleaving the image data of two views. Furthermore, “view_allocation = 0000” is set, indicating that the image data included in the picture data is the image data of the central view.

  Further, in the information corresponding to the picture data in which the image data of the left end view is encoded, it is indicated that the number of views indicated by “View_count” is five. In this information, “single_view_es_flag = 0” is set, and it is indicated that data of a plurality of pictures is encoded in one access unit in this video stream. In this info, “View_interleaving_flag = 0” is set, indicating that this picture data is not encoded by interleaving the image data of two views. Further, “view_allocation = 0011” is set, indicating that the image data included in the picture data is the image data of the two adjacent views from the center to the left side, that is, the image data of the left end view.

  Further, in the information corresponding to the picture data in which the image data of the right end view is encoded, it is indicated that the number of views indicated by “View_count” is five. In this information, “single_view_es_flag = 0” is set, and it is indicated that data of a plurality of pictures is encoded in one access unit in this video stream. In this info, “View_interleaving_flag = 0” is set, indicating that this picture data is not encoded by interleaving the image data of two views. Further, “view_allocation = 0100” is set, indicating that the image data included in the picture data is the image data of the two adjacent views from the center to the right side, that is, the right end view.

  Also, in this configuration example, a multiview stream configuration descriptor (multiview_stream_configuration_descriptor) is inserted in association with one video stream under the video elementary loop (Video ES loop) of the PMT. In this descriptor, “multiview_stream_checkflag = 1” is set, which indicates the presence of multi-view stream configuration information as view configuration information in the user area of the video stream. It is also conceivable to insert this descriptor under the EIT as shown by the broken line.

  As described above, the transmission data generation unit 110 illustrated in FIG. 7 is located between the left end and the right end of at least the left end view and the right end view among the plurality of views for stereoscopic image display. A transport stream TS including a video stream obtained by encoding image data of an intermediate view is generated. Therefore, image data transmission for performing naked-eye viewing of a stereoscopic image with a multi-view configuration can be effectively performed.

  That is, not only the image data of the left end view and the right end view but also the image data of the intermediate view is transmitted, so the relative parallax between the views is small, and the fine parts when interpolating the image data of other views Interpolation around the occlusion associated with this processing becomes easy, and the quality of the reproduced image can be improved. In addition, since the image data of the left end view and the right end view is transmitted, all of the image data of the non-transmitted view can be synthesized by interpolation processing, and it is easy to maintain high image quality with respect to end point processing such as occlusion. Become.

  In addition, in the transmission data generation unit 110 illustrated in FIG. 7, multiview stream configuration information (multiview_stream_configuration_info ()) as the view configuration information is inserted in the video stream layer. Therefore, on the receiving side, appropriate and efficient processing for performing naked-eye viewing of a three-dimensional image (stereoscopic image) using image data of a plurality of views can be performed based on the view configuration information.

  In addition, in the transmission data generation unit 110 illustrated in FIG. 7, a multiview stream configuration descriptor (multiview_stream_configuration_descriptor) is inserted in the layer of the transport stream TS. This descriptor constitutes identification information for identifying whether or not view configuration information is inserted in the layer of the video stream. With this identification information, the reception side can easily identify whether or not view configuration information is inserted in the layer of the video stream. Therefore, it is possible to efficiently extract view configuration information from the user data area of the video stream.

  In addition, in the transmission data generation unit 110 illustrated in FIG. 7, the parallax data between the views is generated by the parallax data generation unit 116, and the parallax stream obtained by encoding the parallax data is converted together with the video stream. It is included in the port stream TS. Therefore, on the receiving side, image data of each view that is not transmitted can be easily interpolated based on the transmitted parallax data without performing processing to generate parallax data from the received image data of each view. Is possible.

"Example of receiver configuration"
FIG. 27 illustrates a configuration example of the receiver 200. The receiver 200 includes a CPU 201, a flash ROM 202, a DRAM 203, an internal bus 204, a remote control receiver (RC receiver) 205, and a remote control transmitter (RC transmitter) 206. The receiver 200 also includes an antenna terminal 211, a digital tuner 212, a transport stream buffer (TS buffer) 213, and a demultiplexer 214.

  The receiver 200 includes coded buffers 215-1, 215-2, and 215-3, video decoders 216-1, 216-2, and 216-3, and decoded buffers 217-1, 217-2, and 217-. 3 and scalers 218-1, 218-2, and 218-3. The receiver 200 also includes a view interpolation unit 219 and a pixel interleave / superimposition unit 220. In addition, the receiver 200 includes a coded buffer 221, a parallax decoder 222, a parallax buffer 223, and a parallax data conversion unit 224.

  The receiver 200 also includes a coded buffer 225, a graphics decoder 226, a pixel buffer 227, a scaler 228, and a graphics shifter 229. Further, the receiver 200 includes a coded buffer 230, an audio decoder 231, and a channel mixing unit 232.

  The CPU 201 controls the operation of each unit of the receiver 200. The flash ROM 202 stores control software and data. The DRAM 203 constitutes a work area for the CPU 201. The CPU 201 develops software and data read from the flash ROM 202 on the DRAM 203 and activates the software to control each unit of the receiver 200. The RC receiving unit 205 receives a remote control signal (remote control code) transmitted from the RC transmitter 206 and supplies it to the CPU 201. CPU201 controls each part of receiver 200 based on this remote control code. The CPU 201, flash ROM 202, and DRAM 203 are connected to the internal bus 204.

  The antenna terminal 211 is a terminal for inputting a television broadcast signal received by a receiving antenna (not shown). The digital tuner 212 processes the television broadcast signal input to the antenna terminal 211 and outputs a predetermined transport stream (bit stream data) TS corresponding to the user's selected channel. The transport stream buffer (TS buffer) 213 temporarily accumulates the transport stream TS output from the digital tuner 212.

  The transport stream TS includes at least image data of a left end view and a right end view among a plurality of views for stereoscopic image display, and a central view as an intermediate view positioned between the left end and the right end. A video stream obtained by encoding image data is included.

  In this case, the transport stream TS may include three, two, or one video stream (see FIGS. 24, 25, and 26). Here, for the sake of simplicity, it is assumed that the transport stream TS includes three video streams obtained by encoding the image data of the center, left end, and right end views as one picture. An explanation shall be given.

  As described above, a multiview stream configuration descriptor (multiview_stream_configuration_descriptor) is inserted into the transport stream TS under the PMT or the EIT. This descriptor is identification information for identifying whether or not view configuration information, that is, multiview stream configuration information (multiview_stream_configuration_info ()) is inserted in the layer of the video stream.

  The demultiplexer 214 extracts elementary streams of video, parallax, graphics, and audio from the transport stream TS temporarily stored in the TS buffer 213. Further, the demultiplexer 214 extracts the multi-view stream configuration descriptor described above from the transport stream TS, and sends it to the CPU 201. The CPU 201 can easily determine whether or not view configuration information is inserted in the layer of the video stream from the 1-bit field of “multiview_stream_checkflag” of the descriptor.

  The coded buffers 215-1, 215-2, and 215-3 are video streams obtained by encoding the image data of the center, left end, and right end views extracted by the demultiplexer 214 as one picture, respectively. Is temporarily stored. Video decoders 216-1, 216-2, and 216-3 perform decoding processing of video streams stored in coded buffers 215-1, 215-2, and 215-3, respectively, under the control of CPU 201. Then, the image data of each view at the center, the left end, and the right end is acquired.

  Here, the video decoder 216-1 acquires image data of a center view. In addition, the video decoder 216-2 acquires image data of the left end view (left view). Furthermore, the video decoder 216-3 acquires image data of the right end view (right view). When two or more views are interleaved and encoded, a coded buffer, a video decoder, a decoded buffer, and a scaler are allocated for each stream.

  Each video decoder extracts multi-view stream configuration information (multiview_stream_configuration_info ()) as view configuration information inserted in the user data area of the picture header or sequence header of the video stream, and sends it to the CPU 201. The CPU 201 performs appropriate and efficient processing for performing naked-eye viewing of a three-dimensional image (stereoscopic image) using image data of a plurality of views based on the view configuration information.

  That is, based on this view configuration information, the CPU 201 demultiplexer 214, video decoders 216-1, 216-2, 216-3, scalers 218-1, 218-2, 218-3 in units of pictures or GOPs. The operation of the view interpolation unit 219 and the like is controlled. For example, the CPU 201 can identify whether or not the image data included in the video stream is the image data of a part of the views constituting 3D by using the 1-bit field “3D_flag”. Further, for example, the CPU 201 can recognize the number of views constituting the 3D service by a 4-bit field of “view_count”.

  Further, for example, the CPU 201 can identify whether or not data of a plurality of pictures is encoded in one access unit of the video stream by using a 1-bit field of “single_view_es_flag”. Further, for example, the CPU 201 can identify whether or not the image data of two views is interleaved and encoded as data of one picture in the video stream by the 1-bit field of “view_interleaving_flag”.

  For example, when image data of two views is not interleaved and encoded as data of one picture in the video stream, the CPU 201 uses the 4-bit field of “view_allocation” to display the image included in the video stream. It is possible to recognize which view image data the data is.

  Further, for example, when image data of two views is interleaved and encoded as data of one picture in the video stream, the CPU 201 uses the 3-bit field of “view_pair_position_id” to display two views in all views. The relative view position can be recognized. Further, at this time, the CPU 201 can know the type of interleaving (type) from the 1-bit field of “view_interleaving_type”.

  Further, for example, the CPU 201 can recognize the horizontal pixel ratio of the decoded image with respect to the full HD by the 4-bit field “indication_of_picture_size_scaling_horizontal” and the 4-bit field “indication_of_picture_size_scaling_vertical”.

  The decoded buffers 217-1, 217-2, and 217-3 temporarily store the image data of each view acquired by the video decoders 216-1, 216-2, and 216-3, respectively. Scalers 218-1, 218-2, and 218-3 are configured so that the output resolution of the image data of each view output from decoded buffers 217-1, 217-2, and 217-3 is a predetermined resolution, respectively. Adjust to.

  The multi-view stream configuration information includes a 4-bit field “indication_of_picture_size_scaling_horizontal” indicating the horizontal pixel ratio of the decoded image and a 4-bit field “indication_of_picture_size_scaling_vertical” indicating the vertical pixel ratio of the decoded image. Based on this pixel ratio information, the CPU 201 controls the scaling ratios in the scalers 218-1, 218-2, and 218-3 so that a predetermined resolution can be obtained.

  In this case, the CPU 201 calculates a scaling ratio for the image data stored in the decoded buffer based on the resolution of the decoded image data, the resolution of the monitor, and the number of views, and scalers 218-1 and 218. Instruct -2 and 218-3. FIG. 28 shows an example of calculating the scaling ratio.

  For example, when the resolution of the decoded image data is 960 * 1080, the monitor resolution is 1920 * 1080, and the number of views to be displayed is 4, the scaling ratio is ½. Further, for example, when the resolution of the decoded image data is 1920 * 1080, the monitor resolution is 1920 * 1080, and the number of views to be displayed is 4, the scaling ratio is 1/4. Further, for example, when the resolution of the decoded image data is 1920 * 2160, the monitor resolution is 3840 * 2160, and the number of views to be displayed is 8, the scaling ratio is 1/4.

  The coded buffer 221 temporarily accumulates the parallax stream extracted by the demultiplexer 214. The disparity decoder 222 performs processing opposite to that of the disparity encoder 117 (see FIG. 7) of the transmission data generation unit 110 described above. That is, the parallax decoder 223 performs a decoding process on the parallax stream stored in the coded buffer 221 to obtain parallax data. The disparity data includes disparity data between the center view and the left end view and disparity data between the center view and the right end view. The parallax data is parallax data in units of pixels or blocks. The parallax buffer 223 temporarily stores the parallax data acquired by the parallax decoder 222.

  Based on the parallax data stored in the parallax buffer 223, the parallax data conversion unit 224 generates parallax data in pixel units that matches the size of the scaled image data. For example, when the transmitted parallax data is in units of blocks, it is converted into parallax data in units of pixels (see FIG. 11). Also, for example, when the transmitted parallax data is in units of pixels, but does not match the size of the scaled image data, the data is scaled appropriately.

  The view interpolation unit 219 generates a predetermined number of views that have not been transmitted based on the parallax data between the views obtained by the parallax data conversion unit 224 from the image data of the center, left end, and right end views after scaling. Interpolate image data. That is, the view interpolation unit 219 interpolates and outputs the image data of each view positioned between the center view and the left end view. The view interpolation unit 219 interpolates and outputs image data of each view located between the center view and the right end view.

  FIG. 29 schematically illustrates an example of the interpolation synthesis process in the view interpolation unit 219. In the illustrated example, for example, the current view (Current view) corresponds to the above-described center view, the target view 1 (Targetview 1) corresponds to the above-mentioned left end view, and the target view 2 (Target view 2) corresponds to the above-described right end. Corresponds to the view.

  The interpolation synthesis of the view located between the current view and the target view 1 and the interpolation synthesis of the view located between the current view and the target view 2 are performed in the same manner. In the following, the interpolation synthesis of the view located between the current view and the target view 1 will be described.

  The pixels of the view to be interpolated and located between the current view and the target view 1 are assigned as follows. In this case, disparity data in two directions, that is, disparity data indicating the target view 1 from the current view and conversely, disparity data indicating the current view from the target view 1 is used. First, as the view pixel to be interpolated and synthesized, the current view pixel is allocated by shifting the parallax data as a vector (see solid line arrows and broken line arrows from the current view to the target view 1 and black circles).

  At this time, the following pixel allocation is performed in the target occluded portion in the target view 1. That is, the pixels of the target view 1 are allocated by shifting the parallax data as a vector as the view pixels to be interpolated and synthesized (see the dashed-dotted arrow pointing from the target view 1 to the current view and the white circle).

  In this way, in the part that becomes the target occluded, by having bi-directional parallax data, the pixel of the view to be interpolated and synthesized can be applied with the pixel from the view that can be regarded as the background. It should be noted that an occlusion area that cannot be handled bidirectionally is assigned a value by a post process.

  In addition, a portion that becomes a target overlapped where the tips of the arrows shown in the figure overlap is a portion in the target view 1 where shifts due to disparity overlap. In this part, which of the two parallaxes corresponds to the foreground of the current view is determined based on the value of the parallax data and selected. In this case, the smaller value is mainly selected.

  Returning to FIG. 27, the coded buffer 225 temporarily stores the graphics stream extracted by the demultiplexer 214. The graphics decoder 226 performs processing opposite to that of the graphics encoder 119 (see FIG. 7) of the transmission data generation unit 110 described above. That is, the graphics decoder 226 performs a decoding process on the graphics stream stored in the coded buffer 225 to obtain decoded graphics data (including subtitle data). The graphics decoder 226 generates graphics bitmap data to be superimposed on the view (image) based on the graphics data.

  The pixel buffer 227 temporarily stores graphics bitmap data generated by the graphics decoder 226. The scaler 228 adjusts the size of the graphics bitmap data stored in the pixel buffer 227 so as to correspond to the size of the scaled image data. The graphics shifter 229 performs shift processing on the bitmap data of the size-adjusted graphics based on the parallax data obtained by the parallax data conversion unit 224. The graphics shifter 229 generates N graphics bitmap data to be superimposed on the image data of N views (View1, View2,..., ViewN) output from the view interpolation unit 219.

  The pixel interleaving / superimposing unit 220 superimposes graphics bitmap data corresponding to the image data of N views (View1, View2,..., ViewN) output from the view interpolation unit 219, respectively. Further, the pixel interleaving / superimposing unit 220 performs pixel interleaving processing on the image data of N views (View1, View2,..., ViewN) to view the three-dimensional image (stereoscopic image) with the naked eye. Display image data is generated.

  The coded buffer 230 temporarily stores the audio stream extracted by the demultiplexer 214. The audio decoder 231 performs a process reverse to that of the audio encoder 121 (see FIG. 7) of the transmission data generation unit 110 described above. That is, the audio decoder 231 performs a decoding process on the audio stream stored in the coded buffer 230 to obtain decoded audio data. The channel mixing unit 232 generates and outputs audio data of each channel for realizing, for example, 5.1ch surround with respect to the audio data obtained by the audio decoder 231.

  Note that image data of each view is read from the decoded buffers 217-1, 217-2, and 217-2, parallax data is read from the parallax buffer 223, and graphics bitmap data is read from the pixel buffer 227. Is performed based on PTS, and transfer synchronization is taken.

  The operation of the receiver 200 will be briefly described. A television broadcast signal input to the antenna terminal 211 is supplied to the digital tuner 212. In this digital tuner 212, the television broadcast signal is processed, and a predetermined transport stream TS corresponding to the user's selected channel is output. This transport stream TS is temporarily stored in the TS buffer 213.

  This transport stream TS includes image data of a left end view and a right end view among a plurality of views for displaying a stereoscopic image, and an image of a central view as an intermediate view positioned between the left end and the right end. A video stream obtained by encoding data is included.

  In the demultiplexer 214, elementary streams of video, parallax, graphics, and audio are extracted from the transport stream TS temporarily stored in the TS buffer 213. Further, the demultiplexer 214 extracts a multi-view stream configuration descriptor as identification information from the transport stream TS and sends it to the CPU 201. The CPU 201 can easily determine whether or not view configuration information is inserted in the layer of the video stream from the 1-bit field of “multiview_stream_checkflag” of the descriptor.

  The image data of each view at the center, the left end, and the right end extracted by the demultiplexer 214 is supplied to the coded buffers 215-1, 215-2, and 215-3, respectively, and temporarily accumulated. The video decoders 216-1, 216-2, and 216-3 decode the video streams stored in the coded buffers 215-1, 215-2, and 215-3, respectively, under the control of the CPU 201. Is performed, and image data of each view at the center, the left end, and the right end is acquired.

  Each video decoder extracts multi-view stream configuration information (multiview_stream_configuration_info ()) as view configuration information inserted in the user data area of the picture header or sequence header of the video stream and sends it to the CPU 201. Sent. Based on this view configuration information, the CPU 201 demultiplexer 214, video decoders 216-1, 216-2, 216-3, scalers 218-1, 218-2, 218-3, view, in picture units or GOP units. Controls the operation of the interpolation unit 219 and the like.

  The image data of each view acquired by the video decoders 216-1, 216-2, and 216-3 is supplied to the decoded buffers 217-1, 217-2, and 217-3, respectively, and temporarily accumulated. . The scalers 218-1, 218-2, and 218-3 are configured so that the output resolution of the image data of each view output from the decoded buffers 217-1, 217-2, and 217-3 is a predetermined resolution, respectively. Adjusted.

  Also, the parallax stream extracted by the demultiplexer 214 is supplied to the coded buffer 221 and temporarily accumulated. In the parallax decoder 222, the decoding process of the parallax stream memorize | stored in the coded buffer 221 is performed, and parallax data are obtained. The disparity data includes disparity data between the center view and the left end view and disparity data between the center view and the right end view. The parallax data is parallax data in units of pixels or blocks.

  The parallax data acquired by the parallax decoder 222 is supplied to the parallax buffer 223 and temporarily accumulated. Based on the parallax data stored in the parallax buffer 223, the parallax data conversion unit 224 generates parallax data in pixel units that matches the size of the scaled image data. In this case, when the transmitted parallax data is in units of blocks, it is converted into parallax data in units of pixels. Also, in this case, the transmitted parallax data is in units of pixels, but if it does not match the size of the image data after scaling, it is appropriately scaled.

  In the view interpolation unit 219, a predetermined number of views that are not transmitted based on the parallax data between the views obtained by the parallax data conversion unit 224 from the image data of the center, left end, and right end views after scaling. Image data is interpolated and synthesized. The view interpolation unit 219 obtains image data of N views (View1, View2,..., ViewN) for viewing a three-dimensional image (stereoscopic image) with the naked eye. Note that image data of each view at the center, the left end, and the right end is also included.

  Further, the graphics stream extracted by the demultiplexer 214 is supplied to the coded buffer 225 and temporarily accumulated. The graphics decoder 226 performs a decoding process on the graphics stream stored in the coded buffer 225 to obtain decoded graphics data (including subtitle data). Also, the graphics decoder 226 generates graphics bitmap data to be superimposed on the view (image) based on the graphics data.

  Graphics bitmap data generated by the graphics decoder 226 is supplied to the pixel buffer 227 and temporarily stored therein. In the scaler 228, the size of the graphics bitmap data stored in the pixel buffer 227 is adjusted to correspond to the size of the scaled image data.

  The graphics shifter 229 performs a shift process on the bitmap data of the size-adjusted graphics based on the parallax data obtained by the parallax data conversion unit 224. The graphics shifter 229 generates N graphics bitmap data to be superimposed on the image data of the N views (View1, View2,..., ViewN) output from the view interpolation unit 219, and Supplied to the interleaving / superimposing unit 220.

  The pixel interleave / superimposition unit 220 superimposes graphics bitmap data respectively corresponding to image data of N views (View1, View2,..., ViewN). The pixel interleaving / superimposing unit 220 performs pixel interleaving processing on the image data of N views (View1, View2,..., ViewN) to perform naked-eye viewing of a three-dimensional image (stereoscopic image). Display image data is generated. By supplying the display image data to the display, an image display for viewing the three-dimensional image (stereoscopic image) with the naked eye is performed.

  Also, the audio stream extracted by the demultiplexer 214 is supplied to the coded buffer 230 and temporarily accumulated. In the audio decoder 231, the audio stream stored in the coded buffer 230 is decoded, and decoded audio data is obtained. This audio data is supplied to the channel mixing unit 232. The channel mixing unit 232 generates audio data of each channel for realizing, for example, 5.1ch surround with respect to the audio data. This audio data is supplied to, for example, a speaker, and audio output is performed in accordance with image display.

  As described above, in the receiver 200 shown in FIG. 27, among the plurality of views for displaying a stereoscopic image, at least the image data of the left end view and the right end view and the intermediate position located between the left end and the right end. View image data is received. In the receiver 200, the other views are obtained by interpolation processing based on the parallax data. Therefore, it is possible to satisfactorily perform autostereoscopic viewing of a stereoscopic image with a multiview configuration.

  That is, not only image data of the left end view and right end view but also image data of the center view is received. Therefore, the relative parallax between the views is small, and interpolation around the occlusion accompanying the processing of fine portions when interpolating the image data of the view that is not transmitted is facilitated, and the quality of the reproduced image can be improved. In addition, since the image data of the left end view and the right end view is received, all of the image data of the view that is not transmitted can be synthesized by interpolation processing, and it is easy to maintain high image quality for end point processing such as occlusion. Become.

  Note that the receiver 200 illustrated in FIG. 27 illustrates a configuration example in the case where a disparity stream obtained by encoding disparity data is included in the transport stream TS. When the transport stream TS does not include a parallax stream, the parallax data is generated from the received image data of each view and used.

  FIG. 30 illustrates a configuration example of the receiver 200A in that case. In FIG. 30, portions corresponding to those in FIG. 27 are given the same reference numerals, and detailed descriptions thereof are omitted. This receiver 200 </ b> A has a parallax data generation unit 233. The parallax data generation unit 233 generates parallax data based on the image data of the center, left end, and right end views that have been subjected to the scaling process.

  Although a detailed description is omitted, the method of generating parallax data in this case is the same as the method of generating parallax data in the parallax data generating unit 116 in the transmission data generating unit 110 described above. The disparity data generation unit 233 generates and outputs disparity data similar to the disparity data in units of pixels generated by the disparity data conversion unit 224 of the receiver 200 illustrated in FIG. The disparity data generated by the disparity data generation unit 233 is supplied to the view interpolation unit 219 and is also supplied to the flux shifter 229 for use.

  In the receiver 200A illustrated in FIG. 30, the coded buffer 221, the parallax decoder 222, the parallax buffer 223, and the parallax data conversion unit 224 in the receiver 200 illustrated in FIG. 27 are omitted. The other configuration of receiver 200A shown in FIG. 30 is the same as that of receiver 200 shown in FIG.

<2. Modification>
In the above-described embodiment, the image transmission / reception system 10 including the broadcasting station 100 and the receiver 200 is shown. However, the configuration of the image transmission / reception system to which the present technology can be applied is not limited to this. For example, the receiver 200 may have a configuration of a set top box and a monitor connected by a digital interface such as (High-Definition Multimedia Interface (HDMI)).

  Further, in the above-described embodiment, an example in which the container is a transport stream (MPEG-2 TS) is shown. However, the present technology can be similarly applied to a system configured to be distributed to receiving terminals using a network such as the Internet. In Internet distribution, MP4 or other format containers are often distributed. In other words, containers of various formats such as transport stream (MPEG-2 TS) adopted in the digital broadcasting standard and MP4 used in Internet distribution correspond to the container.

Moreover, this technique can also take the following structures.
(1) an image data acquisition unit that acquires image data of a predetermined number of views for stereoscopic image display;
An image data transmitting unit that transmits a container of a predetermined format including a video stream obtained by encoding the acquired image data;
(2) The image data acquisition unit includes a view configuration information insertion unit that inserts view configuration information including information indicating a relative positional relationship of at least the predetermined number of views into the layer of the video stream.
The image data of at least the left end view and the right end view and the intermediate view image data positioned between the left end and the right end are acquired from among the plurality of views for the stereoscopic image display. The transmitting device described.
(3) The identification information insertion unit that inserts identification information for identifying whether or not the view configuration information is inserted into the video stream layer into the container layer. Transmitter device.
(4) The transmission device according to (2) or (3), wherein in the video stream included in the container, the image data of the left end view and the right end view is encoded as one picture data.
(5) In the video stream included in the container, the image data of the left end view and the right end view is interleaved and encoded as one picture data. (2) or (3) Transmitter device.
(6) The transmission device according to any one of (1) to (5), wherein the video stream included in the container includes data of one or more pictures.
(7) When the video stream included in the container includes encoded data of a plurality of pictures, information indicating a boundary is arranged between the encoded data of each picture. Any one of (1) to (6) The transmitting device described.
(8) In the video stream included in the container, when image data of a predetermined view is encoded as data of one picture, the view configuration information inserted in the layer of the video stream includes the predetermined view The transmission device according to any one of (1) to (7), including information indicating a position of a view.
(9) In the video stream included in the container, when the image data of two views is interleaved and encoded as data of one picture, the view configuration information inserted in the layer of the video stream includes The transmission device according to (1) or (8), wherein information indicating positions of the two views is included.
(10) The transmission device according to (9), wherein the view configuration information further includes information indicating a type of interleaving performed on the image data of the two views.
(11) The view configuration information inserted in the layer of the video stream includes information indicating whether or not data of a plurality of pictures is encoded in one access unit of the video stream. 1) The transmission device according to any one of (8) to (10).
(12) The view configuration information inserted in the layer of the video stream includes information indicating whether the image data of a view essential for image display is a coded video stream. (1) The transmission device according to any one of (8) to (11).
(13) The view configuration information inserted in the layer of the video stream includes pixel ratio information for a predetermined horizontal and / or vertical resolution. (1), (8) to (12) The transmission device according to any one of the above.
(14) A parallax data acquisition unit that acquires parallax data between the views is further provided.
The image data transmission unit
In addition to the video stream obtained by encoding the acquired image data, a container having a predetermined format including the parallax stream obtained by encoding the acquired parallax data is transmitted. (13) The transmission device according to any one of (1).
(15) The transmission device according to any one of (1) to (14), wherein the container is a transport stream.
(16) An image data acquisition step for acquiring image data of a predetermined number of views for stereoscopic image display;
An image data transmission step of transmitting a container in a predetermined format including a video stream obtained by encoding the acquired image data;
A transmission method, comprising: a view configuration information insertion step of inserting view configuration information including information indicating a relative positional relationship of at least the predetermined number of views into a layer of the video stream.
(17) Image data of at least the left end view and the right end view among a plurality of views for stereoscopic image display and an intermediate view image data located between the left end and the right end are encoded and obtained. An image data receiving unit for receiving a container of a predetermined format including the received video stream;
An image data acquisition unit that decodes a video stream included in the container to obtain image data of each view;
A receiving apparatus, comprising: an interpolation processing unit that acquires image data of a predetermined number of views located between the views based on parallax data between the views by interpolation processing.
(18) The container includes a disparity stream obtained by encoding the disparity data,
The receiving device according to (17), further including a parallax data acquisition unit that obtains the parallax data by decoding the parallax stream included in the container.
(19) The receiving device according to (17), further including a parallax data generation unit that generates the parallax data based on the image data of each view obtained by the image data acquisition unit.
(20) Out of a plurality of views for stereoscopic image display, the image data of at least the left end view and the right end view and the image data of an intermediate view positioned between the left end and the right end are encoded and obtained. An image data receiving step for receiving a container in a predetermined format including the received video stream;
An image data acquisition step of decoding the video stream included in the container to obtain image data of each view;
A receiving method comprising: an interpolation processing step of acquiring image data of a predetermined number of views located between the views based on parallax data between the views by interpolation processing.

  The main feature of this technology is that the view configuration information including information indicating the relative positional relationship of at least a predetermined number of views is inserted into the layer of the video stream, so that the relative positional relationship of each view is received on the receiving side Can be easily grasped and the stereoscopic image display process can be performed satisfactorily (see FIGS. 14 and 15).

DESCRIPTION OF SYMBOLS 10 ... Image transmission / reception system 100 ... Broadcasting station 110 ... Transmission data generation part 111-1 to 111-N ... Image data output part 112 ... View selector 113-1, 113-2, 113 -3: Scaler 114-1, 114-2, 114-3 ... Video encoder 115 ... Multiplexer 116 ... Disparity data generator 117 ... Disparity encoder 118 ... Graphics data output unit 119 ... Graphics encoder 120 ... Audio data output unit 121 ... Audio encoder 200, 200A ... Receiver 201 ... CPU
211 ... Antenna terminal 212 ... Digital tuner 213 ... Transport stream buffer (TS buffer)
214: Demultiplexer 215-1, 215-2, 215-3, 221, 225, 230 ... Coded buffer 216-1, 216-2, 216-3 ... Video decoder 217-1, 217- 2, 217-3: view buffer 218-1, 218-2, 218-3, 228 ... scaler 219 ... view interpolation unit 220 ... pixel interleave / superimposition unit 222 ... disparity decoder 223 ... Parallax buffer 224 ... Parallax data converter 226 ... Graphics decoder 227 ... Pixel buffer 229 ... Graphics shifter 231 ... Audio decoder 232 ... Channel mixing section 233 ... Parallax data Generator

Claims (17)

An image data acquisition unit for acquiring image data of a predetermined number of views;
An image data transmission unit for transmitting a container in a predetermined format including a plurality of video streams obtained by encoding the image data of the predetermined number of views;
A configuration information insertion unit for inserting the configuration information of the video stream into the layer of the video stream ;
A transmission apparatus, comprising: an identification information insertion unit that inserts identification information for identifying whether or not the video stream configuration information is inserted into the video stream layer in the container layer . The at least one video stream among the plurality of video streams includes a plurality of substreams including encoded image data of one picture obtained corresponding to image data of one or a plurality of views. The transmitting device described. The transmission device according to claim 1, wherein when the video stream included in the container includes encoded data of a plurality of pictures, information indicating a boundary is arranged between the encoded data of each picture. When image data of a predetermined view is encoded as data of one picture in the video stream included in the container, the configuration information of the video stream inserted into the layer of the video stream includes the predetermined view. The transmission device according to claim 1, wherein information indicating a position of the transmission is included. In the video stream included in the container, when the image data of two views is interleaved and encoded as one picture data, the configuration information of the video stream inserted into the layer of the video stream includes: The transmission device according to claim 1, wherein information indicating positions of the two views is included. The transmission apparatus according to claim 5, wherein the configuration information of the video stream further includes information indicating a type of interleaving performed on the image data of the two views. The transmission apparatus according to claim 1, wherein the configuration information of the video stream includes information indicating whether or not data of a plurality of pictures is encoded in one access unit of the video stream. The transmission apparatus according to claim 1, wherein the configuration information of the video stream includes pixel ratio information for a predetermined horizontal and / or vertical resolution. An image data acquisition step of acquiring image data of a predetermined number of views;
An image data transmission step of transmitting a container of a predetermined format including a plurality of video streams obtained by encoding the image data of the predetermined number of views;
A configuration information insertion step for inserting the configuration information of the video stream into the layer of the video stream ;
A transmission method comprising: an identification information insertion step of inserting identification information for identifying whether or not the video stream configuration information is inserted into the video stream layer into the container layer . An image data acquisition unit for acquiring image data of a predetermined number of views;
An image data transmission unit for transmitting a container in a predetermined format including a plurality of video streams obtained by encoding the image data of the predetermined number of views;
A configuration information insertion unit for inserting the configuration information of the video stream in the layer of the video stream ;
Above configuration information of the video stream, the video information is contained Ru transmitting device indicating whether data of a plurality of pictures are encoded in one access unit stream. An image data acquisition step of acquiring image data of a predetermined number of views;
An image data transmission step of transmitting a container of a predetermined format including a plurality of video streams obtained by encoding the image data of the predetermined number of views;
The layer of the video stream, have a configuration information insertion step of inserting the configuration information of the video stream,
Above configuration information of the video stream, the video transmission method information Ru include indicating whether data of a plurality of pictures are encoded in one access unit stream. An image data acquisition unit for acquiring image data of a predetermined number of views;
An image data transmission unit for transmitting a container in a predetermined format including a plurality of video streams obtained by encoding the image data of the predetermined number of views;
A configuration information insertion unit for inserting the configuration information of the video stream in the layer of the video stream ;
Above configuration information of the video stream, horizontal and / or vertical predetermined transmission device that is part of the pixel rate information for the resolution. An image data acquisition step of acquiring image data of a predetermined number of views;
An image data transmission step of transmitting a container of a predetermined format including a plurality of video streams obtained by encoding the image data of the predetermined number of views;
The layer of the video stream, have a configuration information insertion step of inserting the configuration information of the video stream,
Above configuration information of the video stream, horizontal and / or transmission method that is part of the pixel rate information for a given resolution in the vertical. An image data receiving unit that receives a container in a predetermined format including a plurality of video streams obtained by encoding image data of a predetermined number of views;
The layer of the video stream, the configuration information of the video stream is inserted,
The configuration information of the video stream includes information indicating whether or not data of a plurality of pictures is encoded in one access unit of the video stream,
A receiving apparatus, further comprising: a processing unit configured to process a video stream included in the container based on configuration information of the video stream to obtain image data of the predetermined number of views. An image data receiving step of receiving a container in a predetermined format including a plurality of video streams obtained by encoding image data of a predetermined number of views;
The layer of the video stream, the configuration information of the video stream is inserted,
The configuration information of the video stream includes information indicating whether or not data of a plurality of pictures is encoded in one access unit of the video stream,
A receiving method further comprising a processing step of processing a video stream included in the container based on configuration information of the video stream to obtain image data of the predetermined number of views. An image data receiving unit that receives a container in a predetermined format including a plurality of video streams obtained by encoding image data of a predetermined number of views;
The layer of the video stream, the configuration information of the video stream is inserted,
The configuration information of the video stream includes pixel ratio information for a predetermined horizontal and / or vertical resolution,
A receiving apparatus, further comprising: a processing unit configured to process a video stream included in the container based on configuration information of the video stream to obtain image data of the predetermined number of views. An image data receiving step of receiving a container in a predetermined format including a plurality of video streams obtained by encoding image data of a predetermined number of views;
The layer of the video stream, the configuration information of the video stream is inserted,
The configuration information of the video stream includes pixel ratio information for a predetermined horizontal and / or vertical resolution,
A receiving method further comprising a processing step of processing a video stream included in the container based on configuration information of the video stream to obtain image data of the predetermined number of views.

Images